Horizontal or Vertical Server Architecture?

zetes asks: "For a while now I have been a supporter of a horizontal server architecture/environment, where one server does one and only one thing and a business/department would then have numerous servers. This way if a service needs to go down (especially in a Windows Server environment), nothing else is affected. However, due to the rising importance of security in the last couple of years, I have decided that having fewer servers is both easier to manage and update and also provides fewer points of entry for hackers or virus writers. I am not going to a purely vertical architecture, where everything is on one server, but more of a hybrid (down from 10-12 servers to around 5). So what I am asking you all today is what you prefer to support or manage? One or the other, or a hybrid? And does the operating system or particular service(s) dictate this architecture to a certain extent?"

Comments

go with horizontal

[Anonymous Coward]

I prefer many servers with one major service each.

I have a group of about 10 FreeBSD servers and I use the "clusterit" port which has "dsh", distributed shell. There are also other packages with similar functionality, i.e., run stuff on many machines at once.

Each time I need to patch them, or do anything really, I just write a script like this (abbreviated):

#!/bin/sh
cvsup
cd /usr/src/sys/whatever
make clean && make depend install && make clean
kill `cat /var/run/whatever.pid` /usr/sbin/whatever

then I test it on my "test group"..

then I use "pcp" to copy it to each machine:

pcp script /tmp

then I run it

dsh /tmp/script

et voila, each machine is now up to date! Some minor testing and you're done. Anything else will be discovered by angry customers. :-)

I can admin many machines this way. I have several other homebrew scripts that do stuff like run and aggregate Tripwire output, check firewall logs, etc. Also similar stuff for the 2-3 Red Hat machines.

It's much nicer to have one service per machine because when one fails, just one service fails. when you have to move a service, you know exactly what the machine does, no "hidden forgotten services" like a funky FTP server on port 3573 or something.

it's just like refactoring your programming code .. break down your machines into easily understood groups and services.

my experience, anyway. grain of salt, $0.02, etc.

Depends on budget

[venom600]

In my ideal world, each server would have one, and only one, purpose. That way if a piece of software is compromised in such a way that remote access is granted, the damage is somewhat contained.

Also, from a maintenance perspective having single purpose machines make life a lot easier....especially if they are redundant single-purpose boxen.

But, sometime reality has to set in and some service consolidation may have to occur to keep hardware costs down.

Re:Depends on budget

[nocomment]

Yes but then you also have a lot more work to do via patching, and a ton more forensics work to do if a system is compromised. I like to cluster related services services together. I guess I'd be a considered a hybrid.

My firewall runs openbsd, pf, bind 9 (chrooted, primary dns) and snort (chrooted). Snort logs to a monitoring server that is running Mandrake 9.1, postgresql, apache 2, mrtg, some homegrown perl scripts, and nagios. The mail server is running all the mail services as well as dns (to speed up blackhole lookups). It all about clustering the services together that go well, and finding a balance that will allow for optimum network performance.

Once Upon A Time

[Circuit Breaker]

Before Microsoft Windows was considered stable enough to run servers, the norm was to pack as much as possible onto each server. On SPARC/Solaris servers, that's still the case, and is the reason why 64-way servers are actually reasonable - every processor can access the file system, and they share the load. To get comparable (in the CPU power sense) from independent computers, you'd need much more CPUs, perhaps 4 times as much, because an idle CPU in one box can't take over an overloaded CPU in another.

But Microsoft has made it standard practice to set up a box for every service (And making you pay lots of microsoft tax in the process). And at the NT4 days, that was a good practice, because the O/S wasn't stable enough to be trusted with more than one task at a time - Exchange, SQL-Server and other services could all bring the entire O/S down when a malformed user request came in.

Nowadays, the Windows 2000 and 2003 systems are stable enough to run more than one service at a time. They don't use their resources efficiently, but they are capable of the level of separation needed, and are generally stable enough. Yet, many applications are built with the older mindset, and make demands that are not reasonable for a machine that hosts many services. For example, they might require a specific service pack, a specific version of another tool (Database, Exchange, whatever). A batch text processing system might require a specific Word version to be installed on the server, and a different system might require another.

If two such demands are incompatible, you're out of luck because Microsoft doesn't let you install two versions of the software at the same time. Most Unix packages can have multiple version co-existing at the same time.

So, to sum up - on modern operating systems, there's no real inherent technical argument against packing all the services on one machine. However, due to historical reasons (tracing back to days when this was unpractical), you often have no choice but dedicate different boxes to different services when running Microsoft O/S and its tools. There is rarely such a problem on Unix systems and their like.

personally

[larry bagina]

I prefer a doggy-style server architecture.

It really depends

[Halvard]

Really it does. If you are after increased security, server consolidation isn't necessarily the way to go in any environment. For example, placing additonal daemons on a single machine adds rather than reduces the number of potential exploitable holes a single system has. Running Exchange and MS SQL or MySQL and Sendmail or whatever on the same box presents a greater number of potential points of vulnerability that a minimalistic install and just Exchange or Sendmail or whatever simply because more things are running on the machine. This same concept can be used to distribute updates from your distribution vendor (if that is the route you take) from a single source which has the added benefit of reducing your bandwidth consumption.

Also, reducing the number of servers doesn't necessary reduce the the amount of administration. Most admins download and patch servers one at a time regardless of the environment. You could for example (in a *nix environment), have a base environment server that you used to do all of your patching work/distribution on/from; then you create the package of your choice (RPM, .deb whatever) for install on the other machines using an update script. Or you could copy the binaries, conf files, etc., over to the other machines instead of packaging. For a Windows machine, you can script the patching as well using a decent login processor like KiXtart with a login script that checks recursively for patches and applies what you want. Or you could use Perl and command line tools to accomplish this across your servers (workstations too). This is beyond most Windows adminstrators. It shouldn't be but it is. I've been administering Windows and networks since just after NT 3.1 Advanced Server shipped, Netware about the same length of time, and Linux servers and networks about 5 or 6 years and most Windows admins are too indimidated or lazy to learn anything other than the GUI that ships. I've done this done on all environments for years and years and have seen or know others that do this too.

Vertical and VMWare

[smoon]

Go vertical whenever you can. Often you can't e.g.: public ftp or web server should probably not host a database full of sensitive information. Often you can.

Big challenge is windows apps. Big name packaged stuff like Exchange or Weblogic or whatever you can probably pack together on one server. Once you get into specialized and very expensive software (e.g.: Sagent, Mercator, Fax Software, etc.) the vendor will insist that you dedicate 2 servers (one production one development/test) and refuse to guarantee performance if you don't dedicate a server.

A solution to reduce machine clutter, if not OS clutter, is to virtualize with something like VMWare. In a DMZ for example you could have separate 'boxes' running SMTP, FTP, DNS, etc. all running on one server. Get two of these servers and you've got a pretty secure setup with load balancing. Another big advantage to that is that migrating between servers is as simple as copying the disk image and booting it up. If a system gets compromised you can save that disk image and boot from a known-good one, patch it, and still have the other one for analysis/prosecution.

Same thing goes for internal networks -- run test/development on a VMWare system with a boatload of 'machines'. If VMWare performance is acceptable, then you can run production as well.