Slashdot Mirror
Horizontal or Vertical Server Architecture?
zetes asks: "For a while now I have been a supporter of a horizontal server architecture/environment, where one server does one and only one thing and a business/department would then have numerous servers. This way if a service needs to go down (especially in a Windows Server environment), nothing else is affected. However, due to the rising importance of security in the last couple of years, I have decided that having fewer servers is both easier to manage and update and also provides fewer points of entry for hackers or virus writers. I am not going to a purely vertical architecture, where everything is on one server, but more of a hybrid (down from 10-12 servers to around 5). So what I am asking you all today is what you prefer to support or manage? One or the other, or a hybrid? And does the operating system or particular service(s) dictate this architecture to a certain extent?"
I have a laundry list of require... oh, sorry, wrong ask slashdot.
More servers = ease of administration only if you have automated patch management tools. Otherwise the "patch one service at a time" advantage is negated by the deployment of said patches to many machines. It is usually the load on the services that really determine how to break out your servers. In a windows enviroment, SQL and Exchange should never be on the same box with any kind of load: both require too much memory. Determine how much load each service places on your hardware and combine one high load with one or more low load services if you want to cut down hardware. Also consider server uptime: some services are more prone to failure modes that bring down entire machines. You want to keep those independent no matter what. Things like DHCP, File Shares and DNS can coexist peacefully in most environments with many other service types.
Sig under construction since 1998.
I prefer many servers with one major service each.
/usr/src/sys/whatever /var/run/whatever.pid` /usr/sbin/whatever
/tmp
/tmp/script
:-)
.. break down your machines into easily understood groups and services.
I have a group of about 10 FreeBSD servers and I use the "clusterit" port which has "dsh", distributed shell. There are also other packages with similar functionality, i.e., run stuff on many machines at once.
Each time I need to patch them, or do anything really, I just write a script like this (abbreviated):
#!/bin/sh
cvsup
cd
make clean && make depend install && make clean
kill `cat
then I test it on my "test group"..
then I use "pcp" to copy it to each machine:
pcp script
then I run it
dsh
et voila, each machine is now up to date! Some minor testing and you're done. Anything else will be discovered by angry customers.
I can admin many machines this way. I have several other homebrew scripts that do stuff like run and aggregate Tripwire output, check firewall logs, etc. Also similar stuff for the 2-3 Red Hat machines.
It's much nicer to have one service per machine because when one fails, just one service fails. when you have to move a service, you know exactly what the machine does, no "hidden forgotten services" like a funky FTP server on port 3573 or something.
it's just like refactoring your programming code
my experience, anyway. grain of salt, $0.02, etc.
In my ideal world, each server would have one, and only one, purpose. That way if a piece of software is compromised in such a way that remote access is granted, the damage is somewhat contained.
Also, from a maintenance perspective having single purpose machines make life a lot easier....especially if they are redundant single-purpose boxen.
But, sometime reality has to set in and some service consolidation may have to occur to keep hardware costs down.
Yes, I know what I'm about to say is heresy to the /. crowd, but...
*takes deep breath
Windows is fairly easy to administer, and it is basically point and click, so I would just stay with the 10-12 servers, balance the load roughly equally, and just periodically have each machine automatically contact Windows Update for certain tasks. Or, have one machine keep checking for updates, and when there are updates, have the machine download them, then "push" them to the other servers.
Before Microsoft Windows was considered stable enough to run servers, the norm was to pack as much as possible onto each server. On SPARC/Solaris servers, that's still the case, and is the reason why 64-way servers are actually reasonable - every processor can access the file system, and they share the load. To get comparable (in the CPU power sense) from independent computers, you'd need much more CPUs, perhaps 4 times as much, because an idle CPU in one box can't take over an overloaded CPU in another.
But Microsoft has made it standard practice to set up a box for every service (And making you pay lots of microsoft tax in the process). And at the NT4 days, that was a good practice, because the O/S wasn't stable enough to be trusted with more than one task at a time - Exchange, SQL-Server and other services could all bring the entire O/S down when a malformed user request came in.
Nowadays, the Windows 2000 and 2003 systems are stable enough to run more than one service at a time. They don't use their resources efficiently, but they are capable of the level of separation needed, and are generally stable enough. Yet, many applications are built with the older mindset, and make demands that are not reasonable for a machine that hosts many services. For example, they might require a specific service pack, a specific version of another tool (Database, Exchange, whatever). A batch text processing system might require a specific Word version to be installed on the server, and a different system might require another.
If two such demands are incompatible, you're out of luck because Microsoft doesn't let you install two versions of the software at the same time. Most Unix packages can have multiple version co-existing at the same time.
So, to sum up - on modern operating systems, there's no real inherent technical argument against packing all the services on one machine. However, due to historical reasons (tracing back to days when this was unpractical), you often have no choice but dedicate different boxes to different services when running Microsoft O/S and its tools. There is rarely such a problem on Unix systems and their like.
I prefer a doggy-style server architecture.
Do you even lift?
These aren't the 'roids you're looking for.
There are lots of things that are best when horizontal ;)
At work I don't have too many services running on each box. We have a regular upgrade cycle, so have a good supply of older server-grade hardware that doesn't have a manufacturer's warranty any more but is still good for less critical tasks. Why wouldn't we use them instead of letting them lie around doing nothing? Patching is easy enough with ssh public keys and a shell script that logs into each one in turn and executes a command, eg allserv sudo apt-get install ssh.
Really it does. If you are after increased security, server consolidation isn't necessarily the way to go in any environment. For example, placing additonal daemons on a single machine adds rather than reduces the number of potential exploitable holes a single system has. Running Exchange and MS SQL or MySQL and Sendmail or whatever on the same box presents a greater number of potential points of vulnerability that a minimalistic install and just Exchange or Sendmail or whatever simply because more things are running on the machine. This same concept can be used to distribute updates from your distribution vendor (if that is the route you take) from a single source which has the added benefit of reducing your bandwidth consumption.
Also, reducing the number of servers doesn't necessary reduce the the amount of administration. Most admins download and patch servers one at a time regardless of the environment. You could for example (in a *nix environment), have a base environment server that you used to do all of your patching work/distribution on/from; then you create the package of your choice (RPM, .deb whatever) for install on the other machines using an update script. Or you could copy the binaries, conf files, etc., over to the other machines instead of packaging. For a Windows machine, you can script the patching as well using a decent login processor like KiXtart with a login script that checks recursively for patches and applies what you want. Or you could use Perl and command line tools to accomplish this across your servers (workstations too). This is beyond most Windows adminstrators. It shouldn't be but it is. I've been administering Windows and networks since just after NT 3.1 Advanced Server shipped, Netware about the same length of time, and Linux servers and networks about 5 or 6 years and most Windows admins are too indimidated or lazy to learn anything other than the GUI that ships. I've done this done on all environments for years and years and have seen or know others that do this too.
Horizontal or Vertical Shooters?
Isn't one of the benefits of the horizontal architecture the fact that, since a server can go down and others can take its place, you can update each server one at a time without taking the service they provide offline? Don't most server systems provide a system where you can manage this also? I know Solaris and Mac OS X do (JumpStart, etc. netbooting, Workgroup Manager, Network Install, etc.).
In terms of security, don't you have a firewall? Please don't tell me that all 10-12 of your servers are readily accessible on the internet...
Gabriel Ricard
who the hell chooses 'colitis' as his slashdot id? what, 'brain aneurysm' was already taken or something?
And does the operating system or particular service(s) dictate this architecture to a certain extent?"
The operating system, particular services, capacity, load and reliability requirements dictate the architecture completely. Every situation is different and they all need to be evaluated based on the metrics of that particular situation. For a small ten user office, I would think nothing of hosting DNS, DHCP, web proxy, email, file storage, print server, SQL database and maybe more on a single box but, if that same office had a large SQL databse and the SQL application was the core of their business, I would put it on its own box.
The same holds true on much larger networks too. There will be situations where you can and will run a mail server and DNS server with 2000 users on the same box and there will be others that will only allow 1000 users on the box and no other services besides mail. Or perhaps the individual requirements will require no more than 100 users on a box and that box needs to be part of a fault tollerant cluster to guarantee uptime.
This sizing and design is what network architects figure out before a network is built. They figure out what the requirements are, the loads that those requirements will create, the level of service that is required, projected growth, business continuity and fault tollerance and then they match all of this to the appropriate hardware to accomplish these requirements. Every design is unique and there is no silver bullet.
On Linux, I may run several services that are related like sendmail/bind or apache/php/mysql.
On Windows:
File/Print, one server.
Exchange, one server.
SQL, one server.
AD/DNS/DHCP on server.
Yeah, I'm sure a ton of serious Linux shops stick Oracle on the same box as Apache. Or their groupware servers on the same box as their file servers. Get real.
That's why these exist. With the managment software combined with all you mentioned and now is a good time for an admin. BTW OT but isn't Slashdot acting unusual lately?
Personally, I like to group services by function and politics. You never want a low-priority function running on the same machine as a high-priority function. You never want to have two different managers arguing about who gets to decide how the machine's used. You DO want interdependent services to be on the same machine if the load is low enough to allow it -- it lets you eliminate a whole pile of variables if something goes wrong.
The DHCP server, DNS server, NTP server, etc. all live on several identical boxes. Network services are nice in that, so long as you have redundant machines, you can take down any single machine w/o any notice at all. That means that isolation doesn't matter much. I upgrade one machine at a time (in case something goes wrong), but I use an automated script so that the upgrades are painless and identical. I use cnames extensively, so network1 is also dhcp1, dns1, and ntp1.
Mail servers tend to have their own special problems and service levels, so I never let a mail machine do anything but mail, though I'm willing to allow it to do everything mail related (MX, SMTP, POP, etc.). Redundancy and cnames are your friend. Mail1 is mx1, smtp1, and pop1.
File servers tend to have weird setup, so they usually need to be dedicated machines. If you're in a multiplatform environment, it's nice to be able to serve the same files via SMB, NFS, and AFP from the same machine.
Your webserver and database machines will vary wildly in importance from company to company. Again, politics matter. At most places, the DBAs don't play nicely with anyone else. Give them their own machine(s). On the other hand, if you're selling stuff on the web (low-volume) with a LAMP setup, screw it. Put everything onto one machine. Better yet, set up a Yahoo store and get rid of that machine.
KDCs, NIS, Windows domain controllers, and network log machines are each their own special class of single-purpose machines. No other services running (except maybe an NTP client). Gotta log in at the console to make changes. These are critical pieces of the security infrastructure, so the pain is worthwhile. These are also the first machines to be patched when a new vulnerability comes out. Again, apply patches with a script to ensure that you don't make a typo and to ensure that redundant machines are identical.
And, of course, you need a firewall. Even if you're a university and the firewall does nothing except filter invalid IPs on ingress and egress, it's necessary when the DOS attacks come.
Forward, retransmit, or republish anything I say here. Just don't misquote me.
I think it depends on the situation. I personally prefer fewer systems, but that's just me. A few things to think about:
1, What are the dependencies? That is, do you have an application server that depends on an RDBMS or the like? If so, it might make sense to put them on one machine, assuming you have enough capacity. In a 2-tier system setup, if one server goes down, the service is unavailable anyway. Or you could take those 2 boxes and cluster them for better availability.
2, How much capacity do you need? I have seen a lot of server environments where there were many systems, but most of them were drastically under-utilised. This is most common, IMHO, in Wintel environments, but I have also seen it in Unix shops.
3, Do you need to separate systems for security reasons (e.g. DMZ's for Internet facing apps)?
4, Do you need to charge different clients/departments for the use of these systems? Do these customers insist on buying their own systems and have you manage them?
5, Do you need to support different O/S flavours or different revs of the same O/S?
6, Are the applications likely to consume *all* available resources on a given system? They sometimes warrant their own system or a logical partition of a bigger box (e.g. domain on a Sun system or LPAR on IBMs)
I am currently doing a consolidation project for a large-ish software development and testing team. I'm taking the 19 server environment and bringing it down to about 11 systems, with more to follow when I can move some licenses that are tied to specific nodes. Most of these machines are under-utilised and have direct attached disk spread all over the place. My strategy was to get a NetAPP filer for NFS in the middle and retire all of the old file servers. The next phase will have me retiring four Oracle servers onto one system with SAN storage. So far, so good.
Use a load balancing device such as F5 BigIP, Cisco CSS, or Foundry ServerIron. Put multiple apps on each machine so they serve more than one purpose, but duplicate those machines and load balance across them. You end up with a bit less admin task since the machines are all identical, and you can take down a service or a whole server without affecting anything.
I don't get why more people don't do this. You get more efficient use of your resources by not wasting dual PIII servers for just a single task that really only requires the equivalent power of a palm pilot, you get redundancy, and you have your coolness factor all rolled into one package. Then again, convincing management that has no experience with load balancers is not always easy. They think they are some magical box and really have no clue how this could possibly help them in any way.
Need Free Juniper/NetScreen Support? JuniperForum
Not really suited to all applications (forget anything with large numbers of short lifespan threads/processes - ie. a webserver), but I have managed to completely remove a number of servers used by one client.
Instead we now have one Mosix box (basically with no additional software loaded) which dynamically takes up the slack when a particular sections server is being swamped. Very nice.
Q.
Insert Signature Here
I used to run hybrid servers packing as many function on a single box as practical. The problem here is that it can be tough to manage a mix of many services on a single server.
We have migrated to single application servers running on User-Mode Linux. The only down side here is that you can burn thru a lot of IP addresses, and in that most of our servers are public, it is not the best usage of address space.
In terms of maintenance, virtuals are ideal. You can individually firewall them opening only a few ports, and often only to select destinations. Updates are easily automatable with ssh pushing rpm's to guest systems.
If a system does get compromized, the damage is usually limited. The amazing part is that I have seen hackers in UML virtuals who did not even realize they were not hacked into a "real" system. This is without having "honeypot" mode turned on.
One nice thing about virtuals and single-service boxes is that you can run different software configurations for different applications without worrying about interactions. If SquirrelMail needs a different Apache/PHP version than your primary web server then just run the two in different virtuals. If you need to patch mod_proxy, then you can do it without worrying about killing your shared hosting customers because you only tested it for 15 minutes.
And the best thing about virtuals is that you can lift them up and plop them down on another system with everything 100% intact. Great when systems crash at inconvienient times.
Finally, prototyping a new server now takes a couple of minutes. Just clone an existing virtual, give it an IP address and boot. 15 seconds later, you have a live system. When you are done with it, delete the COW file (Copy-Over-Write block device file) and all signs of your temporary system are gone. It has been so long since I have done a full install, I am not sure I remember how anymore.
I don't want this to sound like an ad, but Virtuals, whether they are UML, Xen (which looks really neat, now would someone please port it's run-time as a LKM), VMWare, full VM, or whatever are just too usefull to ignore.
I generally recommend that you dedicate a server to critically important services and that you provide a few less important services on other servers, with each server running services that get along well together.
Your decisions should partially be based upon which operating system you are using and the service involved. For example, a unix-like OS running a traditional mail server could handle thousands of mailboxes and possibly still provide your DNS and DHCP services, but I would not ask the same of Windows running Exchange. Things can change dramatically depending on whether you're running Windows, NetWare, Solaris, OS/2, MacOS, FreeBSD, or Linux.
It also partially depends on whether your servers are clustered or not. With a NetWare cluster, each server runs mostly independent, but services running on any server can move to any other server in the cluster due to failures or at your whim (server maintenance or load balancing). The simplest case is moving a "disk" from one server to another without downtime or even anyone noticing! Clustering your servers will change how you think about providing services.
Ouch! The truth hurts!
I would say a hybrid approach would be ideal. Use a *nix and run server machines that function individually in a vertical style systems where each machine can litterally handle EVERY singly service run on the network and uses an external data source/storage /fileserver. Then you can add identical "cluster" machines to handle the load, each machine will take its equal share of the total amount of service requests. They would all boot from the centrallized file source, which would have an identical mirror on the opposite side(physically) of your network. No machine would bear the entire wieght of the users wrath by handleing any one service exclusively, but requests would be filled "round-robin"
I have run 3 networks very successfully with this approach. They are all and schools and all handle min 100 users, max 1600. I have observed that this round-robin cluster system is very reliable and very scalable. Updates are simple, update one system, it essentailly updates them all since they all have the same data storage.
I also like having the ability to put servers in seperate locations around the facility. On two of the networks, i run a seperate switch to handle inter-server communications and bootup, this allows me to run gigabit ethernet links between the servers without having to spend the $$$$ on gigabit switches for the entire network, the gigabit switch only supports the number of servers.
This also allows me to use very inexpensive computer hardware and not spend the money on enterprise grade parts. Their is little fear of one server crashing due to hardware failure and effecting the entire network, it simply lowers the performance potential of the server network.
most of the servers are lower end athlon XP chips in commodity hardware(i like the cheap K7S5A PLUS and M7VIG PLUS boards) as they have onboard 10/100 ethernet, are inexpensive and only riquire a gigabit ethernet card.
--
Another advantage is that i can create a "system image" of the OS on the file server and save it, do an update or upgrade, and test it, then have the option of putting the other image back onto the tftp server(bootp is great!!)
Go vertical whenever you can. Often you can't e.g.: public ftp or web server should probably not host a database full of sensitive information. Often you can.
Big challenge is windows apps. Big name packaged stuff like Exchange or Weblogic or whatever you can probably pack together on one server. Once you get into specialized and very expensive software (e.g.: Sagent, Mercator, Fax Software, etc.) the vendor will insist that you dedicate 2 servers (one production one development/test) and refuse to guarantee performance if you don't dedicate a server.
A solution to reduce machine clutter, if not OS clutter, is to virtualize with something like VMWare. In a DMZ for example you could have separate 'boxes' running SMTP, FTP, DNS, etc. all running on one server. Get two of these servers and you've got a pretty secure setup with load balancing. Another big advantage to that is that migrating between servers is as simple as copying the disk image and booting it up. If a system gets compromised you can save that disk image and boot from a known-good one, patch it, and still have the other one for analysis/prosecution.
Same thing goes for internal networks -- run test/development on a VMWare system with a boatload of 'machines'. If VMWare performance is acceptable, then you can run production as well.
"But actually trying to use m4 as a general-purpose langage would be deeply perverse" --ESR
get two servers(A/B).
A / B run all services.
i you need to upgrade A switch to the second idle one B, then upgrade the first one A and vice versa.
On one of our big disk farms that also has backup tape robots attached, the system engineers have all the devices hooked up via SAN (fibre). So really the problem most folks have is: Will the system grind to a halt when I try to do a little tiny upgrade? If you have two distinct services, like disk farm and backups, on the same machine, place cards from multiple vendors in the machine. In our case, we put two Fibre Channel controllers each from a different vendor. Vendor A serves the disk farm, vendor B serves the tape library. When the Tape Library Vendor says that problem XYZ is related to our Fibre Channel controller, we can update the drivers from vendor B without affecting anything associated with the disk farm or Vendor A.
The phrase heterogeneous applies to more than operating systems, but can also apply to hardware within the machine. Now if only I had more PCI slots...
On the software side, if you are on x86, use something like VMWare to running multiple OSes at different patch levels. Under Solaris and probably other big-iron boxes, the hardware can be partitioned without any crossover between OSes.
Let's say XYZ company upgrades their server(s).
Mail
SQL
File
Print
Accounting Software
Dual/quad xeon 3ghz box with 2GB of ram
Let's say it cost $12,000. Pile them all on there.
Mail
SQL/Accounting
File/Print
Buy 3 $2,000 servers. Split them up.
Now, would you rather have a server crash (admin error, hardware failure, software bug) take down the ONE server, losing all services?
Or, would you rather have a server crash just take down the email system, for example. But, with 3 boxes you'd have 3 times the likelyhood of a crash.
I like to split them up. In a Windows environment, we still have programs that like to bomb out. And it's easier to just kick accounting out than it is everyone in the company.
Put all of your eggs into one basket. And watch that basket.
If tits were wings it'd be flying around.
Using rack-mount components saves gobs of floor space.
--
"Outlook not so good." That magic 8-ball knows everything! I'll ask about Exchange Server next.
Let me explain.
Imagine that the hardware on all the machines is identical and that hardware failures will statistically happen over time, with one machine, you get one failure per time, but with two, you get double the failures, and of course with 15 you get 15 times the failure rate.
Now this is true for hardware, but its also true for software. If you have to restart your server once a week, then you'll need to do that 15 times, thus having outages and user problems for 15 times longer than a single reboot.
So, both hardware and software failures are now causing you more grief because you've spread the load around.
Now in the scheme of things a hardware failure is waaay less likely than a software failure, so you could put all your services on one machine and reboot it once a week to take care of software failures.
Some might argue that you could run multiple virtual machines on one big hardware box, say run 15 copies of VMware, which would allow you to separate out your services, but your software failure rate remains the same - actually it's increased, by at least one, because you're running software underneath VMware - but that's getting picky.
You made another argument about hackability and more servers being more of an open door. This is only true if you leave the default services on on all 15 machines, but you wouldn't do that - right?
My suggestion - without having a clue what kind of load you're subjecting this to, is to setup a system that is a live backup. You run all your services in one place - one place to maintain, one place to reboot, one place to secure, one place to fix, etc. - Then mirror the hardware and software and data on another machine.
If you want to be paranoid, you could run two different OS's on those two machines, even different service providers, just the same data-set, but then you're getting seriously paranoid IMHO.
Finally, you asked about OS and Architecture as well as services and their impact on your implementation.
In my opinion, this is not an issue.
If you need to run Windows services as well as for example Linux services, you can run a copy of VMware on either with the other OS inside.
For a really cool idea, you could run only copies of VMware inside (I would use) a Linux box that has no services open at all. The Linux box monitors the VMware sessions, by poking at the ports on the VMware machines and restarts them if they died. You could even use one VMware box to serve as a file-system server, so the VMware boxes running the services would not have any modifiable bits at all. You could then monitor the integrity of the data from the lower Linux level...
If the last bit didn't make any sense, but you're intrigued about what I'm trying to describe, post here and I'll see if I can draw a picture - worth a thousand words.
Anyway, this post is waay to long already, hope it helps...
|>>?