Slashdot Mirror
Server Redundancy for a Small Business?
SadPenguin asks: "I am currently working for a small company of about 15 people each with one to two workstation/laptop machines a piece. We are looking for a new server solution, as our last one crashed, and lacking any server redundancy, we nearly lost all of our data since our last backup (it was only a few days, but an important few). What the kind of server (and redundancy) solution would be appropriate for a company of my size? Most advertisements are for large scale enterprise serving solutions, but these are costly and excessive for my situation. I'm sure that there is a simple Redundant Server technology out there that is a bit less costly, but won't result in any downtime in the event of a motherboard component failing (like we faced this time when our mysterious surface soldered VRM failed). So what do you use? What should I use?"
I actually run a computer consulting firm specializing in small businesses. I'll outline some of the more common recommendations - with what I think is the most important first.
From my experience, the best approach is to layer your defenses. I'd REALLY recommend a UPS (I generally assume this is purchased with a server, but it isn't always) at very least. Your local power company is only required to provide you with something CLOSE to 120v. They generally can't keep it consistent enough for power supplies (and electronic componentry in general). Protect your investment, UPSes are generally relatively cheap.
The fact that you've got a backup solution is good, but (as you've seen) not enough. Evaluate it, and see if it's consistent with best practices - i.e., is it a tape (or optical) backup system that is done in rotation and taken offsite by somebody in the company? If not, set that in motion first.
Next, some sort of drive redundancy is in order. At very least, mirror your drives. I generally recommend RAID5 (or one of its variants), but in very small companies RAID5 isn't either required or affordable or both. IMO, the jury's still out on the long-term viability of IDE RAID, but I think it looks promising.
Finally, redundant power supplies and NICs (for those of us that are REALLY paranoid ;) ). I've had a couple of servers' power supplies die on me, but the server kept right on ticking thanks to a redundant unit.
If it's affordable to your company, consider hot-swappable server components, as well. This significantly reduces downtime to your coworkers... and expense to your company.
Hope this helps. Good luck!
Oh yeah, FP ;)
We all get along together like tornadoes and trailer parks.
It all really depends on how much money you want to spend. You could roll your own dual opteron server and thrown in a bunch of small (20-40 GB) hardrives and RAID 5 'em. That would be my solution. It would cost you like 2 grand if you do your homework and get a good hardware raid card. 3ware makes good stuff that's compatible with Linux.
>> we nearly lost all of our data since our last backup (it was only a few days, but an important few)
Daily backups !
general recomendations:
quality server (Dell/HP/etc)
NO ide drives!
SCSI in software raid5
minimum software install (e.g. no compilers)
get second 'devel' server to test/compile software before using on production server
If it is not broken, don't fix it. as in screw with the devel server.
Christopher McCrory "The guy that keeps the servers running" chrismcc@gmail.com http://www.pricegrabber.com
I work for a small company that only has three full-time employees (including me). I use two Debian boxes (cheap-o machines that are just retired desktops with some big cheap IDE hard drives in them) running Samba. I use the rsync mirroring technique I found here.
One box is the "live" server and the other mirrors the live server every night. If the main server dies (which happened once - power supply failure), I can "promote" the backup server by changing one line in its Samba configuration. As a bonus, the backup server keeps "snapshots" back a week or two.
I do three types of redundancy/backup at my sites:
* Mirrored Raid in all servers
* A regular workstation with a good, large had drive that copies the server data to itself nightly
* A DVD-RW backup made nightly on yet another workstation, with at least one off site - 5 discs, one each weeknight, replaced a few times a year.
In most cases the server RAID (cheap ATA promise controllers) takes care of 90% of the problems - only one HD goes bad at a time, lightning strikes rarely take out the hard drives at all, nevermind both hard drives, etc. Even if it dies it's unlikely that the problem affected the HD backup on the other workstation, and it definitely didn't affect the cd-rw.
However, whenever you get a catastrophic failure in any component in the server, replace the entire thing. If the MB or power supply fails, copy the data to new hard drives, and use the old ones in less critical applications, etc.
Much cheaper than an 'enterprise' solution, and it should be because your application doesn't require such a solution. Use large tape drives in place of the dvd-rw if you must back up a huge amount of data on a nightly basis.
This sort of solution is very tolerant of cheap hardware, so replacing the server later may not be such a major cost.
-Adam
This is a hard problem(NP-Hard perhaps, I'm not sure), and you need to have a:
List of applications you want to protect
Budgeted amount
What threats you are trying to protect from
What kind of failures you will tolerate(do you need 99.9% uptime? or better? worse?
You could, for simple applications, like web service, bump up a pair of linux machines, gimmick some replication between the two, and hope nothing goes wrong, if you have a very low budget, and you'd probably spend a fair amount of work debugging later on, "synchronisation problems". But for redundant storage. The openssi project is working on highly-available single-image clusters for linux, in an open source model, they might be your first place to look. It's not however, something for the unprepared to do, nor is it something that I'd recommend if you do other tasks for this company. Maintaining such a beast will require a significant implantation investment. The good news is that once everything works to your satisfaction, you can probably take a 4 week vacation somewhere with golden beaches and much sun, and let it take care of itself. I can't stress this enough, this is a hard problem, if you really want to do this right, you'll want to surround yourself with qualified people with experience in this field, it's non-trivial, and mistakes can lead to severe data-loss.
At my place of work (18 people) I have set up spare low end machine (p233) with a 80gb drive as a backup file server. During the day every 15 minutes everything that has changed is copied to the backup server. The backup fileserver is configured as read only so a user cannot accidently change anything.
If the main fileserver goes down I simply change the configuration to read/write and change filemaping on the users machine and they continue to work. The whole process will take about 10 minutes to reconfigure the server and a couple of minutes per user machine.
As a bonus I dont delete the intermediate versions of changes files as I update the server. Instead I compress them with a unique filenames. So I can recover a fairly complete history of any given file. I have yet to fill up the 80gb drive so I havent needed to delete any backups. When the backup drive is full I will start deleting some of the older version, I should have room for about 6 to 9 months of backups at 15 minute intervals.
I've been a system admin for a production webserver for a few years now, and I can tell you this.
99.9% of the time when I've had to retreive data from backup, it was because of human error. I.E. someone deleted something they shouldn't have, or the moved the wrong directory to the wrong place, or an error was made during a software upgrade, etc..
the rest is due to random harware failure which would be a reason for using RAID. But pouring thousands into redundant servers and disks, is overkill for a biz your size.
If someone accidently wipes out a folder or data, your raid disks won't be any help.
Love,
Zaq
Is there any sort of RAID 5 available in the range of between $1000 and $2000? I have a small law firm, and would love to be able to have the redundancy capability offered by RAID 5. I would think there is a market for a stand alone Firewire box that I could pick up. The box could either come with 3, 4, or 5 harddrives, or allow me to pick up my choice of harddrives separately and just plug them in.
Daily backups, #1
What kind of server though?
Mail? SQL? Files?
If you can do it the best way to handle is Clusters with an external Raid 5 device that is a shared resource between the two(or more) servers.
Set them up with a shared hardware Raid 5 device.
There is only one active Node in the cluster at a time, if that one fails the second one assumes the identity. Works great never fails!
We are a bit larger so we use EMC Symmetrix, however a smaller shop could probably do a low end EMC Clariion CX200 or the like.
Power Corrupts,Absolute Power Corrupts Absolutely, leaving one person(group)in charge is absolutely corrupt.
I'm a sysadmin for a tier 2 autmotive company in Michigan with about 35 client machines.
The two main servers are xeons with raid 5, redundant PSU's etc etc. One server runs the domain and as a file server while the other runs the manufacturing software suite (heavy database workload). All the data is very important but I rarely have a problem with lost data , unless some smuck over-writes a file or something stupid like that.
The backup solution I implemented was a Debian box that runs rsync every night backing important data to the hard drive of the backup server. This machine doesn't need to be anything special. Now if someone looses something I have ready access to it. On Fridays I do tape backups of the data on the backup server and then run rysnc with the delete option so only the data currently on the servers resides on the backup server. This solution is quite simple and straight-forward utilizing cron and bash scripts.
So far it has been working quite well...
-Tolerate my intolerance
We just finished building a 2.5 TB (terabyte) server for less than 5000$. You could probably spend even less than that since we spend about 1000$ on two fiberoptic cards. We have 2 6 chanel 3ware RAID cards and 12 250 133ATA Maxtors hooked up to a 520 watt powersupply plus another 520 watt power supply acting as redudant power(we did that mod inhouse). 2.5 TB is probably more than you guys will need unless you are doing some advertising or something like that... so you could probably go for 1 TB, which will cut your costs down even more. So all in all you could probably get it done in about 3000$ not too shabby for 16 ppl. Our server backs up my whole college.
I too have long experience doing small business consulting and in some other areas. One thing you could do is use RAID-1 with a spare drive. That way if you lose one, you aren't screwed. You also could have a couple spare drives in hot-swap carriers. Pull a drive every night and have a duplicate of your server. Fire up the duplicate server and pop in your known good pull and boot if you server fails.
OS dependent, you don't even have to have exactly the same hardware if you use a more generic kernel build and you can list a different NIC for the spare server in the conf file for modules assuming you aren't compiling them into the kernel.
Continue with good backups made to another machine, to tape/CD/hard drive, or off-site. This way, even if your good pulled drive is a little out of date, you can bring to data current in short order.
You don't mention the OS of the server or budget, but I'll assume that since you've got 2 machines per desk time 15, you can afford a spare server. You don't mention OS and that affects cost, but still, if you are doubling up on hardware on desktops, you can afford to do this or most any of the other solutions offered.
Of course, you get what you pay for and if the experience is lacking in house, hire a knowledgeable consultant or company you trust to do it for you.
a relatively cheap setup for data/service redundancy for a small business.
* two identical servers, running linux (of course).
* heartbeat
* drbd
* two UPS
Notes, Ins, Outs and What Have You's
service redundancy
heartbeat is used to make 2 servers look as if they were one. if one of the servers dies, heartbeat makes sure the other assumes the ip address and has all the relevant services started.
data redundancy
drbd is a network block device. again, it looks like one device, but when data is written to it, its actually being written to 2 seperate locations. if one box goes down, heartbeat makes sure drbd makes the other box primary.
hardware
these two call for a dedicated network and serial connection. so 2 nics and a serial port per box.
definitely raid array of some sort.
see drbd.org for more details.
this is not a 100% proof setup, but its cheap and covers most of the bases.
of course, it requires a linux dude to get it all to work.
you may benefit from a combination of heartbeat and DRBD, which respectively provide IP address/service failover and a network (no special hardware required) data replication solution.
If you have appropriate hardware you might also appreciate Stonith, which provides forced-shutdown of a failed node (in the case that the failed node won't release the IP address, and hence you would otherwise have problems switching service).
If you're in the UK then give me a shout and I'll set it up for you (for a reasonable fee)! My contact details are available on my web site.
Yours Sincerely, Michael.
If you're already making regular daily backups, and are only worried about in-between-backups, run RAID on your server -- I forget the specific RAID number, but use the one that mirrors your data on two disks (not the one that speeds up disk access by splitting your data between disks).
.. get people into the habit of running CVS or Subversion client on "their documents" folders. Tortoise integrates right into Windows explorer. Advantages: file versioning, ability to work off line and still sync with the server later, etc.
if people actually work with plain text docs, they would love how CVS,etc will merge multiple users' changes.
Of course you would back up your CVS server but in case of a crash, chances are that very important file can be found on the desktop of the user who edited it the last time. Much better than relying on a network drive and then it is just not there.
It's already been mentioned a little, but a second server kept up to date with rsync may be a cheap way to go depending on how big your server is. While I don't know how much data you are talking about, I would expect rsync could sync a few times a day easily via a cron job.
I would suggest springing an extra $90 to get two extra gigabit ethernet cards and a crossover cable for a dedicated connection for rsync which doesn't compete with office traffic.
Using rsync as a basis, the solution could be made as low tech and simple or automated complex as you feel is needed.
-Pete
Do woodworking? 50 Router Bits
Soccer Goal Plans
One of the things that I think people underestimate is the importance of version control. Far too often data loss is due to somebody accidentally deleting a file they spent a week working on. With version control you should be able to revert and get back most of the work.
The other is redundant hardware. As people point out, RAID etc. only provides protection for the redundant components. If the controller, motherboard or such which is not redundant goes bad, you are screwed. The best solution is two servers with some sort of mirroring.
Another factor to consider is location. One of those servers should be off site somewhere so that if the sprinkler system goes off (or thieves get in) or a disgruntled employee gets access you don't have all your eggs in one basket. It might be as simple as having some employee with a company paid cable modem stick it in a closet.
Finally, you need to make sure that restores really work. Many times I've seen data lost because the restore process failed due to lack of testing, even though all the hardware and software was in place.
If you are using Windows 2000/2003, an easy redundant file serving solution is to setup DFS (distributed file system). Just a tip, don't setup a domain-wide share for a file server that gets a lot of updates. Using DFS like that can create an administrative nightmare (last writer wins situation). You would want to use a domain-wide share if you have a lot of read-only files (like installation files, PDF image archives, etc) and you need a high-availability solution. You would be restoring files from tape a lot. Anyhoo, if your first server crashes, temporarily redirect your users to the second server either via DNS or just renaming the servers. DFS doesn't replicate printers, so you would have to install a new printer two times, once on the first server and a second time on your second server. Shouldn't be too much a problem if you only have 15 users.
If you are using Linux/UNIX/*BSD, you could use Rsync. There was a great article explaining Rsync usage in the June '04 print edition of SysAdmin.
If you can't see the business opportunity for a small and cheap business server distro solutions. Then you must be blind.
1. Do the things mentioned in other posts.
2. Distributed OS.
3. Offer offsite backups
4. Profit!
-- I don't buy it, I grow it.
I'm a sys admin at a small mission in Uganda. We landed in some hot soep after blowing up the server a couple of times. We now use 2 junk computers with suse 8.0 with samba and have (rdiff-backup.stanford.edu)make a differencial backup every night. Works wonderfull!! Also alows you to repair a blunder of some days back. Check it out.
Similar company size, about 20 employees,
we have a nice server with 5 36GB drives, running RAID 5, and another old system, with 2 120 GB IDE drives running RAID 1 in software (redhat), this machine rsyncs every hour with the main server... Its been fine for 2.5 years now.. lost a drive once in the RAID 5, replaced it and everything came back up fine...
Look into this. http://www.hp.com/products1/storage/products/tapeb ackup/obdr_print.html It really works. I have used it in the past to save my ass. In theory, you could set up two identical very low cost servers in terms of hardware. Have one in production and the other in a different location in case of a true disaster. Then, do full back ups. If one server fails, just bring the other one in and restore from tape. Bang! You are up in running in very little time. It will make you look like a hero. It can also be used to covery your tracks in case of a screw up. For instance, if you load the latest patch of whatever and it blows up the OS. Just restore from tape. Basically, coupled with the proper backup solution, this bad boy will do open files, exchange servers, SQL DBs, all while the system is live. This solution is beautiful and I love it. Enough said.
A lot of posts seem to surround getting a large, professional server machine with redundant everything. Those are expensive and still have points of failure.
I would suggest buying a number of the inexpensive wal-mart PCs and clustering them redundantly. Keep spares around for emergencies - emergency switches, nics, drives, etc.
This is a more technically complicated environment, because you have to worry about data consistency between computers, but, these walmart PCs are disposable and can work independent of each other.
fnord.
Unless you can not afford a downtime, you don't need RAID.
In Soviet Washington the swamp drains you.
One thing that everyone seems to be missing is the question of how important the data is to you. IF the loss of a server (for an hour/day/etc.) is going to cost you $10,000 (purely an example figure) then you could probably justify putting around $10,000 or so into a nice top of the line server (you'd still have to skimp on things at that price, but still, it's to give you an idea). IF, on the other hand, having the server down for a day or the data loss you experienced costs your company only a couple hundred and happens very infrequently, then I wouldn't think you should spend $1000s on a beefed up server, but instead find a nice UPS, or just pump some money into a Service Level Agreement for the server, or even for offsite backups. It's all about your bottom line...don't just spend money on a server because of this one time. Fix the problem, but keep in mind how large of a problem it is to start with.
why not used something designed to run "forever," like a nice old Ultra 10 or any other Sun machine? Unlike the majority of choice in x86 land, these computers are actually made to be servers that can't afford to not function.
Sunbelt software makes a product called Double-Take that I like the looks of. It's a bit pricey, but it allows you to mirror your expensive live server with a cheap whitebox PC (assuming it has the processing power to *be* a server for a few hours or days while you fix your broken production box). Plus it's real-time mirroring across the network with automatic failover.
And no, I don't work for these guys, or resell the software, and I don't even actually use it myself. I just think it's a nifty product that could be a good fit, if you can scrape up ~$5,000 for a hardware-independent backup solution.
Almost forgot the URL:
http://www.sunbelt-software.com/product.cfm?id=111
Initially, we mirrored a Snap drive to a remote site via rsync, but dropped that when we downsized. We've used Backup Exec to a 30gig tape, but that's finicky - tapes seem to go south for no discernable reason. Currently experimenting with DVD, but it takes lots of disks to do the full backup, and I'm flagging.
How do you do offsite?
I actually found a company that specialized in a Lunux box that would be ideal for this problem.
http://www.pugservers.com/
Not to tute my own horn, but I have a company that is trying to tackle this problem. We have a small, low-cost RAID server at the moment. http://www.pugservers.com/
I'd love to hear your thoughts...