Slashdot Mirror
Building a Budget Storage Server
An anonymous reader noted an article running over at Firingsquad talking about
building a budget storage server. Talks about cooling, power, RAID, expandability, etc. Good overview type article, with practical application.
Since they couldn't afford RAID, what about software RAID? Way faster than normal IDE operations.
This is a test. This is a test of the emergency sig system. This has been only a test.
Finally a place I can store all my p0rn/warez/dvds. But seriously why did they put in a 3D Graphics cards on a server. Surely any cheap AGP card even without 3D will do. Some basic ATI's are just $20
Rus
Cheap UK and US VPS
I took a Beige G3/266MHz that I got for $50... put a 120 GB WD drive, ACARD IDE Controller, and Mac OS X.
An extra fan, to keep it nice and cool, and a 10/100 NIC.
Runs rather well. Smooth, reliable, and fast. For a very low cost. Mac OS X 10.2 comes with AppleShare, for Macs, and Samba for windows file sharing. Apache for a webserver, and PHP, Perl...mySQL.
You got whatever you really need.
I added webmin, for remote control. Makes it a bit easier.
I built a similar system for the web rack (disks are bulky, compared to 1U motherboards). Gave me 1.5 TB of SATA hardware RAID-5 in 2U. All the other machines boot off it - much better use of space :-)
Simon
Physicists get Hadrons!
Interesting tip. I didn't know that. At the same time, 3Ware recommends using identical drives if you want maximum performace for reads on RAID1.
... it's possible to buy a large PC case and fill it with a large number of drives that add up to a volume of storage that was once considered to be large several years ago. What's new here?
The article could have covered a little more than just the hardware needed to run such a setup, perhaps covering some sort of remote management interface for the storage? It would also have been nice to hear if they solved the problem of backing up this data on a budget too. (Ingoring the possiblilty of burning the data to DVD).
Wouldn't a mini-itx system make more sense here? You're building a simple storage server, doesn't need to be massively huge. A 533mhz processor (the low end with mini-itx boards, I think) is plenty fast enough to run a simple storage server.
Video card? Why on earth would you need a $70 video card for a storage server! He should have gotten a motherboard with integrated graphics, so even if he needed to attach a monitor, integrated graphics would be more than enough to handle anything. What is he building, a storage server or a full fledged PC?
If you want reliability you cannot just rely on ONE server anymore. Just get the cheapest boxes that meen the requirement and get *2* of them. Use DRDB and heartbeat to make the failover seamless. With these two cheap boxes you get 24x7 reliability at a 7-11 price. Raid, cooling, ... will all help in the one box senario delay system failure, but that box *WILL* fail. Two boxes can help not only with outages, but upgrades as well since the primary can be taken offline for upgrades without any upseting of the system.
The latest issue has reduntancy and scalability articles that go from 2 boxen to as many as you want.
http://www.linuxmagazine.com/
I salvaged a derilict dual P3x450, dug up enough 256meg sticks to give it a gig a ram and a salvaged video card.
For drives, I watch and wait until I need more space, then I add a drive, ussually whatever Fry's has on cheap. I use LVM to add it to my partitions. Of course, I can only add a total of 4 drives this way before I'm forced to by a off board controler (I'm at that point now).
The other downside is that there is no redundancy, but oh well. Redundancy is expensive.
Performance stinks as I violate the rules about one device per controler. Of course, I don't care because I'm accessing it over a 10mbit network (via the phone lines in my appartment). It is sufficient to stream video to 2 or more machines so no worries.
Total cost ~$500 worth of hard drives. Everything else was "free".
Andrew
Spell check? Why bother. That is what grammer/spelling Nazi freaks who waiste band width posting "spell right" are for.
ECS Fully-Integrated motherboard
Athlon 1800XP, 256MB Ram
4x 40GB IDE Hard Disks
Promise SX4000 Raid-5 Controller
All in a micro_ATX chassis
Can't get much cheaper than $700 for a 120GB storage server with at least some measure of redundancy.
Next, what are you uses? I mean most small business work groups I have seen might store larger Powerpoint, excel and other files. It takes them a while to fill up dual 160GB hd's in a raid 1.
Still, for our company we purchased 1.6TB Xraid's from apple with Fiber cards. Why? well we are doing a lot of work with FCP and need the quick access times that come with fiber vs. ethernet.
"The problem with socialism is eventually you run out of other people's money" - Thatcher.
What constitutes an identical model? Is it number of cylinders and heads? I don't know much about hardware, but I'd love to try this if I could be sure I would still get that read performance they talk about when using identical drives. I seem to recall they even said from the same manufacturer. Probably because they assume I'll screw it up otherwise. =:-o
Built a storage server two years ago, it's run like a tank since I put it online.
Dual 800MHz PIII in a Supermicro Motherboard.
Cheap-O video card
Gigabit card
40 GB system drive.
6x80MB Maxtor drives (5400 rpm)
Escalade RAID-5 card.
I chose 5400 rpm drives for several reasons:
A) A little bit cheaper
B) Used half the power of the 7200
C) Runs a lot cooler
D) Higher MTBF
Every drive that has ever failed on me has been because of heat. I put several fans in the case to make sure the drives don't overheat. So far so good (knocks wood)
My company just recently invested in a mass storage solution, since it's obvious that mass, redundant storage on SCSI (>300GB) isnt a cost effective option for a small office environment. We took the easy way out and purchased the following:
:-)
Dell PowerEdge 1600SC Server:
Xeon 2.0Ghz
512MB RAM
18GB U320 15k RPM (OS Drive)
32x CD-RW/DVD Drive
We chose this server because it has both PCI33, PCI66, AND PCI-X slots on the bus, supports up to SIX internal hard drives and has two 5.25" drive bays.
For the mass file store we chose Maxtor 300GB 5,400RPM 2MB Cache Drives. You have to remember this is not going to be an active file server but more just a file repository and source control/backup server for a small office (10 Clients).
Our Mass Storage Solution Is:
3Ware 7506-8 RAID Controller
4x Maxtor 300GB Drives
We're going to put the Maxtor Drives on a RAID5 and since the 3Ware is a Switching HARDWARE 64-Bit/66Mhz PCI RAID card for IDE Drives, performance should be stellar.
I think all in all the entire solution ended up costing us around $4,000 for parts and systems, BUT, we also got OS (Win2k 5 CAL) and a 3 Year Dell Warranty on Parts.
I think $4,000 for a 900GB Hardware RAID5 on a Xeon server aint too shabby
Total $3,140
/. (including me) could build a server for less than half that, and I would bet that for storage activities, it would be equivalent or faster than this moron's PC.
Okay, I just looked at the article again. $3,000? Damn. I wouldn't mind having that budget...
Seriously folks, if you think you need $3,000 to build a server, then you're out of your minds. I don't want to be modded as Flamebait, but anyone here at
Video Card? Keyboard? Mouse? No. Shouldn't even be there. Yeah, sure, during initial setup, connect a secondhand monitor, mouse, etc (who doesn't have a spare monitor lying around? I have one 10 yrs old lying around somewhere and it still should work). But after initial setup, after you install and configure Linux/Apache, Windows/IIS, FreeBSD/whatever combos, forget it. After that, you should be able to telnet or remote admin the server.
I'm going to issue a challenge. Alexis Dang (the author of this piece), if you're listening, here's a challenge. Give me $1500 and I'll build you a server that can beat your server in storage related activities. Not video games, not music, not Paintshop testing.... just pure storage. Hell, give anyone on this board $1500, and they can beat your "server" upside down.
I agree, that seemed much more like, "Watch us build an expensive PC with a lot of hard disks" than "Watch us build something useful for reliable network storage."
I am currently trying to put together a RAID 5 file server and they do not cover any topic of use to me in that article. For example, practical backup solution? They chose a DVD burner, why that over similar tape solutions? I would guess price, but it would be nice if they at least mentioned some of their considerations. Especially since it would take 112 DVD-R's to back up a terrabyte?
Also, aside from their DVD backups, they seem to have no data recovery plan in case a hard drive fails. I guess they aren't storing anything important on these drives?
OS X 10.1 users are still waiting for a patched SSH.
While Apple includes server software in OS X, Apple is not excited about you actually making use of this software (they would rather that you buy OS X Server), so it will constantly be a thorn in your side.
I've thought about OS X server applications, but...
It seems like a good deal at first, but look before you leap.
that looks like a fun home project but I would start here when looking for a small office server Penguin Computing Relion Servers
Yup.
And they build a storage server without ECC memory. We had that - by chance we noticed that a memory chip generated single bit corruptions. We were busy for days comparing every file with older backups. And that was around 50 GB data, not 1 TB.
Nice rig. Like the case. Why do you need an extra 500 watt power supply? Did you remove the one that comes with the case?
How are you doing backups?
Isn't the esclade a 64 bit pci card? I don't think the a7n8x supports 64 bit, correct?
If it's a server, isn't data integrity a higher priority than sheer performance? Why aren't they using ECC memory modules? Price is not an issue - I have a dual Athlon MP system which supports ECC and I'm running 1.5 GB of PC2100. The 512MB ECC modules were only like $112 each.
Plus they complained about not having front-panel firewire and USB! WTF? This is supposed to be a server isn't it? Not an iMac!
And my final rant - An NVIDIA FX video card? Are they smoking crack? A Matrox Millenium PCI card is all you need in a server. GeForce FX is the last thing I would ever imagine to find in a budget storage server.
I'd rather be a conservative nutjob than a liberal with no nuts and no job.
"Besides from that, they seem to fall between 2 chairs here, whats with the geforce card for a storage server."
I think they missed the point, they went budget, then went picky with things like expensive keyboard and not using onboard video because they really needed a DVI input. Lame.
I could save $ on that stuff alone.
Keyboard and mouse combo are beyond needs, gimme a $5 keyboard and $5 mouse. A display requiring DVI is probably LCD, we'll say he wants a $250 display (to be cheap to say the least) and %70 for the excess video card. I'll go with onboard video and a cheap $50 monitor (hell I can save on that by just switching one from another PC) for.
I saved $300 there at least.
It's a storage server, you're not supposed to connect to it ever once it's started up. And what's with the statement that it shouldn't run Linux? I couldn't think of a better OS to run a server on under a budget. Why spend bulk cash to pay for an operating system on a machine that's just sitting there? His concern is USB/Firewire compatibility to access old data as it's moved out of the machine... uh, I have no intention on ever bringing my storage servers down after they're up, especially to do a dirty op in removing a storage disk. And what's with DVD writers for backup? You're storing 1TB of data on the machine and you're going to back it up 5 GB at a time? Please connect to a tape library if you're going to back up that much crucial data PER YEAR (as stated the drives should fill in a year).
This is not a good article for any of it's goals.
It doesn't mean anything.
For an organization with a very large number of deployed drives, it helps them estimate the number of spares they need for a given amount of time.
Healthcare article at Kuro5hin
I wish I could buy a cheap firewire-based raid enclosure for multiple drives... I'd be willing to pay $400 for one that'd hold 4 drives in a snap...
We had ca 25% of our deathstars fail inside two years.
Enterprise-class drives come with a five-year warranty. A truly bad batch of drives should be purged by then, leaving the manufacturer to pick up the tab for replacement. Also, you cite having to replace a drive every two to three months, which is a pain but not as bad as patching Windows.
Healthcare article at Kuro5hin
You want budget? I bought a pair of Maxtor 40 GB HDs, a stick of 256 MB of RAM and stuck them in an old P150 with Windows 98.
That thing is used by a workgroup of about 6 for storing just about everything they need with their business. (They're using at most 10 gigs.) Every night, a batch file starts XCOPY to backup all files that have changed during the day from the main disk to the backup. Every little while, I burn CDs of everything with the snappy little script I wrote.
Total cost at the time (two years ago?), probably 500 bucks.
That's what I call a "budget storage server".
Alexis builds a budget storage server and explains why you can't take a random desktop and add a bunch of disks.
Funny, the article never really explains why you need all the super-super parts that he uses to build the server, except for the reasons like the UPS looks cool on your desk. Whoever wrote this really just took a budget and blew it on a bunch of cool parts to build a kick ass server, but would have been out of a job if he did the same at my IT department for wasting IT budget and time.
-$0.02
Anyone can build a PC and fill it with 300GB IDE drives. Big deal.
A few problems:
#1, if you've got BIG files (read: audio/video) or a quite a few users, it won't be fast enough.
#2, It's not a real SAN device. Where's the SCSI and Fibre channel? Can you plug it in to multiple servers?
#3, How do you plan on backing this up? a stack of DVD-R's? A bunch of 80GB tapes?
Here's a good budget config:
The Xserve RAID can sustain roughly 130+MB/sec write throughput on the RAID-5 set
Since we're on a budget, choose the cluster config of the Xserve's with only 10 client - can upgrade to unlimited version for $500.
No storage kept on each head - instead, boot off the RAID. If a head dies for any reason, change the LUN masking on the RAID controller and bring up the second head as the first head. No physical changing of cables necessary. Add a PDU with web management for complete remote operation. Downtime is roughly 5 minutes after admin launches the admin Java application (and the web browser for the PDU). This is budget after all.
This system can server up roughly 2 terabytes on two separate volumes (or do software RAID 0 to combine them) over a variety of network protocols - AFP, NFS, SMB, WebDAV, ftp, and whatever else you can slam into the open source core. It can also be a member or server of a variety of directory services including LDAP, NT PDC, NetInfo, NIS.
Includes network based remote management as well as serial port access for integration into existing console access servers. Solution is easier to manage than most UNIX (including Linux) setups.
Price competitive against similar Wintel setups, especially as the number of clients go up. Server is good for 50-150 clients depending on client performance demands.
Add software and hardware support contracts, rack system, UPS(s), gigabit switch(s), etc.
Price is a bargain $16,770 (ClubMac mostly). Compare against $25k+ systems with similar resiliency, manageability, and performance
Easy there, cowboy. I'm not an apologist for the writer, I'm just pointing out what he said.
Heck, if you're spending $3K, what's a 10% increase to prevent failure?
I was also ticked about the lack of OS explanation other than "Linux won't cut it."
For what they ultimately achieved, you could get fairly comparable results with a well tweaked setup from mwave.com. (Which is basically what they did.)
$30 Off All Plans: Use code TRIPLESAWBUCK
Indeed. They have no budget for a IDE RAID card (?) because they spent it on ... a ninja PC case, snazzy keyboard and mouse, and a graphics card. Whoopee do. That's not going to be very useful stuck in a room by itself, is it? They also go with a big-ass P4 which needs a big lump of heatsink with a high power fan on it. One of the first things I learned building PCs was fans mean failure. Get a joke processor (or as close as will do the job) and stick the biggest passive heatsink you can get on it. I've never seen one of those fail. Not being funny, but I sincerely hope nobody follows the advice given. Oh, and one other thing - why not get 5,400rpm drives? Yeah, they're slower, but it's going over a network and they're data files. they can wait a little, and the heat you don't generate it worth it, surely?
I agree, that seemed much more like, "Watch us build an expensive PC with a lot of hard disks" than "Watch us build something useful for reliable network storage."
My solution to building a cheap storage system was the following:
1. Buy old Netfinity 5000 on eBay.
2. Order 5 x 9GB SCSI drives from my trusty IBM parts guy (csaunders at itexchange.com)for $70 each.
3. Order basic RAID card for said box.
4. Install RedHat 7.1 from a CD in a book under my couch.
5. Install SAMBA
6. Run cron job to back up user data and relevant config files to an external USB hard drive attached to a windows box on the lan.
7. Take external hard drive to safe deposit box weekly. Get second USB drive out of safe deposit box and attach it to windows box at office to await next update. FWIW, I've been thinking about putting the USB drive that is in the office in a safe when the back up is not taking place. This is not for fear of fire or catastrophe -- I just don't want it to walk out the door.
8. The Netfinity server has the RAID 5 array configured for a hot spare drive so that there is failover operation if a drive quits.
9. Installed PowerChute software with a UPS to shutdown the box gracefully if power quits.
External USB -- $100 each (2) = $200 (got enclosures and cheap-o spare IDE hard drives from scavenged boxen)
SCSI Drives -- $70 each (5) = $350
Netfinity box = $300
UPS = $200 (I think)
Redhat 7.1 on CD in book under sofa = priceless
Total: $1,050.
Project results:
RAID-5 with regular offsite storage. Logical disk size is only 27 GB, but you can fatten this by using bigger SCSI drives. I didn't need mondo storage, so I saw no need to go with 36 GB drives, though you certainly could if you had more money.
I am currently trying to put together a RAID 5 file server and they do not cover any topic of use to me in that article. For example, practical backup solution?
External USB drives worked for me. Depends on how heavy-duty you need and how your office works. Perhaps simply connecting up two servers in different offices and doing mutual backups nightly for changed files might suffice. DVDs and CDs are an option, and tape is still useful.
Also, aside from their DVD backups, they seem to have no data recovery plan in case a hard drive fails. I guess they aren't storing anything important on these drives?
My data recovery plan (if everything pukes) is to buy a new chassis and drives and reinstall RH 7.1., connect it to the lan, and download old config files and user data. I think it would take a couple of days (mostly waiting on delivery of the drives and box). That time could be slashed if I were truly paranoid if I simply kept spare parts off-site. I'm just not that worried, however.
FWIW, our office is a small lawyer's office with about 10 people on our LAN. The data we need to store is not huge.
GF.
Lots of petrified grits
There are adapters that turn a cheap ATA drive into a SCSI drive, roughly in the neighborhood of $100 apiece. If these things work well, then one may be able to put a whole bunch of cheap+100 (which is still pretty cheap) ATA drives on a SCSI (160 or 320 or whatever) channel, instead of adding a bunch of ATA cards and using up all the PCI slots.
Yeah, yeah, ATA sucks, whatever. I have actually had good luck with cheap ATA drives (better than my SCSI experience in the 1990s, for sure) and besides, you RAID 'em based on the assuption that yes, they will die. And when one dies, you pay chump change to replace it. (And when they don't die, then you smirk and say, "That's because I'm ready for it. What would be the point of them dying?")
Another good thing about this approach is that SCSI makes it easy (compared to ATA) to use external enclosures. I'm party convinced that a lot of my good and bad drive experiences have been related to cooling issues. (My expensive Micropolis SCSI drives roasted inside my Amiga 3000 case, but my Coolermaster and Lian Li cases are doing a great job of keeping my cheap Maxtors comfortable.)
Anyone have any experience with building RAIDs of SCSI-adapted ATA drives? Can they transfer at Ultra160 (or 320?!) speeds?
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
Ah yes, all excellent. I love the home network.
:)
:)
Currently I have a few of the Shuttle boxes that run 1ghz p3 and 1.4ghz celeron & a white box generic Tyan 1.2ghz Athlon dually.
The p3-1hgz does my personal domain hosting - web (tomcat/apache), mail, openh323 gateway, ut server.
The celery does mail and public ssl egroupware portal for my home biz.
The dually is running rh 9 w/software raid across 2 ata-100 drives. It handles all my MySql stuff as well as the main repository for all my machines backups - it's on an the internal net. Once a week nightly backups are dropped to tape and once a month, the backup dir is wiped. So I always have at least 1 week of backups on tape, which I'm quite happy with. This is also my CVS server too.
All my development is mainly done on me 1ghz Powerbook. I also have a excellent 80gb Seagate Barracuda (fluid bearings) in an external firewire enclosure that has all my mp3z & mov/divx's. Sometime I just drop it on the dually and it mounts it up and shares it on my network, but most of the time I keep it local. My Powerbook is also dual head w/a 17" 760V Samsung LCD. Great monitor for the price.
My last machine is a kick ass little Shuttle box with a Athlon XP 2000+ and Maxtor ata-133 20gb drive. It's dual partitioned w/XP and Fedora. I use the Fedora side to develop for another Linux server that is just easier vs. doin it on my Mac. The XP boot is for Starcraft & UT.
All the machines have 1gb of ram except the two Intel Shuttle boxes. Everything internally is connected via a DLink Fast Ethernet switch. My external net is just a cheapie FE switch which is all bridged together via a even cheaper little broadband 'router'. It has dhcp, firewall and other things built it and has worked pretty well for me. My connecion is 1.5 down and 786 up DSL. No cable here. There's also an Airport Base station on the wall that provides wi-fi for family and friends when they drop by.
I've been running networks for many years outta my various cribs. The biggest and baddest was when the whole boom was goin on. I had a startup w/some friends and I had 9 machines in a closet of my studio apt. This included 2 big (1 size down from rack mount) UPS', a Dell quad Xeon 550 server (scsi 10k rpm drives with raid 5), 512mb ram and hot swappable everything - 4u also. This was out big app server. I was also running 2 RH 5 dns servers, 1 RH 5 mail server, 1 RH 5 ssh server, 3 dell p3-500 2u apache servers and one failover app server (Weblogic), 2 NT SQL 7.0 servers with full replication. I also had a separate tape backup box that was just a p2-350. Ah, the good 'ol days. After a bit we sold the company and I went traveling to Europe never to have another rack in my closet again.
Anyway, I love home networks and my two big obsessions now are: quiet and huge storage. I thought the article was kinda lame when I go to the video card and mouse part - something my servers have never really had or needed (cept maybe the nt junk). One thing I found higly annoying is when RH 9 required a mouse to install. I didn't like that much, but no big I guess.
So I have yet to find the magic bullet machine that will hold a 1TB array for cheap and for quiet. But maybe some day. Could it be that now that there's 5400 rpm 2.5" drives that that could be the thing??? Low power and quiet..
I built this out of cannibalized parts last January 2003. I suppose by now you could probably double the media storage for the same cost -- there's a lot of rebates for PATA drives around.
Supermicro P6DBE (1997 vintage)
2xP3 600MHz
Adaptec 1940UW SCSI
Software RAID 1
x2 36GB Seagate SCSI drives
(web server)
1GB ECC PC100 RAM
x1 WD1600JB PATA drive
(apps)
Promise SX6000
Hardware RAID 5
x6 WD1600JB PATA drives
(media server)
ATI Rage Pro
(it's a server!)
Antec 1040SX Case
Antex True480 - 480 Watt PSU
Basically, all I bought new were the drives, the case, and the PSU. Total cost below $1300. Serves several thousand visitors a day, peaked at 30K hits for a while following a Slashdotting. CPU usage peaks around 20%. Using J River's Media Center, I've tested it serving 6 simultaneous 720x480 DIVX streams to clients over LAN and WAN with no problems.
These chumps spent 3 times what I did, and they don't even have disk redundancy. Who let the dogs out?
Da Blog
RAID isnt about increasing MTBF, its about not having to bring the system down and not losing data when (not if) a drive fails. When one of their four drives fails, they will have to stop the system and restore from an old backup. It takes a while to restore a 250Gb drive. And of course all the data since the last backup is gone.
Then there's the fact that they have four seperate drives, while with RAID you get one big one.
RAID controllers, especially the Escalade one, do a much better job of managing disk writes than your onboard IDE controller. For server usage you will see a much higher response for multiple users using RAID. No reference is made to the different behavior of drives in a server compared to workstation drives.
They remark that RAID wont protect your data because the PSU or motherboard can fail. Ok, I have never had a motherboard fail. That doesnt mean that they don't, it just means that their MTBF is way beyond discs. I have had a PSU fail, but not in a way that damaged the computer. You could consider dual PSU solution. Or a post-psu UPS (i.e. where the battery is between the PSU and the motherboard), as opposed to, or in addition to, a traditional pre-PSU UPS.
But then, the whole article is something of a joke. A $3000 budget server with the most expensive RAM, CPU, Keyboard and a bloody LCD panel??? I dont know what planet they are on but for $3000 I built a dual P4 XEON box with a Promise SX6000 pro raid controller. (I buy escalade now, I might add). And a $160 Keyboard for keyboard and mouse? What's it need a keyboard for? For $160 you can buy a 4 port KVM switch and use a keyboard you've already got. Or spend $5 on a basic one.
They use one gigabit port to connect to the internet. Why not connect both ports to a capable switch and get 2Gb/s?
BTW, I tried low-end Seagate, Maxtor and WD, and finally found that Samsung Spinpoint drives survived the longest in a RAID box.