Slashdot Mirror
1.8TB Of Disk Space In A (Semi-)Normal PC
zdzichu writes "A friend of mine is building a personal server. He bought 17 of the cheapest IDE drives available and used Linux' LVM to get them together. The result? Almost two terabytes of disk space in regular x86 PC. The
most juicy part - photos are here.
For an operating system, he first tried the enterprise-ready PLD Linux Distribution, later he reinstalled Slackware Linux." Update: 03/01 20:24 GMT by T : I'm sure that should be "drives" and not "drivers" :)
You can get 8 of these and make 2TB easy. Most computers support 4 anyway, so another controller for 4 more would be no problem. Sure, it'd cost you a bit, but hey, it's 2TB!
For those with any emotional attachment to the guy- "!!! THIS IS NOT MY SERVER !!! " is indicated on the bottom of the page (I must have managed to get it pre- /. effect)... So thankfully his drives won't be causing a smoke effect large enough to fill a stadium...
For what it's worth the only picture I managed to download looks like three great big sticks with a couple of hard-drives on top of each other in between the sticks (look something like a surreal altar to the god of IDE Drives). Not exactly enthralling to me (obviously not got enough of a life).
And the question that really must be asked (Budha may have the answer)- space is as good as it's content... Maybe he could run a server to help the poor sods affect by the /. effect...
The only thing that saves us from the bureaucracy is its inefficiency (Eugene McCarthy)
Go into your preferences, and put a check box beside 'hardware'. You'll never see one of these stories again.
Another non-functioning site was "uncertainty.microsoft.com."
The purpose of that site was not known.
here.
Who are you? The new #2 Who is #1? You are #617565. I am not a number, I am a free man! Muhahaha.
Here it is before slashdotting.
Okay, with PCI, you shouldn't have to deal with IRQs. If they don't work right, just put them in different PCI slots (also be sure to read your motherboard's manual for it's interrupt routing first.)
Second, 3Ware, and a couple other companies, make 12-drive ATA RAID cards. So one of those, plus onboard ATA would reach 16 drives. Or, a second ATA RAID controller would allow an additional 4, 8, or 12 hard drives without resorting to the onboard ATA. For a max of 24 drives without using onboard ATA. (In my personal server, I have 8 10GB drives on an ATA RAID card... They're in dual 0+5, for a whopping 60GB of space, but it's fast, and reliable. Someday I'd love to upgrade them all to Maxtor 300GB drives, but I'd need a new RAID card in the process. [and a large fortune.])
Another non-functioning site was "uncertainty.microsoft.com."
The purpose of that site was not known.
MIRROR HERE: http://crazyserver.150m.com
Enjoy!
PS: Sorry for the banner ads, it's a free server.
I've set up a mirror here, but decreased the quality on the images to hopefully prevent destruction of my site ;)
temp mirror
He is running the first 4 drives off of the two on board IDE raid controllers and has what appears to be three PCI controller cards in his machine
TechTV did this in XP last year...
Like 8 IDE drives, 1TB+ on XP.
This isnt news
If anyone saw the pictures, his setup looks awfully precarious.
He has 1 normal PC case, 2 homemade stands for the drives, and one more homemade stand for additional power supplies.
The stands with the drives look like they could topple with a moments notice! Why did he put them at the top...?
I agree. It would have been better engineered with 2 power supplies at the bottom of each tower, providing a more solid base for the disks. I'd of made the towers far shorter as well - perhaps even turned them sideways.
I think it would be better to mount as many power supplies and drives in 2 additional cases, with the shells removed.
That would be expensive, and would also make heat an issue. The setup he has allows for passive cooling - even a case with the shell removed would trap more heat. Heat can lower MTBF - not something to do with IDE drives. A proper external disk case would make even more sense, as most come with fans and cooling.
Might be a problem with IDE cable length; maybe you could do 2 next to each side the the master computer.
Now you know one of the reaons why SCSI is king for servers - it's meant to be used both internally and extrenally. I've used 10' long, high quality SCSI cables to attach external disks to servers in my time without issue. As well, you can have 14 disks per SCSI controller - not possible with IDE.
It's a nice hack, even if it had design issues, though.
Soko
"Depression is merely anger without enthusiasm." - Anonymous
After all the stigma sucks for pornstars and they at least deserved to be paid for their horrible jobs
Well, you have to suck to get stigmatized. The real hit on mainstream porn has been amateur porn from overseas. Round up a few starving Belarussian girls pay them what is in their eyes a king's ransom, then take the digicam back to your iMovie-laden iMac and 1,2,3 you are a porn magnate.
If you ever thought Jenna Jameson was getting exploited (which is a tough sell), you haven't seen shit until you see this stuff. Obviously frightened women getting grovel shagged by overweight dudes from Valley Stream who kick 'em back to the cold with 50 bucks and a case of genital warts.
Wait, all you Libertarians, no they have no choice. You gotta keep the lights on somehow. Wait, all you Free Marketeers, go back and actually read Adam Smith. He warns against shit like this, particularly white slavery.
Comparing it to Windows will be a moot point, since El Dorado is going to have a 40% larger code base than XP.
From the pictures, it seems like these are all sitting on a 32-bit 33MHz PCI bus, with a maximum throughput of 133MB/s. My drive, which is getting a little old now, can sustain 20MB/s. Assuming that he's using some kind of striping / mirroring, rather than just plain concaternation, and assuming he gets the same throughput per drive I do, he's going to be needing almost 3 times the bandwidth of the PCI bus. A 66MHz or 64-bit PCI implementation would be less of a bottleneck, but I can see everything else on the PCI bus grinding to a halt when he accesses the disk array. Assuming he's using a PCI base network interface, this isn't exactly what you want on a server...
I am TheRaven on Soylent News
Before you answer porn consider how much money this array cost and how much money it would cost to actually *pay* for the xxx dvd's.
Eh? Who said anything about porn? Maybe this guy wants to mirror linux/BSD isos or other software, docs, other websites, etc. You can never have too many mirrors, after all. Or maybe it's just a fun project. It would be nice if not everyone jumped to the conclusion that this guy is setting this all up for some grand warez site or what have you. At least I hope so, anyway.
I disagree with what you say, but will defend to the death your right to tell such LIES!
This setup is not stable. I get regularly filesystem corruption if I stress the system. Apparently linux can't deal with the fact that the total transfer rate of 12 modern ultra dma133 disks more than maxes the PCI bus.
I don't think it's Linux, bud - my suspicion is that the controllers themselves can't deal with a maxed out PCI bus. They are normally bus-master cards, so it's possible that one is grabbing the PCI bus and holding it for so long that the one of the others is giving up, instantly corrupting your array. Promise likely didn't take into account such setups where there's more than one or two RAID contollers per machine.
Currently I am thinking about changing the raid 5 arrays into just plain volume groups without stripping. This would allow me to lose some of the transfer rate and avoid stressing the pci bus.
I doubt it - you still are tryng to squeeze way too much data through the northbridge chip on your motherboard. You may be able to do something with PCI bus timings, but you stand a very good chance of hosing the whole setup that way. You simply need more motherboard bandwidth if you want to support that much disk space - sorry.
What you need is a dual (or triple) peer PCI bus motherboard, so you an have 2 controllers per northbridge channel. Look into SuperMicro and one of their ServerWorks GC LE based boards.
Soko
"Depression is merely anger without enthusiasm." - Anonymous
But on a much smaller scale.
(4) 120Gb Maxtor UDMA/133 drives from the local "megamart" computer store (Best Buy in the NW) for $89/each (after mail-in rebate, of course). Cost (after rebates) $372.00 (Had to pay the state sales tax, sigh, Washington sucks sometimes!)
(1) Promise SX4000, 4-Channel hardware RAID-5 controller that can handle UDMA/133 drives. Cost = $145.00 from you favorite PriceWatch merchant. Free shipping, no tax.
Slap it all together, format, viola - 360Gb of redundant space for a total of $517.00
My big concern was long-term backup - I opted to go with a DVD-R/+R Sony drive. Drive ran $350 at the local office supermart (Plus that d*mn sales tax = $381.10
100 4x capable DVD-R discs were $1.61ea via an online source. 4.7Gb/ea, a total backup capacity of 470Gb. Cost = $161.00, not tax, free shipping.
Drives: $372.00
Controller: $145.00
DVD-R/+R: $381.00 (Could have gone with the cheap one for $199, but wanted the dual-capability)
100 DVD-R discs: $161.00
Total cost = $1059.00
Total capacity = 360Gb (RAID-5)
Backup time = 15m per disc, ~20h for 360Gb (swapping discs sucks, but sure beats paying tape backup prices)
What is the space used for? Try DV video editing sometime and tell me how far you get with a 40Gb drive in your machine.
1800gb / 0,7gb dvdrips = 2571 DVDs
2571 * $20 (at least, here) = $51428
What? Unfair comparison? Well you're comparing with an extreme machine. Maybe kazaa sucks, I don't use it. But at my uni there's no problem getting more movies than you'll ever see, mostly in quality DVD rips by ripgroups in a matter of an hour or less per rip. Not that it makes it right, but if you want me to do a pure financial estimate less moral costs, rips win hands down. That's not a troll or a flame, that's a fact. Even if you factor in the chance of getting caught and fine, it still wins hands down. And no, having a client running in the background of my machine isn't really costing me much time, it's a fire and forget thing, check back later.
Kjella
Live today, because you never know what tomorrow brings
I wonder how much this would cost scsi wise.
Figure a 72G 10K rpm SCSI disk at $500 times 20 + 4 spares = 12000 for the disks. Then figure that a raid controller runs $500 - $2000 and add a large hot plug chassis and you're looking at $15k. However, You now have hardware supported RAID at up to 400MB/s sustained and all of those drives are covered by a 5 year warranty. The 4 spares are just insurance against a supply problem down the road. Of course, you need to buy you disks from different lots (5 per dealer perhaps) to minimize the effects of a bad lot. Yeah, SCSI is expensive, but you get better reliability.
"We returned the General to El Salvador, or maybe Guatemala, it's difficult to tell from 10,000 feet"
The 2.5 kernel has a new compile option that makes the sector count a U64. Check out CONFIG_LBD. (Configure Large Block Disk).
The block devices in 2.4 kernel can't go over 2TB right now. It's fixed in 2.5, but I don't know if they are going to backport or not.
We have run into this barrier at work several times. With large ATA arrays, it's getting almost trivial to amass 2TB+, so I sure hope this gets fixed post-haste.
AMS DK-035A (ignore the Ultra SCSI reference on that page, the A suffix is for ATA), available for $159 from Central Computer
I just set up an eight drive RAID using one of those, and one 3-drive-in-2-bay version, the DK-023A ($119 from Central Computer). That way eight removable trays fit in my 5-bay 4U rack mount case.
I used a 3ware Escalade 7500-8 RAID card rather than Linux software RAID, but there's no reason why it wouldn't have worked with software RAID. I just couldn't find a "dumb" eight-port ATA-133 card. (And I like the 3ware cards.)
I initially tried to use Serial ATA, using the Promise SATA150-TX4 4-port Serial ATA controller and some Highpoint RocketHead 100 Serial ATA adapters for the drives. The Highpoint web site claims that the RocketHead 100 is not available for sale as a separate product, but lots of retailers including Central Computer seem to have them. The cabling was *much* nicer, but to make the SATA150 work with Linux a binary-only driver was required, so I decided not to use it until there's a driver available in source form.
I thought about using the 3ware Escalade 8500, which is the Serial ATA version of the 7500, but there's quite a price premium, so I decided to stick with parallel ATA for now. Maybe next year I'll set up a bigger RAID using Serial ATA.
This is pretty easy to do, but this person was lucky he used 100 gig drives. Lots of motherboard IDE controllers won't support more than 137 gigabytes/drive, and neither will older versions of Linux. RH starts supporting the larger drives in 7.3. I think any controller promising 133 mbits/second will also support the large drives. I posted some details at www.nber.org/sys-admin/maxtor-160.html because most of the discussion in mailing lists was questionable.
INVENTORY
- 1 x *nix host
- 4 x PCI dual-channel Ultra160 or 320 adapter
- 8 x racks (42U needed)
- 112 x 3U RAID array w/ Ultra160/320 interface
- 1568 x 320GB maxtor drives
1. one big fat software RAID 0 layered on RAID 0.
500TB of storage. Find more PCI slots or
quad-channel SCSI adapters and make it an
even 1.0PB.
2. each 3U array conf as stripe of RAID 1+0
then exposed as 1 SCSI device.
arrays paired up, across racks.
each array connected to different SCSI channel
on different card.
each array pair software RAID 1+0.
grand software stripe across mirrored arrays.
yields 125TB (mirror to power of 2 cost).
- supports up to 7 drive failures per 3U.
- supports up to 56 3U array failures
- probably could support up to 4 total rack
failures
- support up to 2 SCSI card failures
- support up to 8 SCSI channel failures
- fails when *nix host fails
Cost not including rack/space: $1.2 million
$2.41/GB
$9.66/GB (double RAID-1 layers)
$4.82/GB (single RAID-1 layer)
~$4.00/GB (double RAID-5 layers)
~$6.00/GB (single RAID-1 layer plus RAID-5 layer)
Estimates:
$400 per 320GB drive
$5000 per driveless 3U RAID ARRAY with cartridges
$100K for host, plus SCSI adapters/cables.
"even Windows XP does now" Actualy XP(NT) supports and has always supported partitions and file sizes up to 16 exabytes, which is 18,446,744,073,709,551,616 bytes. A far cry above 2 and 4 TB partitions. Anyone ever see www.terraserver.com, thanks to NTFS and NT, several TBs of information are easily searchable and viewable, and have been for several years now. (The only exception to the partition limitation in the NT history, is that Windows NT 4.0 and earlier versions required the SYSTEM/BOOT partition to be under 7.8gb do to the boot hardware specifications of the time.) But other partitions are and have only been limited by hardware capabilities, at least until you hit the 16 exabyte NTFS limit. Windows 2000 and XP (NT) can have a 16 exabyte boot/system partition if hardware supports it. (This is why the NT Team that were some of the lead UNIX programmers of the era chose to create something that wasn't UNIX and therefore leave behind all the UNIX limitations, giving birth to NT with Subsystems, NTFS, etc, etc.)
Without wanting to take away from the coolness factor... Wouldn't it be easier to just go and grab your self one of these?
The Apple XServe RAID have almost as many drives, but they also have all the extra stuff that goes with it. I would like to know how he has got it all set up. What happens when one drive goes south?
At least with the XServe RAID, you can set it up with hardware RAID 5.
Uh yeah, right.
Big deal.
I suppose you know all about the NTFS limitations by reading the source, too, eh?
; -- the corruption of government starts with its secrets. a truly free people keep no secrets. --
You call them whatever you want. It's only the device numbers that matter to the kernel, and most distributions don't even have prefab specials in /dev beyond about hdk anyway. You could call the 27th drive /dev/hdA or hdaa or drive27 or anything that looks good to you.
http://slashdot.org/~psm321/journal
Stupid lameness filter, wouldn't let me post it here.
All the current responses are wrong. Read Documentation/devices.txt. There are no defined major/minor numbers past the tenth IDE interface, hence no device special file in /dev.
However, you can put in a boatload of SCSI disks. After the 26th disk (sdz), it rolls over to sdaa, sdab, and so on. Of course, for that many drives, you would need to create those device special files yourself (or use devfs).
He could've just used 500GB HDD's. Although it wouldn't have been half as fun...
Joe Llywelyn Griffith Blakesley
[This post is in the public domain (copyright-free) unless otherwise stated]