Slashdot Mirror

Their answer is the Petabox. It's a server setup designed to be "shipping-contained friendly", meaning they can build out a container stuffed with these racks, and have it operational on site with connections for power, cooling, and bandwidth. With this design, they can deploy a mirror of the Internet Archive anywhere that's willing to host it, without having to build a machine room or individual racks on site.

Capricorn Tech of San Francisco builds these machines and their site has more info.

--Pat

And isn't she BEAUTIFUL, too??

Here's the Motherboard Info:
Motherboard/Processor:
* 1GHz VIA C3 CPU
* VIA CLE266 Northbridge
* VIA VT8237 Southbridge
* DDR266 RAM - Up to 1GB
* 2 USB 2.0 ports
* 1 Serial port
* 1 Parallel port
* 1 VGA port
* PS2 mouse & keyboard ports

Anybody have performance numbers for these units? A 1GHZ CPU can be hard-pressed to run an OS, serve disk and support a gige connection at full throughput. I'd be weary of looking at these for a data center without knowing how fast they can serve out the disk over a single gige connection. In fact, I see a distinct lack of information about this unit functions as a "storage node". Are you buying a 1U, 3.0TB node on which you need to install an OS and fileserver? Doesn't look like it would have the horsepower to run an iSCSI driver in additional to software raid drivers and still produce any real transfer speeds.

While a rack of these sounds nice in cost/wattage terms, it appears that you would have just purchased a cluster of storage nodes. A cluster of storage nodes with no way to present the available 120TB's as any kind of coherent storage space. You might be able to run Lustre, PVFS or GFS on them, if that's even possible, but that's a level of complexity the price and performance don't warrant.

If you figure in the cost of a Storage Engineer and lack of performance, this looks less appealing at the full rack level. Doesn't mean some PHB's won't buy into the the whole "Cheap Cluster Disk!" theme though. I pity the sysadmins who get a 120TB of raw disk and 40 more nodes to admin.

They aren't using SATA, they are using PATA. You can see the cables in this picture:
http://capricorn-tech.com/images/mobo176.jpg
They're barely-visable at the top.
Certainly they are a good deal to a company buying a rack full, but for someone like me who's doing it as a hobby, the $3k isn't worth it given I'm only going to have one (ok, maybe 2 :-) and will be setting up the system to my liking anyway...
Still, I wonder if the Sun box that has the 48 drives (vertically) in 3U or 4U that was on slashdot not long ago might be denser...

Robert

http://capricorn-tech.com/

It all depends on what you are trying to do

For some workloads, many servers with four drives each may work. This is the Petabox/Google model. This works if you have a parallelisable problem and can push most of your computation out to the storage servers.
Remember, you don't have a 300Tb drive, you have 300 servers, each with 1Tb of local storage.

For other workloads, you need a big disk aray and SAN, probably from Hitachi, Sun, or HP. This is the traditional model. Use this if you need a really big central storage pool or really high throughput.
Many SAN arrays can scale into the PB range without too much trouble.

Nowadays, a PB is enough to arch your eyebrows, but otherwise not that amazing. It seemes that commercial storage leads home storage by 1000. When home users had 1GB drives, 1TB was amazing. Now that some home users have 1TB, many companies have 1PB.

With the Eternus you get a disk array that you attach to your SAN.

They are two completely different products for totally different jobs.
You'd run you financial database on a Sun attached to a Eternus.
You'd run your Google clone off of a PetaBox.

The Internet Archive Project http://www.archive.org/ is running on the PetaBox http://petabox.com/ rack system, which was commercialized by Capricorn Tech http://www.capricorn-tech.com/ more than a year ago.

This system uses absolutely no board/controller lever redundancy, instead they use a separate file system on every disk, then mirror pairs of 1U units, and finally mirror the entire (mirrored) rack to a geographically distant location.

I am currently testing a much denser solution, the SATABeast http://nexsan.com/products/products/satabeast/sata beast.html from nexsan http://nexsan.com/ which manages to pack 42 500 GB SATA drives into a single 4U rackmount box. With multiple RAID5 volumes and shared hot spare drives, this results in about 17-18 TB of usable file system space.

According to the nexsan engineer I spoke with today, they do so much burn-in testing of the Hitachi Deskstar drives they ship, that over the 15-18 month period they've used these drives, the total error rate has been just 0.4%.

Even if these numbers are somewhat skewed due to many systems (i.e. drives) being relatively recently installed, it is still very impressive.

For our setup we plan to use multiple full boxes, each connected to a separate NFS server. Each server has multiple FC host adapters, so if a server crashes, the corresponding box can be connected to one of the other servers.

We will also use rsync to mirror all data across the country to a secondary site.

Terje

Sounds like Google is trying that out.

There's nothing that exotic about this. The military builds racks of electronics into shipping containers all the time. It's mostly a cable management and maintenance access problem. You have to be able to do everything from the front of the rack, which requires some design work but isn't rocket science.

I ran accross this a while back at linuxdevices it is supposed to scale to Petabytes and is the main technology used for the Internet Archive.

Capricorn Technologies Petabox
http://www.capricorn-tech.com/

Linux Devices Review
http://linuxdevices.com/news/NS2659179152.html

The folks at the Internet Archive have already done the hard work of figuring out how to create a petabyte storage system using commodity hardware. The system works so well they started a company to sell PetaBoxes to others. Why reinvent the wheel?

Capricorn Tech? They power the Internet Archive. "Capricorn Technologies was founded in 2004 and provides petabyte-class storage solutions for organizations worldwide. Capricorn's PetaBox technology grew out of a search for high density, low cost, low power storage systems for the world's largest data collections. Capricorn Technologies is proud to be a leader in the next data storage revolution."

archive.org made a petabox

http://www.archive.org/web/petabox.php

There is now a company that seems to make the same design:

http://www.capricorn-tech.com/products.html

I don't know what FS they use, but apprently it is redudent.

Howabout the PetaBox, used by the Internet Archive ?

I heard I love lucy original telecausts are just leavin our solar system.

Thank God. I've been wanting to get rid of her for *years*. Can she take Gilligan's Island and the Brady Bunch with her?
Okay, to stay on topic...just get a PetaBox

http://www.capricorn-tech.com/press_rel/pr050622.h tml

First off, this isn't quite an example of a company suddenly deciding to donate stuff to the Archive. As can be seen on their own website, Capricorn was spun off from the Archive on July 1, 2004. To a large extent, Capricorn exists for the specific purpose of providing storage to the Archive, and if that same storage can be sold to others so much the better.

Second, what about interconnects and performance? The product descriptions say nothing about SCSI or FC or other storage-oriented connectivity, so one must assume that the connection to these boxes is through a network. That would mean each node is an NFS server (or similar), serving up 1.6TB using a 1GHz C3 processor, a maximum of 1GB of memory (for caching etc.) and what appears to be a single GigE link. Can you say unbalanced? The Internet Archive might be the only system with an access pattern so sparse that the ratio between capacity and performance wouldn't be crippling. Don't try using one of these with any other kind of application if performance is a concern...and BTW they don't seem to say anything about high availability or other storage functionality (e.g. integrated backup or snapshots) either. Capricorn's big play seems to be power consumption, but there are other players that can beat them on density (e.g. Copan with 224TB per rack) and multitudes who can offer better performance/functionality. I hate to sound negative, but this is a product so specialized as to be uninteresting.

Disclaimer: I think I met some of the Copan guys once and they seemed cool enough, but there's no other relationship between me and them. That just happened to be the first name I thought of in this space.

It's right there under the pictures :

http://capricorn-tech.com/

The site is rather empty right now, but it seems this is the company that will market this petabyte machine... er... box... er... whatever the name is.