Slashdot Mirror

Coraid (ATAoE) is deployed on production networks that make your infrastructure look like a phone booth. We're talking multi-petabyte installations. You obviously have some (very limited) experience, and I certainly don't want to discount it, but you're not qualified to speak to solutions you haven't personally deployed successfully. Sorry.

Speaking of EtherDrives, it appears from Wikipedia that they don't connect using SMB, or anything that is TCP/IP based. They use another protocol called ATA over Ethernet.

Not sure about Coraid's or NetApp's line of products, though.

First of all for the purposes mentioned in the original post, a USB HDD is by far the fastest, cheapest and most reliable solution. Transfer rates of 500MB in as long as it takes to walk from one desk to another.

But if you are looking for a network based solution (and have $10k), I am surprised no one has mentioned CoRaid. CoRaid is a AoE SAN hardware company but they offer NAS solutions too (basically a 1u linux box connected to their SAN) - for about $8-9k ($6.6k+15 1TB drives) you can have a 15TB (unraided capacity) NAS with advertised throughput of 100MB/60MB (read/write).

-Em

http://www.coraid.com/

I've used several in my line of work (Linux Sysadmin), and have been very impressed with how easy they are to work with and how reliable they've been. Very Linux friendly; newer kernels support AoE natively. Might be out of your price range, but they'll do RAID 5 or 10. You buy the enclosure, and the drives separately.

Have you considered Coraid's offerings? It doesn't matter how you slice/dice this problem... 100 disks will be expensive to link up. It might serve you a bit better to group the drives by speed and size then you can build individual boxen using various commodity components. You'll just need some add-in cards that support ATA RAID.

We have a SR1521, and it seems to do its job pretty well. It provides lots (over 7TB) of cheap storage space to the network. It probably isn't as fast as some other solutions, but our application doesn't need it to be.

These are theones I found useful

http://www.etherboot.org/wiki/sanboot/debian_and_ubuntu
http://www.s-mart.net/aoe.txt
http://www.coraid.com/support/linux/contrib/vantuyl/aoeboot.txt

There's also the SLAX Aoe boot disk http://www.lbserver.org/aoe/

Here's my writeups

http://maht0x0r.blogspot.com/

I wonder if the Coraid ATA-over-Ethernet would be good enough? It ditches TCP/IP in favor of raw Ethernet frames so has much lower overhead than iSCSI and only major loss is no routing. http://www.coraid.com/

BTW, I read recently that where 4Gb FC really excels is in large block sequential transfers and that small random access transfers are actually better over gigabit iSCSI. Check it out: http://searchstorage.techtarget.com/columnItem/0,2 94698,sid5_gci1161824,00.html

Plus you really have to think about other bottlenecks. How many disks need to be striped to consistently saturate the bandwidth of 1Gb Ethernet? 10Gb ethernet? What about the bus that the host adapter/NIC is on? Precious few boxes have 4x PCIe and then what about CPU overhead, managing all this streaming data? Just food for thought...

What, exactly, is the point of ATA-over-Ethernet? Honestly it sounds like a stupid idea, whose only merit is being implemented by someone and getting a Slashdot post about it. (oh, and by supporting Linux first)

If you want a real external storage solution, there are plenty of better technologies... If you must use Ethernet for your interconnect, just use iSCSI (who cares what the drives are). Remember, you're not having drives talking naked over the wire to the server anyways most of the time.

I've got a coraid array that can saturate the host PCI bus running on ATA-over-Ethernet technology, which is faster & simpler than SCSI-over-IP. Performance comparable to my giant expensive fiber channel SAN at a tiny fraction of the cost.

These guys are behind the technology: http://www.coraid.com/

If you don't like Open Source, you won't like it yet. Wait a few years and there will be a version you'll like, the economics of it are compelling. But right now you need to be able to write your own init scripts.

Why buy a $600 RAID controller when I can get the same performance from a $60 gigabit ethernet card?

If $/GB is a dominant factor, I would suggest Coraid's products. They have a pretty niffy technology which is dead simple and extensively leverages OSS. From my personal experience as a customer, I think they are a bunch of good folks as well. They also seem to constantly be wringing more and more performance our of their systems. Anyway, something to explore if I were you.

Here are some guys my friend was looking at for a storage solution. Basically they just ethernet-ify as many hard drives as you want. How you configure them is up to you. It's a bit expensive, but it's incredibly simple and flexible.
http://www.coraid.com/

Yeah, I was thinking his problem sounded like a good choice might be an ATA over Ethernet-type solution, like Coraid.

If I were tasked with this same job this is the solution I would use:

http://coraid.com/

Basically, you add your own disks, and have up to several terabytes of RAID storage. The best part is that teh RAID and all the complicated stuff is ghandled by the drive unit, to the OS it just looks like one huge drive.

You can add a NAS (SAMBA/NFS) server (or roll your own), to make accessing the drive from Windows / Mac even easier.

I don't have one of these myself, but have been drooling for a while...

-Ms2k

http://www.coraid.com/products.htm

I haven't used it, but it caught my eye a while back and looks promising. 500GB per disk, 15 disks per 3U shelf, and up to 65,536 shelves per network means it's expandable from 7.5 TB for one shelf up to (theoretically) 480 PB or so.

I've been working with a guy helping him setup his home theater system. Right now he's upto 9 TB of storage all running off of external firewire drives (http://www.zipzoomfly.com/jsp/ProductDetail.jsp?P roductCode=102670-1 1TB each drive). What I really want to do is get him setup with some sort of easily upgradeable system that is redundent. Price is not the biggest issue, easy of use and ability to upgrade is the most important. Right now he's expanding at 1TB every 1.5 months. I was looking at http://coraid.com/, anyone have any other suggestions.?

Wouldn't something like Coraid's ATA-over-Ethernet based product EtherDrive product make more sense for building massive storage array like this?

Wouldn't something like Coraid's ATA-over-Ethernet based product EtherDrive product make more sense for building massive storage array like this?

Like the http://coraid.com/ boxes?

You could try installing a Coraid box, which has a 10 disk chassis for $2500. You can load any disk you want, and attach it over the network.

Over on Linuxdevices there's a intresting article on ATA over Ethernet which a startup, called CoRaid has made a protocol for as well as, developed affordable hardware and submitted a driver to be included in linux.
This is poor mans SAN - the future looks bright for storage geeks ;-)

Sorry, Daniel :) As Linux's network driver maintainer and author of a Serial ATA stack which goes through the Linux kernel SCSI layer... as well as being someone who reviewed ATA-over-ethernet ...

I can say that ethernet hardware and the Linux SCSI stack does not handle retranmits that are needed at the ethernet layer. HP's Graham Smith is precisely correct. As some other slashdotters pointed out, the HyperSCSI code includes logic to handle retransmits and failures -- as it must.

Jeff