Slashdot Mirror
"iSCSI killer" Native in Linux
jar writes "First came Fibre Channel, then iSCSI. Now, for the increasingly popular idea of using a network to connect storage to servers, there's a third option called ATA over Ethernet (AoE). Upstart Linux developer and kernel contributor Coraid could use AoE shake up networked storage with a significantly less expensive way to do storage -- under $1 per Gigabyte. Linux Journal also has a full description of how AoE works." Note that the LJ article is from last year; the news story is more recent.
I didn't know Age of Empires can do network storage! WTG Microsoft!
MidnightBSD: The BSD for Everyone
Some significant caveats mean that not everyone is so keen on the technology. For a start, it's a specification from Coraid, not an industry standard. Its networking abilities are limited. And its detractors include storage heavyweights such as Hewlett-Packard and Network Appliance.
So will this ever develop into a real standard or will it remain the sole domain of one company? I do not know if I want to invest time and money into it if the latter is true. From a comp sci point of view this is a great approach to networked storage. It uses what people already have to make storage reletively cheap. I am going to wait to see where this technology goes. Maybe it will blossom and become a serious contender.
Information wants a fueled airplane waiting at the hangar and no one gets hurt.
Why does it seem like everything open-source is a proprietary "killer"?
Having said that, this looks pretty neat. It will probably be more widely accepted, being open source (it is, right?), so it can be ported easily. Features will grow quickly, and other OSS advantages and so forth.
However, why is this better than NFS or Samba/Windows shares? Is it faster? It seems like AoE is offloading more of the low-level stuff to the client. Is this a good thing? It doesn't seem like one...
I have developed a truly marvelous proof of this comment, which this signature is too narrow to contain.
I guess I don't really see how it's cheaper that iSCSI? Sure, there's less overhead from the lack of TCP/IP, so you may not need as massive a network to drive it equally. But I've been under the understanding that iSCSI doesn't require SCSI drives, so you could build an iSCSI target out of the same machine/drives as an AoE host, correct? For some applications, I think the lack of TCP/IP might be a benefit - less opportunity to hack. (Then again, I'd expect anybody deploying something like this or iSCSI would drop the few extra $$$ to build a parallel network that transports just storage.)
This
Not sure if I follow this. Harddrives are well under $1/GB. If you buy several 400 GB drives and just connect them in an old PC thats on the network, aren't you accomplishing the same thing? I have a terraserver at home and it cost http://religiousfreaks.com/
People often forget there is a considerable difference in the reliability of ATA drives versus SCSI. If you are going to use some sort of ATA based SAN be prepared for disk failures much sooner than if they were SCSI.
UNIX/Linux Consulting
Everything over Ethernet!
Media that can be recorded and distributed can be recorded and distributed.
-kfg
TFA isn't responding, so maybe I'm missing something but how does this new protocol actually result in cheaper costs per GB? It's already possible to get an iSCSI SAN which uses SATA drives, and one of the major cost differences is the type of drive. What else is new here?
I like the look of this technology. The great thing it has going for it is that most of the non-hard-disk infrastructure (switches and cabling) leverages the tremendous investment in ethernet. That is great.
The thing that needs work, in my view, is that the bit that links the disks and the rest isn't cheap enough. In fact what would be awesome here is if, say, Seagate provided disks with native ATAoE connectors built-in. They might have to buy Coraid for that to happen.
In case anyone thinks I'm out of my mind here, don't forget that disks can already be had with ATA interface, SCSI interface, FCAL interface, SATA, SAS - that's five and there are probably more. Yes they might be a bit more expensive, but if they come in under the combined price of "regular ATA disk" + Coraid ATAoE disk adapter then you'd come out ahead. Someone like Seagate would, I think, have the industry-wide clout and respect to succeed in making this an open standard. Something that will be a challenge for Coraid for a long time (I have nothing against them, btw, they are friendly and their mailing list didn't spam me when I signed up).
When I was on the OpenSolaris pilot project I tried to get people interested in using this with Solaris. I think it might be great for ZFS, for example. At that point the real storage wizards were more interested in iSCSI, but I respectfully disagree, OpenSolaris + ZFS + cheap storage = awesome file server. Emphasis on the cheap. As Sun people will admit, their previous attempts at RAID were more like RAVED (Redundant Array of Very Expensive Disk). Coraid does have a Solaris driver, so this is definitely feasible.
In the context of using this in low-cost environments with Linux I can hardly see how this could kill iSCSI. Last week I implemented an iSCSI setup for about 500 euros (target serves out 500GB disk space for non-critical backup) using standard components, FC5, iSCSI Enterprise Target and Microsoft iSCSI Initiator.
Works great and is a lot (>10x) faster than the about similarly priced NAS device that was used for the same task before.
iSCSI is a protocol. ATA disks are a physical medium. They work together, and you get the benefits of SCSI commands with the price of ATA disks. Just because iSCSI is the protocol does NOT mean that you need to use SCSI disks. You might even be talking to a RAID of ATA disks and not know it.
So, why would you need AoE? It's already cheap, and been for sale for some time.
Is it possible to boot WindowsXP via AoE or iSCSI? I want a diskless WindowsXP box.
1) Complexity for RAID and volume management is not centralized and is pushed to individual hosts. One of the main benefits of SAN technology is that you can just carve out storage from a single interface and assign it to a server and the server simply sees it as a block device. With AoE each drive is addressed separately by the server, which means it is up to the server (and server admin) to figure out how to handle distributing over multiple drives, handle drive failures, and expanding volumes. This is huge.
2) It is not a standard and is only really supported by one vendor. This may change in the future but it is significant right now. It is registered with the IEEE but that hardly makes it a peer-reviewed standard with input/improvements from many experts.
3) No boot from SAN. Until someone makes some sort of mini bootstrap system on a CD or a hardware card implementation of AoE that can be addressed as a block device admins will be unable to host the root filesystem and/or C: drive on an AoE SAN
4) No multipath (that I can see). Perhaps I misunderstand this, but it seems like there is no way to do multipath IO with this system. That is, all the drives are single-connected to a network. If that network switch goes down, all drives on that network are inaccessible.
So AoE looks like a neat technology for pushing drives out of the box and potentially sharing them among hosts, but there is no intelligence there. It is just dumb block addressable storage with no added availability or management, and therefore is far from being an iSCSI or FC killer.
Ok, so the Coraid people are selling their ATA over Ethernet 15 slot version for $3,995.00. That's apparently around EUR 3133. I can get something proven iSCSI based from Promise here in Germany for 4.499,- (a Promise M500i). Ok, that is almost 50 percent more expensive, but the iSCSI solution is supposed to work under all operating systems (Linux, *BSD, Windows, etc.) more or less out of the box, while for AoE you will have to buy drivers for Windows, and has generally worse support for other operating systems.
Now, suppose you will really use this baby and you want to have *lots* of storage.
So you buy 15 SATA drives, like say Seagate ST3750640NS for EUR 444 each. Now the difference between AoE and iSCSI becomes less:
AoE solution: EUR 9793
iSCSI solution: EUR 11159
Now the iSCSI solution is only 14 % more expensive.
Now it would be clear for me to go for the "safe" path of something proven and widely supported like iSCSI instead of AoE. The infrastructe you need will be the same anyway (Gigabit Ethernet, Gigabit ethernet switch).
reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b')))
From reading the news article, it seems that they are selling $3,995 ATA -> ethernet converters (disk sold seperately). Each box will hold 15 drives and offer a simple raid controller inside. It still has the same performance issues of iSCSI (a bit lower overhead, but not by much).
I dont see what the point is other than the fact that they are offering yet another transport protocol. Given that one can install iSCSI target software on linux/solaris/windows... whats the point? Anybody who read the article on Sun's 4500 (48 500G drives w/ a dual proc opteron) know that you can cheaply assemble a box with a ton of drives, slap the iSCSI target software on it and call it networked storage. Redundancy sold seperately.
I've seen FC -> SATA arrays that run around $10K for 6TB. And thats for a system with active/active controllers, mirrored cache and 4Gb FC interfaces. And these systems had the ability to expand to 120 drives on the same controllers (final price: around $1 per GB). I think that FC has a bad rep for being expensive due to companies like EMC and HP. If you know what to look for, you can actually get the stuff for a relatively low cost. Given the performance boost compared to iSCSI, I'm going to stick with FC for the time being (I will still pay for the EMC kit when it makes sense from a support/reliability standpoint, but I also mix in things from companies like NexSAN when cost is more important)
You can use Ethernet-based multipath IO, a lot of switches can be stacked to provide redundancy (and load-ballancing).
AoE is a COOL thing exactly because it's a 'dumb' technology. You can buy a switch, a bunch of disk drives and AoE adapters, a small Linux PC - and your storage system is ready. There is a lot of existing RAID manipulation and monitoring tools for Linux, so RAID configuration is not a problem.
You also can boot from SAN, it's not a problem. Just add required modules and configs to initrd and place it on a USB drive.
So. Coraid has not, in a whole year, explained why iSCSI is somehow more expensive (disks + Linux kernel + network.. all the same) than their ATAoE implementation.
They'll give excuses about the cost of iSCSI hardware offload.. but you don't need that. ATAoE is all software anyway it's just a protocol over ethernet, rather than layered on top of TCP/IP.
What is wrong with using TCP/IP - which is already standard and reliable? Nothing. We know TCP/IP provides certain things for us.. resilience (through retransmits), and routing, are a good couple, and what about QoS?
ATAoE needs to be all the same network, close together, they're reimplemented the resilience, you can't use inbuilt common TCP checksum, segmentation and other offloads in major ethernet chipsets because they're a layer too low for it.
No point in it. Just trying to gain a niche. They could have implemented products around iSCSI, gotten the same performance with the same features, for the same price. Bunkum!
I prefer the hardwired Ethernet on Ethernet action myself.
The inter-specification is my favorite, I like with a male Cat 6 cable in a female Cat 5 plug....
Next week, Horatio and the team are called in to investigate when a relatively unknown l33t hax0r is murdered at a hip Miami internet cafe. It turns out he was an investigative aide working for Comcast and was monitoring bandwidth usage. As Horatio interviews the cafe's owner, another patron sneaks out of the cafe and crashes his RAID while installing Debian. However, the case gets more complicated when they discover another murdered victim inside the server.
It may be cool, but it is WAY too expensive. 4000 dollars for a 15-disk box without disks, come on!
I am looking for an affordable storage box for my home network, but for this kind of money I expect SMB/NFS functionality, not a dumb ATA interface over ethernet.
While I'm not especially interested in network storage, and I know very little about SANs and AoE, I still thought I'd give my input.
1) The "server", or drive array, handles the RAID, and all space carving (LVM, EVMS). AoE tools then export block devices.
2) Yup, no argument there.
3) VMs can boot from AoE, unless you use RedHat in which case it's not stable.
4) Multipath ethernet (or bonding) can be done trivially at the kernel level on all connected devices. Both to double the throughput, or just increase the reliability.
This is perhaps a poor comparision, but AoE is somewhat akin to Linux, as a SAN is to MS Windows.
AoE is a very important part, but a kernel is just a kernel.
SAN is a whole system, complete with GUIs, and reports.
Will it kill iSCSI, who knows, I don't, but I don't care.
I'd be far more interested in AoE and iSCSI if I could buy a few bare bridge boards to retrofit some RAID cages I have now.
Nothing in the world is more dangerous than sincere ignorance and conscientious stupidity.
while I agree with you, the issue of multipath is moot. Any device with only one port has that port as a single point of failure. You solve that problem with two ports and redundant switches. It is no different for SAN's, iSCSI or FC.
SAN's management capability is also it's downfall. Expensive, complicated and vendor-specific.
I read some of the responses. First off, I still don't see how you can have multipath if the actual trays only have one port (which looks to be the case). Their EtherDrive units where a RAID card and drives sit in one unit and then the already-raided volumes are exported over AoE is a bit different, but at that point you could just make your own iSCSI target and be more compatible with existing mature iSCSI implementations (such as the software initiator in windows). /can/ be done on a linux server is missing my main point, which is: raid and volume management should not /have/ to be handled on the server in a SAN environment, it is one of the main selling points of SANs. A single place to manage your storage configuration, all servers just see a chunk of storage and don't have to worry about the details.
All this talk of how raid
I am not saying AoE is not useful in some settings or a neat technology, my main point is that it is not an iSCSI killer because it does not fill all the functions iSCSI does.
A wise man once told me there is a fine line between them.
ATA is a crappy protocol, even when local. It's only good for squeezing that last $0.03 out of the controller cost. Once you are using ethernet cables ($1) and links and PHYs on each end ($4 each), it makes a lot more sense to put some brains back in. Use SCSI. Heck, even ATAPI optical drives (the optical drive in your computer) uses ATAPI, which is SCSI in packetized ATA transfers.
Also, I'm a bit nervous about the packet CRC validation being done in the ethernet controller/layer itself. The problem is that if an ethernet switch between you and the storage device stores packs and forwards them (as all smart switches do), it may also chose to regenerate the CRC on the way. If it corrupts the packet internally and generates a new, valid CRC for the new, corrupt packet, you have undetected corruption. I'd be a bit nervous about that for my hard drive.
I do think using GigE is a smart way to attach hard drives to servers. I look at the back of an Apple XServe and see two GigE ports and a fibre channel card. Why can't one GigE port be used to attach to the network and one to the XServe RAID? Why do I need to get a multi hundred dollar card to attach to the XServe RAID when that GigE port is fast enough? It'd sure save a lot of cost, and hopefully reduce the price ot the end user.
Anyway. I'm pro GigE attachment, not sure I'm for this AoE.
http://lkml.org/lkml/2005/8/20/95
Agreed on your multipath statements, except that in this case, due to the topology, losing a switch means losing all drives connected via that switch. In most SANs (sometimes even iSCSI) you have multiple routes from the raid card, through different networks, to the servers.
As far as SAN's management capability being its downfall, I disagree. I find it isn't terribly complicated (much simpler/faster than the Linux LVM tools I've used) and it can even be done in a standard way by creating your own iSCSI target and using those very same LVM tools. Someone even commented on this story that they had done this. The point is that all the raid and volume administration is done in one place, and individual servers do not have to worry about that complexity.
For small time users, I think this will catch on.
But you can't ignore a lot of the issues that high end storage systems address that this doesn't.
- Reliability
- Availability
- Smooth failure/degredation (when weird shit happens, how well does the device handle it, while maintaining uptime)
- Mutli-path I/O
- Cache writing. ATA has a notorious problem that drives cache and write things at their own will. However it's not on a level that the operating system is aware of, which make for some potential disasters when you use stuff in a transactional environment.
- Performance
- Response time (a lot of people ignore this one)
- Throughput
- TCO
- How long do you spend fucking with a broken components?
- How much money do you spend on replacing broken components?
For home users and environments where the admin really has nothing but time, yeah, pretty good setup. But for enterprise environments where the company you're working for cannot afford for its storage setup to have high downtime, no way. Enterprise storage systems are still leagues ahead of home brew software solutions because they throw lots of money at R&D, not just time and community testing.
Got to love a good British parody... Age of Umpires :-)
If I'm reading the Wikipedia AoE article right... AoE is a L2 protocol that can not cross routers. That would immediately rule out the office I work in, in which floors and the data centre are on separate TCP/IP subnets. Small offices only, then?
But, as noted above, if they are claiming that they avoid the cost of ToE NICs for iSCSI, that's a spurious claim, since they are an optional performance enhancer, not a requirement for iSCSI. I've seen surprisingly decent performance without them, with the HP EVA iSCSI bolt-on from QLogic and ordinary Broadcom server NICs.
(this is not a
Does anyone here have experience with vblade? Is my understanding correct?
Thanks.
Novel theory: Modern Man evolved from psychopath
This thread in the iSCSI Killer story is ultimate proof that teenagers in their parent's basements all around the world have taken over Slashdot. In the days of yore, there would have been a lot of loud rejoicing at ATA over Ethernet. Today, nothing but a bunch of lame jokes based on gaming by high school drop outs. Yes, the days of Slashdot have come and gone. There is no hope.
-"...bad old ideas look confusingly fresh when they are packaged as technology" - Jaron Lanier (Digital Maoism on Edge.o
It may be cool, but it is WAY too expensive. 4000 dollars for a 15-disk box without disks, come on!
Have you prices rack-mountable boxes with space, airflow and power for that many drives? They cost clost to that much even without the AoE adapters.
I am looking for an affordable storage box for my home network, but for this kind of money I expect SMB/NFS functionality, not a dumb ATA interface over ethernet.
Coraid's stuff is obviously not for home use. For home use, use an old PC filled with disks. Add more PCs filled with disks as needed. Because AoE is dumb, you'll have to figure out how to set up RAID and volume management appropriate to your needs. OTOH, you can set it up so it does exactly what you need. Or, if you want to spend a little more money, do a little less work and have a little less flexibility, buy one of the standalone storage boxes that provide CIFS or NFS services.
Personally, I have a home file server that is full of disks -- it actually has two more disks than places to mount them, so those two are sitting on the floor of the case. I'm thinking seriously about taking an old 500Mhz K6 I have sitting around and moving some drives to it (along with the ATA133 controller card they're connected to), then exporting them to the file server via AoE or iSCSI. I think I have another Gig-E card lying around.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
Yeah, instead of mildly amusing jokes about video games, there would have been a dozen poorly disguised goatse.cx links, some obscene ASCII art and references to Natalie Portman and hot grits. Anyone who misses the "good old days" of slashdot wasn't really there.
0 1 - just my two bits
Seriously, I don't think it needs any help whatsoever.
sic transit gloria mundi
yes, removing routing takes away that option but a fully redundant connection doesn't really need that. what it needs is fully independant paths and multiple routes aren't needed there.
The holy grail of SAN's is unified SAN management and the number one complaint of SAN's is that management is too complicated and causes vendor lockin. No doubt that a centralized place for storage management is frequently desirable. If it weren't SANs would never exist in the first place.
I believe when they say "server", they mean the server that is hosting the drives and serving them to the network. So that DOES centralize all the management. You just serve out the logical volumes using AoE and you're all set.
-"...bad old ideas look confusingly fresh when they are packaged as technology" - Jaron Lanier (Digital Maoism on Edge.o
AoE rocks. It is very easy to set up, way simpler than iSCSI or fibrechannel or any other SAN technology I have used. And it enabled us to have many more options for high availability or clustered filesystems (which we are not yet using but I have been following the progress of GFS and Lustre, learning towards Lustre). We did not buy the Coraid stuff but instead used vblade on our own disk machines. A disk node in our cluster has 4 300G SATA disks which we RAID 5, 512M RAM, and the cheapest CPU Intel currently makes. We have dual core Opterons with 4G of RAM each with no internal disk. They PXE boot and then mount root straight off the AoE. Then we run Xen on the Opteron boxes. This is the killer setup. We can migrate xen domains avoiding downtime for hardware maintenance and if a machines dies we can instantly restart it on another machine because it all runs off the AoE SAN.
So far I am very pleased. Just make sure you get hardware that can do jumbo frames as this will increase your performance by 50%.
Me too. I need ~5Tb storage for my film and music collection, but I can't find a good solution.
Plain PC with Linux doesn't suit me because there are only 3 or 4 ATA controllers on a typical motherboard. Additional RAID controllers help but not much.
AoE solution allows to install literally dozens of cheap disks in a cheap gigabit switch.
Price on AoE should go down - electronics in AoE controller should cost no more than $20-$30.
Why can't you just use an NBD server and client and setup software raid1?
Why read the article when I can just make up a snap judgement?
AoE just plugs into the network switch. SaMBa requires an installation, configuration, virusation, god-knows-what-else.
A Fortune 500 company doesn't care, but 2 million small businesses and nonprofits just want plug and use. This is a BIG deal for us little guys. I hope it makes it to market, it'll be even better than the Linksys Slug.
Please provide a vendor that offers this. How did you like their customer service also?
has been reported from Suntang (google cache), however it can be difficult to find out information from them as their website is almost always down and they never reply to email: office@suntang.com. They primarly install Linux based thin client systems around China, and have moved development over to supporting Windows and concepts like VDSA (the Virtual Disk System Architecture) and Virtual Hard Drive Technology.
:(
They manufacture some great looking thin clients.
perhaps an interesting idea, but just because I can build a computer out of old, recycled clock parts doesn't mean it is going to become my server. Also, iSCSI adoption has increased something like 40% this year. Windows support for iSCSI will improve dramatically with the next revision, and iSCSI costs are only going to decrease.
Also, consider management of one of these AoE boxes. What sort of tools are out there to simplify provisioning, deployment, snapshots and backup, etc. In order for this to go anyplace but the basement of 'the IT guy at work' a whole lot more stuff will be required. Oh yeah, and that probably isn't going to happen with 1 vendor controlling the market.
AoE is not fully baked yet. Put it back in the oven and let me know when the timer goes off.
we bought coraid devices, and they are AoE is much simpler (read: cheaper) than iSCSI. when using jumbo frame switches/cards, we were able to get transfer rates very near theoritical limits on gigibit links, something I have never seen on iSCSI or fc for that matter.
the only thing that bothers me about AoE is there is only a single vendor supporting it at the moment. other than that, it is great stuff. while it is not routable in the sense ip is routable, you can do creative things with ethernet switches and vlan basically giving san like functionality at a fraction of the cost. no longer do you have to keep dual fc/cat6 infrastructure in your server farm.
it's cheap, and if/when it supports bonding lines, well beat fc in performance (comparing two gigabit fc vs/ bonded gigabit ethernet).
merlin
.. is that the specs for the AOE protocol is so simple that you could probably make your own little testbed with some cheap microcontrollers and some ram for caching. It wouldn't be very fast, but it would be one cool project.
Doolittle :
Bomb no.20 : To explode of course.
I do not need hot-swap capability. When I want to add or replace a drive I can just powerdown the unit.
However, all solutions I have looked at (the Coraid included) have this useless (for me) feature.
Currently I am looking at a 3E high 19" cabinet I have, to construct some disk mounting hardware (horizontal rails across top and bottom) and put a small board (ITX) in it. The thing can then run as a (Linux) server and export the disks as SMB or NFS instead of AoE, so they are directly accessible to my satellite receiver/recorder (Dreambox) as well. When going AoE I would have to do NFS->AoE in my normal PC.
I want the whole thing as low-power as it can be. So, lowpower board, stopping the disks when not in use, automatic fan control. Ideally, when everything is idle it should consume 15-25W.
I don't intend to use RAID because with the usual RAID schemes you always have to have all disks running. Separate volumes and some manual copying of important files to different places should do, in this case.
If I could go to Best Buy and get a 5.25" enclosure with an ethernet port on the back from Linksys or Netgear or their ilk, pop in my favorite SATA or IDE disk, and stick it on a private gigabit LAN this would be fantastic.
Right now the cost of entry discourages experimentation. Having to buy a $3,000+ chassis plus all the drives is going to require funding that I have to fight for. If I can implement a proof of concept for under $500, I don't even need my manager to sign off on the expense. I can just do it and then show him "hey, this works, and if we buy this $3,000+ chassis and a bunch of drives, I can give you 5TB for a fraction of what we're paying now".
This is how Linux caught on in the Enterprise. Start at the bottom, work your way up. It's a lot easier to encourage experimentation and tinkering if the cost of entry is under $100 for a single drive enclosure.
Personally, I'd love to play with something like this at home for my MythTV setup and if it works introduce it at the office as an alternative for low cost / low performance / low redundancy / high capacity storage (yes, there are some great applications for this if you accept its limitations).
AoE is a SAN technology and it is no different from other SAN technologies in this regard. AoE just provides a block device. Just like Fibrechannel or iSCSI or any other SAN technology. If you want RAID and volume management centralized instead of being pushed to individual hosts do what the Fibrechannel guys (and everyone else) does: Attach all of the disk to one host and use that host to aggregate it and manage it and export it to the rest of your clients. Even in the big Clariion disk systems from EMC and other SAN products they contain an internal PC which aggregates all of the disk for presentation to other hosts on the SAN.
It is a spec released to the public without strings attached. GPL'd implementations of both target and initiators exist. Lots of people have looked it over to the extent that it has even been included in the stable Linux kernel.
Uh...I have a number of machines in a rack right across the hall from me which do indeed boot from SAN. All you need to do is tftp an initrd containing the AoE driver and boot with that.
How is this any different from fibrechannel? IIRC fibrechannel disks have a single connection also. What most systems do is mirror or RAID them. What I have done is I have made a mirrored pair of AoE disks on different switches and use Linux's md driver to mirror them.
Fibrechannel or iSCSI by themselves are just dumb block addressable storage also. The management layer generally has little to do with the technology that implements the block layer.
Hey, thanks for the info. Looks like promise has the same enclosure with FC and plain SCSI ports. What are the advantages/disatvantages of iSCSI, FC, and plain SCSI? I am specifically wondering about M500i, M500f, and M500p? Seems like they all have the same features, but plain SCSI is faster.
___
If you think big enough, you'll never have to do it.
I'm pretty confident that you can prevent unintended data corruption. TCP/IP manages it so there's your proof of concept :)
Did you read the article?
The article explains how AoE is much better because it's much cheaper because you don't need to calculate all those checksums like TCP/IP does. It says just rely on the link layer checksums (note - they aren't checksums, they are CRCs).
I agree you can manage unintended data corruption at higher layers. iSCSI does this. And it's pretty much my entire point. That you should do so and AoE doesn't.
This protocol seems to cheap out in ways that don't make sense.
ATA is not a perfectly fine protocol for block storage. It works, true. But it's far from optimal. Its methods of handling error signalling is poor. Its methods of handling delayed error signalling is very poor.
I think the idea of using ATA and all its cruft to save a couple pennies on a big-tyme storage system doesn't make sense to me. If I can buy an optical drive for $30 that has SCSI (ATAPI) in it, I think the benefits of SCSI can be affordably brought to a large storage system too.
http://lkml.org/lkml/2005/8/20/95
iSCSI has a whole bunch of things going for it -- it allows sharing SAN storage over ethernet on the WAN, it lowers the cost of SAN clustering. It works on existing NAS storage that supports iSCSI, it works with SAN storage with a FC to ethernet switch like the Cisco MDS 9000
ATA over ethernet doesn't work on enterprise NAS or SAN storage. ATA over ethernet doesn't provide any redundany or fault tolerance without hardware RAID on storage or software RAID on the client.
ATA over ethernet also is not available for a wide variety of guest OS platforms, whereas iSCSI currently is available for many OSes.
Very good thinking otherwise.
It's supported in the linux kernel. I don't know about the bsd's or OSX, and there's a commercial driver for Windows.
As a point of contention, that is booting from USB and running the root filesystem over AoE, which isn't quite the same thing. You could also do a PXE, CD-ROM, Floppy, or HDD boot and use the AoE block device to the same result. Booting from a device means that the bootstrap is loaded from the device itself. Currently AoE can't do that and even if you created an HBA for AoE, it would still put the intelligence of where to load the bootstrap from in the client. The solution would be to add some sort of redundant, intelligent proxy to the mix (which you can actually do -- they sell one) but then you are no longer cost-comptitive against iSCSI and low-end FC.
So I saw AoE about a month ago and I started a graduate thesis project for analyzing and specifying different mail server architectures for my institution. The institution currently has 330Gig of mail storage and complains that this is too much usage when somebody complains their quota is too small.
I don't have the bucks to buy a terabyte of storage to prove them wrong but I do have a lab with 16 computers. So we recently ran the vblade servers on 10 of the machines and exported a 130Gig partition from each machine. Then we took one of the other machines to use as a file server. It builds a RAID6 /dev/md0 out of the 10 exported /dev/etherd/eX.Y devices and then exports /dev/md0 via NFS (currently). The result is a terabyte mail storage that can suffer the loss of any two vblade machines (loss of the single NFS server and central ethernet switch is still a problem.)
How well does it work? Well we just set it up so results aren't conclusive but bonnie results show the various disk operations to the /dev/md0 device is almost identical to a raw local partition. Nice.
An iSCSI killer? no, different purposes. But AoE is excellent and has it's place. A very good place.
I will never live for sake of another man, nor ask another man to live for mine.
If you're interested in an industry perspective, I'm the CTO at an iSCSI SAN provider. Marketing fluff has been left out, leaving my two cents on AoE as I understand the technology. TCP is a universally accepted transport that guarantees data delivery over any Quality of Service, along with many optimizations that switch and network accelerators use to optimize its performance on lossy networks. AoE will not work over lossy networks, therefore it's not a solution outside the data center (i.e. campus SANs or Remote Copy). iSCSI has a built-in security model with IPSEC and CHAP. AoE SANs must be secured at the switch level, which introduces more management complexity with having to setup VLANs or other features on the switch, which may also require the purchase of higher cost Ethernet switches. If the switch is compromised the storage is completely exposed. An encryption and authentication model is also missing. AoE doesn't support IP addressing & management. There is a compelling business case for IP convergence. IP offers one technology and management framework for the Internet, VOIP & iSCSI/Storage and the rest of the IT infrastructure. This being said, AoE isn't all bad but it's not an iSCSI killer. It could offer a cost effect storage solution for small businesses or home offices. John Spiers
I've been doing this kind of thing for a long time.
Using VMWare, I can set up a virtual IDE hard disk that is mapped to a file that resides on a network-mounted filesystem.
Once you've mounted all these disks, you could even provide an iSCSI interface to network clients.
Personally I'm waiting for a cheap USB style enclosure. What could really nail this in the home market is some kind of clustering filesystem for windows, allowing multiple hosts to read and write to cheap network attached storage.
09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
Yes, they were funny a lot of the time. My point was just that there was never a golden age where everyone was mature all the time and only posted on-topic, well-reasoned, polite comments.
0 1 - just my two bits
But IP is very little work for the average host - there's some initial handshaking to find out which network connector to use and what MAC address to put on the Ethernet packets, and if you're using a protocol stack there's a handoff from the Ethernet handler to the IP handler. Routers may have to do a lot of work deciding where to route packets, but hosts usually just slap a MAC address on each packet and hand it out the single Ethernet port. You're not going to waste much time, and in return you get several benefits
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
SATA hot swap is a lot cheaper than SCSI hot swap. The SATA drives are probably hot swappable. I've also seen ATA hot swap drive carriers that are fairly inexpensive. SATA hot swap is a lot simpler than SCSI on the software front - there is no need to suspend activity on the entire SCSI bus during removal/insertion. Just unmount the drive/partitions in question, and replace away.
...belong to them
'tis better to leverage the protocol across a cluster of machines.
Dedicate one or two disks and a GB port per blade and isolate it to a VLAN.
Use AoE to expose the disks as a remote aggregation target or for a cluster aware FS.
(I recommend partitioning them and using 60-70% for a cluster FS and the latter 30-40% for emulating networked tape drives for backups).
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
> If you really want a diskless system now
There exists a solution for diskless Windows XP stations. It is called "Venturcom BXP" and consists of server software and client drivers.
The server installs on WinXP or Win2K3. You also need DHCP, BOOTP/PXE and TFTP servers. BXP includes them if you don't have them already in your LAN.
The client drivers link to the NIC drivers and also include a tiny status tool and the disk-copy program.
After installation, you create a virtual disk on the server and assign it to a client. Once the client boots, this disk appears as additional driveletter. Using the disk-copy program, you copy your system partition over to the virtual disk. Then you change the server configuration to instruct the client to boot from the virtual disk.
At this point, the physical harddrive will appear as additional driveletter, while the virtual disk becomes C:\. You can remove the physical harddrive now if you want, or you can assign the windows pagefile to it for faster operation.
With 100mbit ethernet you can achieve about 8-9 MB/s virtual disk performance. However, ethernet has considerable latency, and a BXP'ed machine doesn't feel as snappy as a real one.
A nice extra feature is the possibility to "write-lock" the virtual disk. That is, all changes made to it can be stored in a separate file, which is deleted when the client is shutdown. Using this feature, the client always boots the same state. This is perfect for classrooms or webcafes where users may modify the configuration, delete system files, or infect the machine with trojans or viruses. All changes are magically undone when rebooting.
It is also possible to assign such a "locked" image to several clients at once (the virtual disk is accessed read-only, and each client gets an individual temporary write-cache for its session). Using this feature you can install and customize one box, and then boot the image on a dozen boxes. For this to work, it is necessary that all clients have identical hardware (down to the PCI card order!).
Other advantages are that the virtual disk is just a file. This provides for easier backup or versioning / branching. Using tools to snapshop the server partition, you can even backup the client while it's running.
You can also use BXP to test-install software. Games for example can't be tested in VMware (lack of virtual 3D hardware). With BXP you can, because only the harddrive is virtual - the box remains physical!
Marc
No, SCSI doesn't have 1000x the cruft of ATA.
ATA has at least 3 ways of even specifying a block. CHS, LBA and LBA48. SCSI has 1.
ATA has 10 ways of reading data, not counting the ATAPI (SCSI) ways and the Compact Flash ways.
READ DMA
READ DMA EXT
READ DMA QUEUED
READ DMA QUEUED EXT
READ MULTIPLE
READ MULTIPLE EXT
READ SECTORS
READ SECTORS EXT
READ VERIFY SECTORS
READ VERIFY SECTORS EXT
SCSI has 4.
Which has more cruft here?
So, if you've never had a problem dealing with ATA errors, let me ask: How do you deal with errors on ATA write-behinds? Do you even know how the drive signals them? If so, how do you detect them and handle them? I know it can be done, but it's wedged into ATA rather poorly.
ATA error handling isn't reimplemented with a new physical layer. SATA (for example) just serializes the data that was already sent over ATA. It still has all the DRQ bit stuff. Error handling is actually in the physical layer in ATA, but since they made it appear in the command registers, it has to be implemented at higher layers now and is not subject to redesign with new physical layers.
As to ATA being leaner, did I forget to mention it has 2.5x as many read commands as SCSI? 3.3x as many mandatory ones?
http://lkml.org/lkml/2005/8/20/95
Preserve Our Essence.