Slashdot Mirror
IEEE1394-based Storage Area Network?
Hank asks: "I work for Hewlett-Packard and just recently installed my first SAN at a customer site. It was much fun, I was blown away by the ease of the storage device management and the allocation of storage space across the systems. Being a professional environment, it was high-available, ran over FibreChannel through switched fabric, and cost upwards of US$250k -- not really affordable for most households. Roughly at the same time I started looking at IEEE 1394 cards for some video-editing, and an idea came up: Would it be possible to build a lowcost SAN based on Firewire cards, hubs and devices? How would storage device mgmt look like (the (de-)allocating of LUNs / slices / partitions)? What about support of multiple OS's on the SAN? How about this: would it be possible to create a Linux-based disk-array with an IEEE1394 interface (Old P200, crammed with disks, software RAID, lots of RAM for caching, Firewire interface, looking/acting like a single disk to the outside world, storage device mgmt via web-frontend)?"
A cool thing to do with a firewire SAN would be clustering, ala TruCluster, which presents a single filesystem across many machines (with kernel hacks to allow different files for different machines, eg hostname etc.).
I'm not sure if I have seen any PC-oriented FireWire SAN solutions though as FireWire hasn't really been something you would see in a lot of computers until recently.
I did find a couple when doing a search for "FireWire Network Storage":
http://www.adept.net.au/1394/nas.shtml
http://www.networkcomputing.com/1118/1118sp3.html (this is probably what I was thinking of)
http://www.turnover.com/news/mdm/firenas.html
USB 2.0 could also be used.
I think it would be cheaper.
You're missing the point: Using firewire you have the high performance of Firewire. Cat5 and you're back into ethernet space and packets.
Firewire supports sustained high bandwidth transfers between multiple drives and multiple computers.
I mean, if you don't need the performance of a SAN, then sure, use Cat5 and you have a fileserver.
But if you're looking for something between FCAL and Ethernet, then Firewire is likely a great midrange choice.
Yeah, and you guys panned the ipod too: http://apple.slashdot.org/article.pl?sid=01/10/23
Um, no. You've just described NAS - Network Attached Storage. Shared storage from NAS devices appears as NFS (or Samba, Mac, or whatever) and you can mount it on any client.
A SAN - Storage Area Network - is when you have lots of RAID storage being shared by several servers. Each server believes it is directly attached to a physical disk, when actually it's just getting one or more slices of the pooled RAID units.
You're confusing SAN (storage area network) with NAS (network-attached storage). Understandable mistake, but a mistake nonetheless.
i know that it isn't ieee 1394, but if you want SAN capability hosted by an off the shelf linux box, you may want to take a look at some early implementations of the draft iSCSI spec. basically, it'll let you present scsi devices over IP, giving you a SAN over any IP network (preventing you from dropping $$$ on fibre channel infrastructure).
--------
Granted, firewire is cheap, but it still has distance limitations. You also need those special cables.
I've thought about this some, and was thinking iSCSI as an option.
If performance is REALLY an issue, I suppose you could invest in GigE.
As for the SAN / NAS issues, what we are seeing in the industry is that people want / need both. Some vendors are starting to deliver devices that do both in one box. Raw disk for databases and such, and network file systems for other tasks.
Frankly for home systems, NAS should be just fine.
As others already pointed out, you are confusing the NAS and SAN concepts (but is it your fault? look at stuff like EMC Celerra HighRoad and then you'll be confused :-) )
:-)
Anyway,
Want to exploit 1394 (heck, we can finally call it Firewire!) to mount a disk? You just need a 1394 enclosure for your regular IDE disks. Example.
Want to exploit 1394 to access a network share via SMB/NFS? You can, with Ip-over-1394 (works on Apple, Linux, Win ME and XP. Not on 2000).
You just load the correct modules and it shows up like a network interface.
Just my 0.02.
I am not associated with the linked shop, I just happen to be a happy customer of theirs. Their Fire-I webcam is really cool (640x480x30fps) and it's amazing how well it can focus on extremely near objects, it's almost a microscope. I put it in contact with the screen and was able to focus on single pixels.... now that's a nice way to really study ClearType
Vacuum cleaners suck. Kings rule.
The upcoming serial ATA standard will give you better performance at a lower cost. A firewire drive is large, expensive, and consumes slightly more power. All you gain over current IDE technology is hot swap, and that will be solved with serial ATA.
But what you are really after are the tools to manage such a beast. The physical implementation shouldn't matter to the developers - all the software needs to know is that storage exists that the user needs to use, and how to read from and write to said storage. It shouldn't matter whether it's an IDE drive, a friewire, a usb, a scsi, a 1000 tape library, or any combination of storage devices which, IMHO, will be a great differentiating feature from commercial packages.
Yes, the free SAN package handles your old room size tape robot as well as this rack of serial ATA drives, and will treat them accordingly - near line storage in the tapes (semi archive), on line storage in the HD, and off line (off site) over the WAN link to the storage cluster at your other shop. If you need an extra terabyte just go to officemax and plug in a firewire drive until the tech comes out and adds more serial ata devices to your drive chain.
Of course, you could buy the SAN package available from x, or y, but you'll pay dearly for it, and you can't add storage to it yourself. Oh, and it only works with their hardware.
-Adam
Ummm... no.
A storage area network consists of storage devices directly attached to computers, via a star, bus, or fabric topology. Computer-to-computer networks are not storage area networks. They are simply LANs, local area networks.
What you described-- a server with filesystem sharing software-- is technically just a file server. A file server that comes with software and storage preconfigured, all in one box, is called a filer or a NAS appliance. Since what you described isn't preconfigured, it's just a file server.
Right now, the vast majority of storage area networks use fibre channel as the physical interconnect. Fibre channel was designed to be switchable, in fact; you can connect multiple storage devices-- devices, not servers; things like RAIDs and tape libraries-- to multiple servers via a switch, and those things can all talk to one another via the fabric.
...unless your hobies include rendering and editing Pixar-like shorts with your wife/girlfriend/dog/hamster working in tandem on one or more other workstations...
Come on, dude, don't be silly. Everybody knows hamsters are no good at animation.
I've been using a 12 foot FireWire cable for awhile no with no trouble.
Firewire2 will allow optical fiber as an alternative which has essentially an unlimited run lenght. Well, unlimited inside of an office, not kilometers.
I think that even Gigabit Ethernet will not provide the same performance as Firewire. Effectively you can get 500Mbps over GigE and thats' awfully close to current Firewire's 400Mbps-- - except that Firewire will sustain that transfer rate, and I'm not sure any ethernet can. But I could be wrong.
There are tradeoffs for both. If you want to link a cluster of machines and drives at high performance without going to FCAL, Firewire is a great way to go. IF you want to link a lot of computer to the storage, then Gigabit ethernet is probably the way to go-- but that ethernet could terminate in a server that has a Firewire network behind it linking a bunch of drives.
Yeah, and you guys panned the ipod too: http://apple.slashdot.org/article.pl?sid=01/10/23
Ok, it's not clear from your posting exactly what you want.
c t- links.html#IEEE1394
Do you want NAS?
That's Network Attached Storage. Currently almost entirely Ethernet based. You get a box with some disks and software, and it sits on the Ether looking like a fileserver, maybe just a CIFS server for Windows boxes, more likely both CIFS and NFS to support Windows and UNIX.
Do you want a SAN?
That's a Storage Area Network.
A bunch of disk boxes connected together with a switched Fibre Channel network. Servers connect by Fibre Channel directly into the network.
Do you want a NAShead on a SAN?
A NAS device acts as a front-end to the SAN, so you have an Ethernet file-sharing frontend onto a Fibre Channel storage network backend.
The problem with implementing any of these is they're about more than a transport medium. A NAS is more than Ethernet. A SAN is more than Fibre Channel. Those media mostly just pump the data around. It's a ton of software that handles the sharing of files.
So sure, you can string a bunch of disks and CD burners and whatnot together with FireWire. No problem. I do it myself. "FireWire" disks are almost entirely just an enclosure with a normal ATA disk inside and an ATA-to-FireWire bridge. Adds a small cost onto the price of a regular IDE drive, that's it. You can buy the enclosures yourself and do it quite cheaply.
However, the operating systems that you connect to the FireWire are going to have no freaking idea about filesharing. If you try to connect more than one host, it won't know what to do.
What you need is FireWire ***PLUS*** filesharing software.
Unibrain makes something they call FireNAS
http://www.unibrain.com/home/
That's about the closest thing in existence to what you describe.
If you're wanting to use IP-over-1394 (RFC 2734), be aware that Microsoft's stack is the main working one. The Linux stack is in beta and Apple has no plans to implement IP-over-FireWire at all.
You can find more info on IEEE-1394 at
http://www.cs.dal.ca/~akerman/gradproject/proje
Also check out the Linux 1394 project
http://linux1394.sourceforge.net/
Just one point -- Serial ATA is going to run in parallel with vanilla ATA for a while, while vendors skim money off high-end users. It's not going to be "SCSI at ATA prices", at least not right away.
May we never see th
As others have pointed out, what you're talking about is a fileserver, not a SAN.
However, interest in cheap SANs is rising, and I suspect it won't be long before a couple of projects start up to build these, then they get polished, then corporate types get interested in the big cost savings, and they start using these. It'd be particularly cool if Linux beat Windows to the gun here.
Before you scoff, remember that that's what happened with the advent of clustering cheap PCs -- the custom supercomputer is nearly a dead beast now.
There are enormous profits on SANs, so an open-source project could do wonders here.
May we never see th
The linked example is quite expensive. It is better to buy an empty firewire enclosure and put a 120GB WD drive in it.
Actually, Gigabit Ethernet should sustain something around 400Mbits/sec just like Firewire I. The difference is that next year we're still going to have Gigabit Ethernet but we're much more likely to have 800Mbits/sec sustained Firewire transfers. For some use cases, that's going to be a key feature.
11*43+456^2
Or I could have 10Gig ethernet and wait a half decade for you to catch up. Well not me personally since it costs well over $1,000/port. But I will use it for the backbone of my iSCSI SAN.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
AFAIK there is no built-in support in MacOS for IP-over-1394.
Drop what you're doing and go to your local book monger. Go get the ORA book "Using SANs and NAS". Read the descriptions of each.
Then come back here and ask that question without laughing hilariously.
I hadn't kept up with the 802.3ae working group. You're correct there. However on pricing, try $80k per port.
Ugh!
I would expect Firewire-2 to be somewhere in the sub $1 per port range. At that price differential, I would expect Firewire to win out for quite some time.
I thought CDMA was what caused most of the drop-off in efficiency. How are you going to stop that?
And more importantly, with ethernet, nobody makes anything like Hubzilla.
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
That's not what he was talking about. He was talking about using NFS or another IP-based networked filesystem. That's very different from what you're talking about.
And what you're talking about isn't such a great idea, either-- no offense. Leaving aside for a moment the technical challenges-- how do you turn a Linux system into a FireWire target, anyway?-- there would he serious cache coherency issues. How would you remotely invalidate a filesystem buffer cache, or a cached inode?
This issue is far more complex than you think it is.
How is it handled in a SAN normal envorment? Do it the same way.
Now you're starting to understand. Shared-access SANs are highly proprietary things requiring complex multi-node-aware filesystems, like Centravision or CXFS. It is not an easy thing. In most situations, it simply doesn't work at all.
Big SAN arrays have internal cache so I think they like for the OSes to do as little cacheing as possible.
First of all, there's no such thing as a "SAN array." There are disk arrays and storage systems that can be used on a SAN, but there's nothing special about them. They're just RAIDs, essentially, albeit sometimes with a few more bells and whistles.
And as for the caching thing, every operating system uses cached I/O for practically everything. (Direct, or unbuffered, I/O can be used in some situations where the data can be handled more efficiently by the application than by the OS; these situations are rare.) So let's say you and I are hooked up to the same hard drive. I open a file. Then you open the same file. I seek into the file and start reading. Let's say I read 1 MB of data into memory. Then you seek into the file and start writing. You write over the same blocks that I just read. I have no way of knowing this, of course, so I just keep doing what I'm doing, oblivious to the fact that the data I'm caching is out of date.
It gets worse. Say I decide to unlink the file... while you're in the middle of writing it. If we were talking about two applications on the same computer, I'd get an error back from the OS saying that you can't unlink an open file. (Or something, depending on the environment.) But since we're talking about two programs running on two different computers, I get no such error. Can't get one, in fact, unless my OS is keeping track of which files are currently open on your computer, and vice versa. Suddenly a normal filesystem won't work any more. We need something new, like CXFS or Centravision.
By this time, of course, we've given up on the whole damned thing and gone back to network-attached storage with Gigabit Ethernet interconnect. The last straw was the fact that my reading a file and your reading a file sent the shared drive into conniptions as the heads skittered all over the place. Disk contention is a bitch.
Shared-access SANs are incredibly complex. And, in general, they suck.
Dude, with all respect, you're not hearing me. It is not possible to lock files at the application layer with a shared-access SAN. You have two independent application on two separate computers trying to access the same filesystem. No synchronization mechanism exists between them... unless you're planning to write one, which means we're back in special filesystem land. That would be an absurd amount of work for what you're actually trying to accomplish.
As for NFS-style locking, there is no server on which to run the lock daemon. If there were a server, we'd be talking about NAS instead of a SAN, which is a different thing altogether. With a SAN, there's nothing but computers and disks, and the disks are not smart. You can't do file locking at the disk level. You have to have a mechanism through which the two computers can communicate with each other directly... which puts us back into special filesystem territory yet again.
I'm not quite sure why, but it's clear that we're not communicating well here. What can I say to make this more clear to you?
Ok, let's take your figures, instead of $80,000:$1 it's $12000:$1. Somehow I think that 10GB ethernet is going to stay out of the disk array field for quite some time while Firewire has realistic possibilities in the relatively near future (ie as soon as somebody releases the necessary drivers).
First of all, lock files aren't going to cut it. All we have to do is come up with a very simple scenario. You open up a file on the disk for reading, and start reading data out of it. You read a chunk, do something, then read another chunk.
Meanwhile, I come along and unlink the file.
In a single-access system, this isn't a problem. If I unlink the file while you've got it open, nothing really happens until you close it. After you close it, the file disappears and its space is reclaimed. This is because the kernel keeps track of who's got which files open, and prevents one process from pulling the rug-- so to speak-- out from under another process.
(This is also the source of the age-old trick of opening a file and immediately unlinking it. It's a good way of handling the automatic reclamation of temporary files.)
But if we've got two systems, each with a totally independent kernel, talking to the same disk, the problem suddenly gets a lot harder. When I unlink the file, my kernel will look at its tables and conclude that nobody else has the file open, so it will immediately try to reclaim the blocks on the disk. This will send your application, which is actively trying to read from the file, into shitfits.
No amount of lock files would save you in this situation. All I did was type "rm somefile." The rm command doesn't check for the presence of a lock file. Unless you want to rewrite all the file utilities, lock files aren't going to do the job.
(And extended attributes, of course, aren't portable. They're also tied to the inode, which raises its own set of problems.)
I just think you're grossly underestimating the complexity of this issue. If all you want is to share files across a network, then just use NAS. It'll work better and your life will be easier.
What you're talking about is absolutely possible. More than that, it's a key feature of fibre channel technology. And, in my sincerely humble opinion, it's really the only effective way to do a SAN: connect all the computers and all the storage devices to a big fabric, then give computers exclusive read-write or non-exclusive read-only access to each LUN. Shared access is hardly ever worth the trouble that goes into setting it up.
Your HP solution sounds pretty much like IRIS FailSafe, from SGI. The storage device is shared between two (or more) hosts, but only one host has access to it at any given time. When that host fails, the other host automatically mounts the storage device and takes up the failed node's responsibilities. Shared storage is a requirement for that kind of configuration; in the old days, we used to do it with SCSI.
I have no idea whether FireWire can support multiple hosts on a chain. Obviously it can handle more than one target, but I don't know about more than one initiator.
Also, the "LUN security" feature you talked about is sometimes called "LUN masking" or "LUN mapping," depending on how you do it. You can map a LUN to a switch port-- that's LUN mapping. Or you can set it up so that a port specifically doesn't see certain LUNs; that's LUN masking. I haven't worked with fibre channel on Windows in an age, but back when I did we had to use LUN mapping/LUN masking. At that time, the Windows host would try to send a device reset command to every LUN when it initialized its HBA. This was, of course, a very bad thing. So you had to set up your switch so that the Windows machines only saw the LUNs that they would be mounting. It was a little bit of a pain in the rear to set up, especially if the switch only supported LUN masking and not LUN mapping. If you add LUNs to your fabric, you have to go in and adjust all of your masks to mask the new LUNs out. On the other hand, if you use LUN mapping, the mapped ports will just ignore the new LUNs unless you specifically tell them not to.