IEEE1394-based Storage Area Network?

Hank asks: "I work for Hewlett-Packard and just recently installed my first SAN at a customer site. It was much fun, I was blown away by the ease of the storage device management and the allocation of storage space across the systems. Being a professional environment, it was high-available, ran over FibreChannel through switched fabric, and cost upwards of US$250k -- not really affordable for most households. Roughly at the same time I started looking at IEEE 1394 cards for some video-editing, and an idea came up: Would it be possible to build a lowcost SAN based on Firewire cards, hubs and devices? How would storage device mgmt look like (the (de-)allocating of LUNs / slices / partitions)? What about support of multiple OS's on the SAN? How about this: would it be possible to create a Linux-based disk-array with an IEEE1394 interface (Old P200, crammed with disks, software RAID, lots of RAM for caching, Firewire interface, looking/acting like a single disk to the outside world, storage device mgmt via web-frontend)?"

Comments

Lose the buzzwords

[billcopc]

If you're setting up a single PC with lots of storage, then it's not a "storage area network", it's just a "plain old file server". The difference between the two is that SAN sounds more professional to PHB types, and it is generally seen as a turn-key solution for storage. You just plug in some Cat-5 and give it a password, TADA. I wouldn't be surprised to find out they're just linux boxen with custom software for storage management. That's the big Plus of SAN devices : easy to install and use. You don't need a linux admin to setup a Maxtor MaxAttach (sp?) rig, you just need any old NT/Novell twit who at least knows how many megs are in a gig.

Re:Lose the buzzwords

[BitGeek]

You're missing the point: Using firewire you have the high performance of Firewire. Cat5 and you're back into ethernet space and packets.

Firewire supports sustained high bandwidth transfers between multiple drives and multiple computers.

I mean, if you don't need the performance of a SAN, then sure, use Cat5 and you have a fileserver.

But if you're looking for something between FCAL and Ethernet, then Firewire is likely a great midrange choice.

Re:Lose the buzzwords

[rpresser]

Um, no. You've just described NAS - Network Attached Storage. Shared storage from NAS devices appears as NFS (or Samba, Mac, or whatever) and you can mount it on any client.

A SAN - Storage Area Network - is when you have lots of RAID storage being shared by several servers. Each server believes it is directly attached to a physical disk, when actually it's just getting one or more slices of the pooled RAID units.

Re:Lose the buzzwords

[foobar104]

You're confusing SAN (storage area network) with NAS (network-attached storage). Understandable mistake, but a mistake nonetheless.

Re:Lose the buzzwords

[walt-sjc]

Granted, firewire is cheap, but it still has distance limitations. You also need those special cables.

I've thought about this some, and was thinking iSCSI as an option.
If performance is REALLY an issue, I suppose you could invest in GigE.

As for the SAN / NAS issues, what we are seeing in the industry is that people want / need both. Some vendors are starting to deliver devices that do both in one box. Raw disk for databases and such, and network file systems for other tasks.

Frankly for home systems, NAS should be just fine.

Re:Lose the buzzwords

[Anonymous Coward]

umm..no. a SAN is kinda like several NFS mounted disks shared between servers. SAN doesnt have to be from RAIDed units or anything else..could be a JBOD.
in reality you can do it with a linux box with software RAID and IDE drives using NFS/CodaFS/AFS and LVM.

Re:Lose the buzzwords

[foobar104]

Ummm... no.

A storage area network consists of storage devices directly attached to computers, via a star, bus, or fabric topology. Computer-to-computer networks are not storage area networks. They are simply LANs, local area networks.

What you described-- a server with filesystem sharing software-- is technically just a file server. A file server that comes with software and storage preconfigured, all in one box, is called a filer or a NAS appliance. Since what you described isn't preconfigured, it's just a file server.

Right now, the vast majority of storage area networks use fibre channel as the physical interconnect. Fibre channel was designed to be switchable, in fact; you can connect multiple storage devices-- devices, not servers; things like RAIDs and tape libraries-- to multiple servers via a switch, and those things can all talk to one another via the fabric.

Re:Lose the buzzwords

[felddy]

SAN? Ok. I know where the area is in a LAN, and WAN. Are we talking about peripherals here? Is my KVM the middle of my MAN (Monitor Area Network)?

Please someone just try to sell me a SAN.

-Felddy

Re:Lose the buzzwords

[BitGeek]

I've been using a 12 foot FireWire cable for awhile no with no trouble.

Firewire2 will allow optical fiber as an alternative which has essentially an unlimited run lenght. Well, unlimited inside of an office, not kilometers.

I think that even Gigabit Ethernet will not provide the same performance as Firewire. Effectively you can get 500Mbps over GigE and thats' awfully close to current Firewire's 400Mbps-- - except that Firewire will sustain that transfer rate, and I'm not sure any ethernet can. But I could be wrong.

There are tradeoffs for both. If you want to link a cluster of machines and drives at high performance without going to FCAL, Firewire is a great way to go. IF you want to link a lot of computer to the storage, then Gigabit ethernet is probably the way to go-- but that ethernet could terminate in a server that has a Firewire network behind it linking a bunch of drives.

Re:Lose the buzzwords

[Anonymous Coward]

What is FCAL?

Re:Lose the buzzwords

[Anonymous Coward]

FibreChannel Arbitrated Loop - that's the cheapo way of connecting more than 2 FibreChannel devices through a FC Hub. The Hub will establish a loop (think of something like TokenRing probably) between all FC Host Bus Adaptors.

Re:Lose the buzzwords

[dbrutus]

Actually, Gigabit Ethernet should sustain something around 400Mbits/sec just like Firewire I. The difference is that next year we're still going to have Gigabit Ethernet but we're much more likely to have 800Mbits/sec sustained Firewire transfers. For some use cases, that's going to be a key feature.

Re:Lose the buzzwords

[afidel]

Or I could have 10Gig ethernet and wait a half decade for you to catch up. Well not me personally since it costs well over $1,000/port. But I will use it for the backbone of my iSCSI SAN.

Re:Lose the buzzwords

[Omega996]

ugh, and watch your bandwidth go down as your load goes up? no thanks...

Re:Lose the buzzwords

[AlecC]

Gbit Ethernet can sustain more than 500Mbit/sec, but usually doesn't get the chance to do so. The problem is running TCP/IP protocol over it. TCP/IP rins, over dodg links, the breadth of the world. In order to achieve this, it has things like internal checksums, time-to-live etc which are required to achieve this - but require significant intervention at the driver level to manage. And then you probably have timeout and retry systems to manage as well.

Firewire, Fibre Channel, Scsi etc. don't have this flexibility. They assume that the device you are talking to is is on this bus/ring and that messages will, give or take parity errors (which can be reported) get through reliably. Therefore they can have much ligher protocols. You can usee what, in Ethernet terms, is a MAC address not an IP address.

The firewire suggestion at the head is perfectly snesible. But if you were to go into the ethernet driver at the packet level, not at the OP level, I reckon you could get 90% of theoretical bandwidth out of an ethernet-based connection. We currently get over 500Mbit/sec using UDP connections. Drop a layer of protocol and it could be even faster.

Re:Lose the buzzwords

[dbrutus]

I hadn't kept up with the 802.3ae working group. You're correct there. However on pricing, try $80k per port.

Ugh!

I would expect Firewire-2 to be somewhere in the sub $1 per port range. At that price differential, I would expect Firewire to win out for quite some time.

Re:Lose the buzzwords

[dbrutus]

I thought CDMA was what caused most of the drop-off in efficiency. How are you going to stop that?

Re:Lose the buzzwords

[AlecC]

CDMA only causes a lot of slowdown when you have a lot of collisions - i.e. when you have a lot of small packets fighting for the bandwidth. Which is another of the consequences of TCP/IP - flow control packets etc., and obviously occurs on a "workstation" environ,ment with many independant systems talking to servers. In a SAN type network using direct Ethernet packets, you would notrmally send bulk data in large packets which would swamp the bus for a while, then back off. But, like a scsi bus, you would tend to get one logical transaction loading the net for a while, then backing off while it seeks to the next logical block. And SAN type applications tend to have a few disk users, rather than hundreds of light users.

Re:Lose the buzzwords

[afidel]

I could be wrong but I believe the 10Gb module has more then one port. Plus a module would only cost me about $12,000 because of internal discounts.

Re:Lose the buzzwords

[Sloppy]

And more importantly, with ethernet, nobody makes anything like Hubzilla.

Re:Lose the buzzwords

[1101z]

What he is talking about is SAN but using a linux box instead of embedded hardware. The linux box has a bunch of disk which it raids together and then acts as a big disk drive to the hosts on the other end of the firewire cable. The linux box would be a target mode firewire disk.

I want to do this as well, for a small network of video editing machines. But have not found any project beond to planning stage. It seem that anyone that needs to do SAN has a crap load of money to spend.

Re:Lose the buzzwords

[foobar104]

That's not what he was talking about. He was talking about using NFS or another IP-based networked filesystem. That's very different from what you're talking about.

And what you're talking about isn't such a great idea, either-- no offense. Leaving aside for a moment the technical challenges-- how do you turn a Linux system into a FireWire target, anyway?-- there would he serious cache coherency issues. How would you remotely invalidate a filesystem buffer cache, or a cached inode?

This issue is far more complex than you think it is.

Re:Lose the buzzwords

[hank4snow]

That's it - that's exactly what I am after!!!! Let me know if you get any further!!

Re:Lose the buzzwords

[1101z]

Actual it is what he is talking about see later post.

There is a standard for firewire disks sbp2. You implement that as the disk and not as the host.

> This issue is far more complex than you think it is.
Well I think that it is very complex that is why I have not done it yet and selling TB ide arrays with a firewire ports that can be used as a SAN.
> -- there would he serious cache coherency issues. How would you remotely invalidate a filesystem buffer cache, or a cached inode?
There are patches by oracle to allow linux system to share a common firewire disk. How is it handled in a SAN normal envorment? Do it the same way. Big SAN arrays have internal cache so I think they like for the OSes to do as little cacheing as possible.

Re:Lose the buzzwords

[foobar104]

How is it handled in a SAN normal envorment? Do it the same way.

Now you're starting to understand. Shared-access SANs are highly proprietary things requiring complex multi-node-aware filesystems, like Centravision or CXFS. It is not an easy thing. In most situations, it simply doesn't work at all.

Big SAN arrays have internal cache so I think they like for the OSes to do as little cacheing as possible.

First of all, there's no such thing as a "SAN array." There are disk arrays and storage systems that can be used on a SAN, but there's nothing special about them. They're just RAIDs, essentially, albeit sometimes with a few more bells and whistles.

And as for the caching thing, every operating system uses cached I/O for practically everything. (Direct, or unbuffered, I/O can be used in some situations where the data can be handled more efficiently by the application than by the OS; these situations are rare.) So let's say you and I are hooked up to the same hard drive. I open a file. Then you open the same file. I seek into the file and start reading. Let's say I read 1 MB of data into memory. Then you seek into the file and start writing. You write over the same blocks that I just read. I have no way of knowing this, of course, so I just keep doing what I'm doing, oblivious to the fact that the data I'm caching is out of date.

It gets worse. Say I decide to unlink the file... while you're in the middle of writing it. If we were talking about two applications on the same computer, I'd get an error back from the OS saying that you can't unlink an open file. (Or something, depending on the environment.) But since we're talking about two programs running on two different computers, I get no such error. Can't get one, in fact, unless my OS is keeping track of which files are currently open on your computer, and vice versa. Suddenly a normal filesystem won't work any more. We need something new, like CXFS or Centravision.

By this time, of course, we've given up on the whole damned thing and gone back to network-attached storage with Gigabit Ethernet interconnect. The last straw was the fact that my reading a file and your reading a file sent the shared drive into conniptions as the heads skittered all over the place. Disk contention is a bitch.

Shared-access SANs are incredibly complex. And, in general, they suck.

Re:Lose the buzzwords

[1101z]

All good points execpt as I said earlier I want to use it for video, which is something that SAN is already used for. Files are locked at the application layer. Or an NFS style lock system could be used on the target side. All files being available all the time is not important. I do not need a general solution.

Re:Lose the buzzwords

[foobar104]

Dude, with all respect, you're not hearing me. It is not possible to lock files at the application layer with a shared-access SAN. You have two independent application on two separate computers trying to access the same filesystem. No synchronization mechanism exists between them... unless you're planning to write one, which means we're back in special filesystem land. That would be an absurd amount of work for what you're actually trying to accomplish.

As for NFS-style locking, there is no server on which to run the lock daemon. If there were a server, we'd be talking about NAS instead of a SAN, which is a different thing altogether. With a SAN, there's nothing but computers and disks, and the disks are not smart. You can't do file locking at the disk level. You have to have a mechanism through which the two computers can communicate with each other directly... which puts us back into special filesystem territory yet again.

I'm not quite sure why, but it's clear that we're not communicating well here. What can I say to make this more clear to you?

Re:Lose the buzzwords

[dbrutus]

Ok, let's take your figures, instead of $80,000:$1 it's $12000:$1. Somehow I think that 10GB ethernet is going to stay out of the disk array field for quite some time while Firewire has realistic possibilities in the relatively near future (ie as soon as somebody releases the necessary drivers).

Re:Lose the buzzwords

[1101z]

I am hearing you. When I said that the files are locked at the application layer I ment that ether extended attributes, or lock files are used. The files are not really locked. This is already done some applications. You have to depend on a well behaved application. If you want to call that a filesystem you can but seems to fall below my definition of a filesystem.

There is a "server" but it is emulating a disk with a standard file system.
Or with the lock deamon on the "server".
You try to open a file that is already open for writing and it fails. I am reading a file, some other system trys to open it for writing it fails.
You don't just open a file then decide what you are going to do with it. That is why the open function has arguments.

Now I never said it was easy(but you make it sound imposable) or that I was going to waste my time tring to do it. It is just that it is what I would like, and I am not the only one that wants something like this. But most of the Uber hackers out there today work for companies that want to make money off of the Big Boys and or people that can aford the current solutions not the little guys.

Re:Lose the buzzwords

[foobar104]

First of all, lock files aren't going to cut it. All we have to do is come up with a very simple scenario. You open up a file on the disk for reading, and start reading data out of it. You read a chunk, do something, then read another chunk.

Meanwhile, I come along and unlink the file.

In a single-access system, this isn't a problem. If I unlink the file while you've got it open, nothing really happens until you close it. After you close it, the file disappears and its space is reclaimed. This is because the kernel keeps track of who's got which files open, and prevents one process from pulling the rug-- so to speak-- out from under another process.

(This is also the source of the age-old trick of opening a file and immediately unlinking it. It's a good way of handling the automatic reclamation of temporary files.)

But if we've got two systems, each with a totally independent kernel, talking to the same disk, the problem suddenly gets a lot harder. When I unlink the file, my kernel will look at its tables and conclude that nobody else has the file open, so it will immediately try to reclaim the blocks on the disk. This will send your application, which is actively trying to read from the file, into shitfits.

No amount of lock files would save you in this situation. All I did was type "rm somefile." The rm command doesn't check for the presence of a lock file. Unless you want to rewrite all the file utilities, lock files aren't going to do the job.

(And extended attributes, of course, aren't portable. They're also tied to the inode, which raises its own set of problems.)

I just think you're grossly underestimating the complexity of this issue. If all you want is to share files across a network, then just use NAS. It'll work better and your life will be easier.

Re:Lose the buzzwords

[hank4snow]

Guys, I think you're both absolutely right about the complexity of simultaneous access to shared disks - something has to control that, and it's NOT the disk array. At my customer site, I was using HP's high-availability cluster software MC/ServiceGuard, and the operating system's logical volume manager controls the shared access (you prepare the volume group as shared, then mount it in 'exclusive' mode where it keeps track somehow how's mounting it - so that no simultaneous access is possible - MC/SG is for high-availability only, not for parallel performance like Oracle RAC).

Let's take a step back. Let's assume I don't want too much smarts, and that I will look after the mounting so that no two SAN nodes will attempt to mount the same volume / partition. That should make it easier, right? You mentioned something about reversing the IEEE1394 stack - can you be a bit more precise?

E.g. the home built disk array is partitioned into two slices ("1" and "2"). Computer A mounts slice 1, computer B mounts slice 2.

BTW: The disk array I used has a great feature - you associate LUNs (space allocated on the disk array) to the World Wide Name of the FC HBA (like a MAC address in FibreChannel land). It's called "LUN Security"... Do Firewire controllers have an Identification of themselves?

Re:Lose the buzzwords

[foobar104]

What you're talking about is absolutely possible. More than that, it's a key feature of fibre channel technology. And, in my sincerely humble opinion, it's really the only effective way to do a SAN: connect all the computers and all the storage devices to a big fabric, then give computers exclusive read-write or non-exclusive read-only access to each LUN. Shared access is hardly ever worth the trouble that goes into setting it up.

Your HP solution sounds pretty much like IRIS FailSafe, from SGI. The storage device is shared between two (or more) hosts, but only one host has access to it at any given time. When that host fails, the other host automatically mounts the storage device and takes up the failed node's responsibilities. Shared storage is a requirement for that kind of configuration; in the old days, we used to do it with SCSI.

I have no idea whether FireWire can support multiple hosts on a chain. Obviously it can handle more than one target, but I don't know about more than one initiator.

Also, the "LUN security" feature you talked about is sometimes called "LUN masking" or "LUN mapping," depending on how you do it. You can map a LUN to a switch port-- that's LUN mapping. Or you can set it up so that a port specifically doesn't see certain LUNs; that's LUN masking. I haven't worked with fibre channel on Windows in an age, but back when I did we had to use LUN mapping/LUN masking. At that time, the Windows host would try to send a device reset command to every LUN when it initialized its HBA. This was, of course, a very bad thing. So you had to set up your switch so that the Windows machines only saw the LUNs that they would be mounting. It was a little bit of a pain in the rear to set up, especially if the switch only supported LUN masking and not LUN mapping. If you add LUNs to your fabric, you have to go in and adjust all of your masks to mask the new LUNs out. On the other hand, if you use LUN mapping, the mapped ports will just ignore the new LUNs unless you specifically tell them not to.

Re:Lose the buzzwords

[billcopc]

Well then, I think I'll stick with my banzai IDE Raid-0 for now :) And D-Link Quad-ethernet is good enough for me.

Or clustering ...

[TRS-80]

A cool thing to do with a firewire SAN would be clustering, ala TruCluster, which presents a single filesystem across many machines (with kernel hacks to allow different files for different machines, eg hostname etc.).

Re:Or clustering ...

[foobar104]

Actually, that's not "a cool thing to do" with a SAN. That's the sole purpose of a SAN: to let multiple computers talk to the same set of storage devices. Depending on how your SAN is set up, you may use software to connect (logically) only one computer to each storage LUN, or you may be able to have multiple computers talking to the same storage LUN through some kind of arbitrated filesystem like CXFS.

Re:Or clustering ...

[afidel]

Umm don't be a dumbass and go off half cocked. Clustering is not the only use for a SAN, in fact I run both NAS boxes and a SAN that have nothing to do with clustering. We use both to give our boxes managed expandable storage at a cost much less then direct attached storage and with the ability to expand far beyond what any case can hold and even beyond what most boxes could talk to directly using scsi.

Re:Or clustering ...

[foobar104]

Uh-huh. Sounds to me like you're using your SAN "to let multiple computers talk to the same set of storage devices." In your case, you may be using a shared filesystem, or you may use LUN mapping to assign a unit of storage carved out of a central system to each computer on the SAN. In either case, this is a good example of what I was talking about.

SANs for storage consolidation, good. SANs for shared access to read-only data, good. SANs for shared access to read-write data, bad.

FireCube?

[questionlp]

I remember seeing a while back that a company had a storage device for Macs that allowed several users attach to the device using FireWire and the supplied software to access the drives in the unit. I can't remember who made it or what it was called... but a Google search can probably bring up a couple of hits.

I'm not sure if I have seen any PC-oriented FireWire SAN solutions though as FireWire hasn't really been something you would see in a lot of computers until recently.

I did find a couple when doing a search for "FireWire Network Storage":

http://www.adept.net.au/1394/nas.shtml
http://www.networkcomputing.com/1118/1118sp3.html (this is probably what I was thinking of)
http://www.turnover.com/news/mdm/firenas.html

Re:FireCube?

[flux4]

I retail this product. It's called the SANcube. A Windows-Compatible variant is also available, but it appears to be a single-user Raid 0 only.

Just think

[Anonymous Coward]

You could hack an 802.11b router with a big antenna and give storage to everyone wirelessly! Wow that would be so cool, then if anyone ever picked up their cordless phone you could... hey ... hmm....

CAT5!!! Why would you want to use firewire? Are you putting up something fun just for the hell of it or are you going to actually make it plug-it-in-the-network style? Sounds way too complicated for what sounds like a storage cluster with a front-end package.

USB2

[Yohahn]

USB 2.0 could also be used.
I think it would be cheaper.

Re:USB2

[BitGeek]

USB is really slow. USB2 I mean. Its theoretical top performance is 480mbps, but Firewire is actually 400mbps-- sustained.

You cannot sustain 480mbps over usb2.. Its really a very slow protocol. Especially if you plug in a 12mbps device into it.

Even if you don't it really isn't up to speed for talking to more than one disk drive.

Re:USB2

[Yohahn]

Are you sure that you aren't just hitting a bottleneck in a USB2 device running connected to a PCI slot? (PCI is the bottleneck)

Re:USB2

[Gruturo]

Are you sure that you aren't just hitting a bottleneck in a USB2 device running connected to a PCI slot? (PCI is the bottleneck)

Not likely (unless the USB controller chip is real crap, which does happen).

USB2's bandwidth is 480 Megabits which equates to 60 Megabytes per second.

PCI, in its slowest incarnation (32bit, 33Mhz, the most common flavour) does 132 Megabytes per second, aka 1.056 Gigabits/sec.
So the culprit is hardly PCI (unless some other PCI card is hogging the bus).

Re:USB2

[MrResistor]

I paid $19.95 for my last FireWire card, and that was about a year ago. Looking at Pricewatch, the cheapest card I can definately say is USB2 is $16.85, vs $18.25 for a FireWire card that comes with a 6ft cable and Ulead software (which sucks, btw, but at least it's free).

Not much of a price difference, especially when you consider FireWire's considerable performance advantage, but that's already been discussed by other responders.

Re:USB2

[Yohahn]

These are all theoretical numbers, try it out and measure it.

Re:USB2

[Anonymous Coward]

Are you sure that you aren't just hitting a bottleneck in a USB2 device running connected to a PCI slot? (PCI is the bottleneck)

All USB data must flow through the CPU, and so the CPU is usually the bottleneck. And watch as your proccessor usage jumps to 100%! Firewire is a much better interface and does not have this limitation.

Re:USB2

[Omega996]

those specifications are worthless. the only thing you should be interested in when measuring bandwidth is sustained throughput. a real-life 32-bit PCI bus (not just the slot, but the entire bus, which may have as many as 6 devices on it) has a sustained throughput of about 80 MB per second. even 64-bit 66MHz PCI 'only' does about 220MB/sec, depending on the implementation.

Re:USB2

[Dahan]

Been measured. Firewire is faster than USB2. Isn't that what BitGeek's been trying to tell you?

Re:USB2

[Yohahn]

It's not been my experience

Re:USB2

[red_dragon]

IIRC., on a given USB chain/bus/tree/thingie/etc., there must be one *and only* one master device (generally a computer) that controls it and manages the transfer of data between devices, while other devices act as dumb peripherals waiting for the master to do something with them. Firewire, OTOH., resembles a peer-to-peer network, in that each device can be an intelligent controller and can initiate transfers to/from other devices on its own. Thus, Firewire is ideally better suited to building a SAN than USB. I'm not saying that a USB-based SAN would be impossible to build, but that it'd require some serious hacking in order to coax it into something more useful than a very fast serial port.

Re:USB2

[Yohahn]

On the other hand, there are 1 chip usb-client solutions. Point me at a 1 chip firewire solution and I'll believe that firewire could do it cheaper and better.

The engineering would be cheaper for usb2.

Re:USB2

[Yohahn]

Where did this rumor come from?

how about iSCSI over ethernet?

[aderusha]

i know that it isn't ieee 1394, but if you want SAN capability hosted by an off the shelf linux box, you may want to take a look at some early implementations of the draft iSCSI spec. basically, it'll let you present scsi devices over IP, giving you a SAN over any IP network (preventing you from dropping $$$ on fibre channel infrastructure).
--------

Re:how about iSCSI over ethernet?

[questionlp]

The only downside of iSCSI is the fact that Gigabit Ethernet will give you a maximum burst rate of 125MBytes/sec (+/- 5MB/s), but that is without the overhead of the TCP/IP protocol and the fact that you may have other devices sharing the bandwidth on the same PCI bus. Of course, good Gig Ethernet NICs and switches aren't as expensive as FC switches and host bus adapters. 10Gig Ethernet will definitely provide a lot of available for iSCSI, but getting 10Gig Ethernet NICs and switches capable of 10Gig Ethernet isn't exactly cheap.

Fibre Channel is expensive, but at least you get more of the 1Gb/s or 2Gb/s bandwidth than you would with Ethernet + TCP/IP overhead.

Now what would be nice is a "SAN" or shared storage unit that can support multiple Serial Attached SCSI channels :) Of course, SA-SCSI isn't available just yet.

Re:how about iSCSI over ethernet?

[aderusha]

understood about throughput issues (even w/ gbit ethernet), but as as the question was for a home setup, he obviously isn't going to be dropping the quid on a new brocade switch and an emc symmetrix. iscsi can run _right_now_ over hardware he probably already has.

just not ieee 1394 that i know of :)

Re:how about iSCSI over ethernet?

[questionlp]

That I definitely understand as I also fall into that category... somewhat :)

I think iSCSI wouls be a nice solution for those who can't spend a whole lot of greens or quids... even if he needed special NICs (I think those NICs offload some of the processing onto a chip on the NIC rather than pelt the system processor) they aren't as expensive as 2Gb/s Fibre Channel adapters.

Re:how about iSCSI over ethernet?

[aderusha]

not only do the iscsi nics (at least from emulex) offload the processing, they also allow remote booting over iscsi for completely diskless servers, much like your typical FC HBA.

You haven't been very clear, but you can do both

[Gruturo]

As others already pointed out, you are confusing the NAS and SAN concepts (but is it your fault? look at stuff like EMC Celerra HighRoad and then you'll be confused :-) )

Anyway,
Want to exploit 1394 (heck, we can finally call it Firewire!) to mount a disk? You just need a 1394 enclosure for your regular IDE disks. Example.

Want to exploit 1394 to access a network share via SMB/NFS? You can, with Ip-over-1394 (works on Apple, Linux, Win ME and XP. Not on 2000).
You just load the correct modules and it shows up like a network interface.

Just my 0.02.

I am not associated with the linked shop, I just happen to be a happy customer of theirs. Their Fire-I webcam is really cool (640x480x30fps) and it's amazing how well it can focus on extremely near objects, it's almost a microscope. I put it in contact with the screen and was able to focus on single pixels.... now that's a nice way to really study ClearType :-)

Overkill for household use.

[akgunkel]

A true SAN is probably overkill for household use unless your hobies include rendering and editing Pixar-like shorts with your wife/girlfriend/dog/hamster working in tandem on one or more other workstations. That's about the only thing I can think of that a home user could use that kind of storage and speed for. Or you want to build some kind of 4 pc DIY TiVo with shared storage. Of course I don't know why you would.

I'm agreeing with Billco. If you've got a Switched 100Mb Ethernet LAN in your house (Since you're toying with building a DIY SAN, I'm sure you do), just build a fileserver. The cost, effort and extra cable spaghetti just don't seem to be worth it. If you build a server, it can do a hell of a lot more than just locally share files too. (DHCP, LDAP, E-mail, HTTP.... ) And considering what you'd spend on a SAN implementation, you could get a pretty nice server for your home.

As questionlp pointed out, if you've got Macs, the SANcube is in a price range that's manageable for the hard core (employed) geek.

Remember, use the right tool for the job. Don't kill flies with a bazooka.

Re:Overkill for household use.

[foobar104]

...unless your hobies include rendering and editing Pixar-like shorts with your wife/girlfriend/dog/hamster working in tandem on one or more other workstations...

Come on, dude, don't be silly. Everybody knows hamsters are no good at animation.

Re:Overkill for household use.

[akgunkel]

Come on, dude, don't be silly. Everybody knows hamsters are no good at animation.

You need to upgrade your hamster. Mine is great at animation. He's working on this awesome shot right now with some alien warships bombing the Earth from orbit. Great stuff.

Re:Overkill for household use.

[Burl Ives]

Dude, I just rendered my shorts...

I did something similar

[dasunt]

My "fileserver" (also DHCP server) is a Pentium 166 with 32 megs memory and an old 10 mbit 3com509b card. Basically, $25 on ebay. A floppy drive was used to install linux (net install of Debian), then removed. There is no cd drive, and no video card in the machine. A 2 gig HDD is used for booting, and for most of the system files, and an 80 gig HDD is used for storage. (It was big at the time).

Running Samba, I can saturate my 10 mbit network with the machine. With tests done on a 100 mbit network, I reach about 30% use. However, the bottleneck is not the CPU or memory, it seems to be the onboard IDE. With a PCI ATA 100 card, performance should go up.

All in all, its a nice machine. Since its a desktop, it fits nicely under the printer it shares. An SSH server allows me to securely log in, change any system settings, and do updates. Its quiet, cheap, and effective. With only a power cable, an ethernet cable, and the printer cable, its neat. And did I mention upgradeable?

A hardware RAID-IDE card should cost me about $250. [Haven't tried software RAID in a P166 and I have no urge too.] That shouldn't put any load on the CPU, and would provide redundancy. Getty on a serial port would be nice as well. If I want to, I can also swap the drive for something bigger without worry about the system supporting it. With ext3, it handles power outages well.

It works.

Re:I did something similar

[Anonymous Coward]

You stupid twat. Do you have the slightest fucking idea what a SAN is? Do you? You built a fucking file server, and from the sound of it a shitty one at that. (Only 30% of wire speed on a 100BASE-T network? Asshole, I can do 100% of wire speed between two fucking Mac laptops.)

I don't ever want to hear you say "I did something similar" again unless you have some tiny, microscopic clue about the subject of conversation.

You are hereby banned from posting to Slashdot for twenty-four hours. It's early autumn in the northern hemisphere and early spring in the southern; there is no habitable point on Earth where the weather is not absolutely beautiful right now. Go outside and exercise something other than your wanking hand for a while.

Re:I did something similar

[dasunt]

Well, either you're a troll or an idiot. Lets assume you are the latter.

From the article -
How about this: would it be possible to create a Linux-based disk-array with an IEEE1394 interface (Old P200, crammed with disks, software RAID, lots of RAM for caching, Firewire interface, looking/acting like a single disk to the outside world, storage device mgmt via web-frontend)?

Your reading comprehension is horrible. So is your technical knowledge. Let me educate you.

First of all, this guy wants to use a pentium 200 as the basis of the system. This places technological limitations on the system. For example, some pentium chipsets have a caching problem with anything over 64 Megs of memory. The kernel can work around this (basically by using anything above 64M as a swap file) but there are limitations to performance. There is also a limitation of how much memory you can put in the old pentium motherboards. If you're looking at a pentium-based solution, you're looking at something that's cheap and not appropriate for heavy loads.

Now what disks are you going to put in this cheap system? SCSI? Only if you have the brains of the anonymous coward that I'm replying to. On pricewatch, a 146 GB SCSI drive is just under a grand. A 120 GB IDE driver is about $150.

You talk about getting 100% speed between two laptops. Interesting. Not sure how that applies, since I'm willing to bet money that my packets traveled just as fast. If you're talking about bandwidth, we have a different problem. In a sustained read from a device, the limiting factors will be the HDD speed, IDE Bus speed, and ethernet card. Lets look at the HDD speed. Tom's hardware benchmarked a recent 120 GB HDD at between 20 - 40 mbytes/second. So, the hard drive should be able to saturate a 100 mbit/second ethernet network in a sustained read. Unfortuneately, when you run a new HDD over a 5 year old ide bus, the performance goes to hell. Say hello to 3 mbytes/second. If you don't realize that there will be a performance difference between a new mac laptop and some hardware that is a half decade old, then you're naive. No matter who you call 'Asshole', you won't get 100mbit/s out of a 5 year old IDE bus.

So, how do we fix this? By either putting in a new IDE card, or RAID. Lets buy a nice card that does everything in hardware. The ATA-100 specifications gives us more then enough speed to match the hard drive. A RAID card allows us to combine hard drives and increase the maximum data transfer rate. However, we have a 132MB/s limit on the PCI bus (32 bit 33mhz bus). This should be fast enough to max out an ethernet network (even gigabit) or firewire.

Up to now, this looks like building a bloody file server, doesn't it? Figuring out bandwidth and speeds of the components. *Slap* There goes your 'similar' complaint. In fact, some ethernet-based NAS are nothing more x86 Hardware and a custom BSD system. (Yes, I know SANs and NASes are different, but the storage media tends to be the same.)

You can't replace the PCI bus, so if you're using a pentium based system, you're limited to 132mb/s, at peak efficiency. In practice, you'll get less then this. If the hard drive will be handling non-sequential reads (which it probably will), expect another drop.

Basically though, building a NAS is nothing more then building a file server, then instead of running ftp/smb/nfs/afs or the like, hunting down or hacking something to provide NAS over firewire. From a hardware perspective, this project is easy. Software might prove a challenge.

As for the weather outside, not all of us live between 45N and 45S. Some parts of the world are rather chilly this time of year.

And that, sir, is why you are an idiot.

Don't use firewire.

[stienman]

The upcoming serial ATA standard will give you better performance at a lower cost. A firewire drive is large, expensive, and consumes slightly more power. All you gain over current IDE technology is hot swap, and that will be solved with serial ATA.

But what you are really after are the tools to manage such a beast. The physical implementation shouldn't matter to the developers - all the software needs to know is that storage exists that the user needs to use, and how to read from and write to said storage. It shouldn't matter whether it's an IDE drive, a friewire, a usb, a scsi, a 1000 tape library, or any combination of storage devices which, IMHO, will be a great differentiating feature from commercial packages.

Yes, the free SAN package handles your old room size tape robot as well as this rack of serial ATA drives, and will treat them accordingly - near line storage in the tapes (semi archive), on line storage in the HD, and off line (off site) over the WAN link to the storage cluster at your other shop. If you need an extra terabyte just go to officemax and plug in a firewire drive until the tech comes out and adds more serial ata devices to your drive chain.

Of course, you could buy the SAN package available from x, or y, but you'll pay dearly for it, and you can't add storage to it yourself. Oh, and it only works with their hardware.

-Adam

Re:Don't use firewire.

[foobar104]

Adam, you've confused a SAN with an HSM (hierarchical storage management) system. They're not the same thing. In fact, they're really kind of incompatible ideas. In a SAN, you have a number of computers talking directly to a number of storage devices, without any arbitration in between. In other words, it's like talking to a file server, only without the file server*.

An HSM system, on the other hand, uses software running on a file server to consolidate several different kinds of storage devices into one logical filesystem. As you write to the filesystem (over the LAN), the server puts the data on disks. When the disks start to get full, the server begins, in the background, moving data from the disks to an automated tape library, gradually freeing up disk storage as it goes. This happens without the client's knowledge; it looks like the server just has a whole lot of disk space available. When the client requests a file that's not on disk, the server stalls for a bit while it retrieves the data off of tape, then it returns the data to the client. So in an HSM system, client-to-server writes are really fast, but reads can be really, really slow.

Since a SAN depends on directly attaching computers to storage without a server in between, and HSM depends on having a server there to manage the different types of storage devices, they're kind of incompatible ideas.

* Reminds me of the old Einstein quote about radio. "You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles. Do you understand this? And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat."

Re:Don't use firewire.

[Omega996]

you forgot to mention something he'd gain by using firewire - it's available now. he won't have to wait a year before native serial ata drives and controllers are out, and he can use 1.0 drivers with this new hardware...

Re:Don't use firewire.

[Anonymous Coward]

The difference between NASes and SANs are 99% marketing.

A NAS runs NFS over IP over Ethernet. A SAN runs a Global FS over SCSI over FibreChannel.

The difference is purely the protocols chosen.

Re:Don't use firewire.

[foobar104]

That's a common, and totally 100% wrong, misconception.

SAN and NAS are fundamentally different implementations. They have the same basic purpose: computer A and computer B need to access the same data on the same storage device at the same time. But that's where the similarities end.

NAS is strictly a client-server system. All the clients talk to the server, but not to each other. Clients make read requests to the server, which queues and handles the requests, caching the data along the way. The server handles things like file permissions, access control, locking, and synchronization issues. The server also arbitrates contention situations, by putting I/O requests into a queue.

Shared-access SANs are completely different. In a shared-access SAN there is no server, which means there's no central arbiter of things like permissions, access control, locking, synchronization, and request queuing. Instead, each client computer simply talks directly to the disks. In theory, taking out the middle man this way decreases latency and increases bandwidth, but in practice contention issues arise that eliminate any gains. Since there's no arbiter for things like permissions and access control, the clients all have to talk to one another somehow; that's where cluster-aware filesystems like CXFS or Centravision come in. These filesystem are complex in ways that most people fail to realize, and they are highly prone to failure. In particular, an election-based system like CXFS doesn't tolerate the coming and going of nodes to and from the cluster very well. At any time, any node can be the control node, and if it disappears, the filesystem can become wedged for a time until a new election occurs. So these types of filesystems work best in tightly coupled groups of systems, like highly available clusters, or parallel processing clusters.

SANs and NAS are only similar on the surface. Beneath the surface, they're very, very different.

SANcube

[Anonymous Coward]

SANcube by MicroNet.

Re:SANcube

[questionlp]

Thanks... that was the one I was thinking of. The Network Computing article had mentioned it.

terminology, technology

[rakerman]

Ok, it's not clear from your posting exactly what you want.

Do you want NAS?
That's Network Attached Storage. Currently almost entirely Ethernet based. You get a box with some disks and software, and it sits on the Ether looking like a fileserver, maybe just a CIFS server for Windows boxes, more likely both CIFS and NFS to support Windows and UNIX.

Do you want a SAN?
That's a Storage Area Network.
A bunch of disk boxes connected together with a switched Fibre Channel network. Servers connect by Fibre Channel directly into the network.

Do you want a NAShead on a SAN?
A NAS device acts as a front-end to the SAN, so you have an Ethernet file-sharing frontend onto a Fibre Channel storage network backend.

The problem with implementing any of these is they're about more than a transport medium. A NAS is more than Ethernet. A SAN is more than Fibre Channel. Those media mostly just pump the data around. It's a ton of software that handles the sharing of files.

So sure, you can string a bunch of disks and CD burners and whatnot together with FireWire. No problem. I do it myself. "FireWire" disks are almost entirely just an enclosure with a normal ATA disk inside and an ATA-to-FireWire bridge. Adds a small cost onto the price of a regular IDE drive, that's it. You can buy the enclosures yourself and do it quite cheaply.

However, the operating systems that you connect to the FireWire are going to have no freaking idea about filesharing. If you try to connect more than one host, it won't know what to do.

What you need is FireWire ***PLUS*** filesharing software.

Unibrain makes something they call FireNAS

http://www.unibrain.com/home/

That's about the closest thing in existence to what you describe.

If you're wanting to use IP-over-1394 (RFC 2734), be aware that Microsoft's stack is the main working one. The Linux stack is in beta and Apple has no plans to implement IP-over-FireWire at all.

You can find more info on IEEE-1394 at

http://www.cs.dal.ca/~akerman/gradproject/projec t- links.html#IEEE1394
Also check out the Linux 1394 project

http://linux1394.sourceforge.net/

Serial ATA not the answer

[0x0d0a]

Just one point -- Serial ATA is going to run in parallel with vanilla ATA for a while, while vendors skim money off high-end users. It's not going to be "SCSI at ATA prices", at least not right away.

Re:Serial ATA not the answer

[stienman]

It never will be scsi at ata prices. Serial ATA will reduce pin count on chipsets (lower cost), reduce cable clutter and cooling issues (lower cost) and is still significantly faster than any physical hard drive out there right now anyway.

It isn't meant to replace scsi, it's meant to replace parallel ATA.

-Adam

Cheap SANs

[0x0d0a]

As others have pointed out, what you're talking about is a fileserver, not a SAN.

However, interest in cheap SANs is rising, and I suspect it won't be long before a couple of projects start up to build these, then they get polished, then corporate types get interested in the big cost savings, and they start using these. It'd be particularly cool if Linux beat Windows to the gun here.

Before you scoff, remember that that's what happened with the advent of clustering cheap PCs -- the custom supercomputer is nearly a dead beast now.

There are enormous profits on SANs, so an open-source project could do wonders here.

The linked example is quite expensive.

[Futurepower(R)]

The linked example is quite expensive. It is better to buy an empty firewire enclosure and put a 120GB WD drive in it.

Ah yes, hamster upgrades.

[Futurepower(R)]

You forgot to provide a link for hamster upgrades. I hope they are downloadable. Can an upgraded hamster pull a bulldozer out of a muddy hole?

Mental Illness

[Anonymous Coward]

Mental illness is terrible, as the hostility in the above AC's comment demonstrates.

Re:Mental Illness

[Anonymous Coward]

the AC was hostile because dasunt wasted people's time. the question was about SANs over IEEE1394, two concepts that dasunt obviously knows nothing about. and when he saw those strange new terms "SAN" and "IEEE1394" did he bother to look them up? no. he instead posted some crap about how he setup a file server.

personally, i found the comments about current seasons in the hemispheres making every part of the world beautiful to be very uplifting.

SAN (as opposed to NAS) is possible with FireWire

[MrRobahtsu]

Oracle has a modified iee1394 kernel module that would allow multiple hosts to use FireWire-attached drives just like shared SCSI. They did it for cheaply testing their cluster file system, but, hey, if it works ...

Multi-Drive Enclosures?

[algae]

Speaking of this sort of thing, can anyone point me to a FireWire external disk box that holds more than one drive? I'm thinking something along the lines of this only using 1394 instead of fibre or SCSI.

Re:Multi-Drive Enclosures?

[Omega996]

places like firewire depot sell firewire raid enclosures, is that what you're asking about? it might be cheaper to get a mini-tower, some firewire bridge boards, and make your own disk box.

A SAN is what I'm after...

[hank4snow]

I appreciate everyone's comments about SAN and NAS, and apologize for not having made my point clearer.

I'm not interested in NAS / Fileserver / anything running over Ethernet, simply coz it's a no-brainer to set them up. The whole post to ask-slashdot was probably more theoretical than anything else, to discuss what started as a crazy idea with fellow geeks.

I know I can get a Firewire-IDE enclosure, but the question is, what happens if it hangs off a Firewire hub together with two computers? Will both be able to see it? How do you partition it (just normal fdisk I assume)? What if the two computers have different OS's? Then of course you've got to make sure that no two computers mount the same partition...

Then I started taking it further: The device used at the customer site was a real disk array with RAID5DP and lots of cache. Would it be possible to build a low-cost disk array e.g. using linux - very much similar to the SanCUBE. Then you could do much more than with just a FireWire-IDE disk; think of "LUN security" - ensure that computer X only sees the partitions that it's supposed to see...

Again, thanks for all the comments!

Re:A SAN is what I'm after...

[yancey]

FireWire is not designed for such use. FireWire is optimized for large data transfers and the typical SAN transaction is under 2 kb in size. USB handles small transfers better, but requires significant amounts of CPU overhead that would not make it practical for anything like a SAN.

I like the idea of a "cheap" SAN, but I think it still needs to perform well. I would have my doubts about doing anything like this with FireWire or USB.

Now if only Xiotech would make an IDE version of their SAN!

Re:A SAN is what I'm after...

[Omega996]

if your 'sanbox' could run in target mode, and keep track of file locking issues, this might work. the only computers i know of that can operate in target mode with firewire are macs, and i don't think they can be used simultaneously by more than one machine when in target mode.
I think you'd still need some sort of agent on the client (device driver, whatever) to communicate with the management device (or OS running on your single device) to get the right partition, or at least keep track of file-locking issues on a shared partition.

Re:A SAN is what I'm after...

[hbackert]

Hi,

I was waving the same idea as you have. Since Firewire can support multiple masters (unlike USB) and technically there is no difference in hardware for client and master, it is technically possible to use a PCI hostadapter, given the correct driver software, to emulate a Firewire external disk (or many disks) to another computer. Apple calles it "target mode" for their notebooks.

Now from what I heard and tried, this is not possible given the usual Firewire external disks, as those are "bridges" from IDE to Firewire, not full Firewire controllers. I was not able to see the disk on a second computer connected by the second Firewire port of the disk.

However since you want to emulate the disk using a full blown computer, it's up to the software to do that. Some people have pointed to the Oracle project handling this (or somewhat handling this, as I cannot find out, since their "create a new account" page throws a JSP error... It might be not exactly what you need, but might be a good starting point.

For myself, I came to the conclusion, that it's not as useful as I thought as first, since you cannot boot from Firewire (Macs can, I know, but my computers are no Macs). And for me a NAS featuring RAID, LVM, xfs/ext3 do support what I mainly want: resizing partitions/space in a flexible way for the client PCs.

But having a SAN on top of Firewire would be nice and would be getting useful as soon as you have bootable Firewire cards. I anyhow wonder why no company does those. You can boot off USB (onboard USB), Ethernet, SCSI, floppy, IDE, but even on on board Firewire controllers you cannot boot from them. This would enable you to have a no-disc PC, running usual OS'es, not just special ones which can boot via NFS (dunno about Win2k).

Harald

Oracle worked on this already

[photon317]

Here's a blurb from Oracle's Linux Page about some patches they've done to linux for low-cost firewire SANs:

Firewire Patches fixes some issues with Firewire on Linux and enables shared disk on top of firewire drivers. Firewire allows developers to easily and cheaply build a clustered system on a shared disk, which is useful for testing clustered applications and checking out the advanced features of Oracle's Real Application Clusters technology. The Firewire cards needed to build a cluster can cost as little as 10% as much as the required FiberChannel hardware.

Re:You haven't been very clear, but you can do bot

[rakerman]

AFAIK there is no built-in support in MacOS for IP-over-1394.

USB 2.0?

[Anonymous Coward]

What about usb 2.0?

I recently picked up an ADS usb external drive enclosure with internal ide and slapped in a WD 120 GB harddrive. It's useful as a "really big floppy".

The usb 2.0 spec should allow for faster than ultra scsi transfers with the added benefit of hot-plugging. Also, the hdd's can be plugged into most pc's albeit at the usb 1.1 transfer rate...

Make it STOP!

[Enry]

Drop what you're doing and go to your local book monger. Go get the ORA book "Using SANs and NAS". Read the descriptions of each.

Then come back here and ask that question without laughing hilariously.

Re:Make it STOP!

[rakerman]

Heck you don't have to go that far, just read The Top 10 SANs vs. NAS Decision Factors by the author of the ORA book. Or sign up and read the book online at Safari.

RE: Lose the Buzwords

[chunkwhite86]

Wow you are ignorant. The purpose of the SAN is certainly not ease of installation. And a Maxtor MaxAttach certainly is not a SAN! It's a NAS! Get it straight!

A SAN as it is defined today, MUST be created using FC loops or fabric. There is no other topology (unless it could be done with firewire).

Also, ease of *management* not installation, is the bonus of the SAN. Also, SANs use expensive FC HBA's which have almost 0% cpu utilization even when streaming data at 2Gb/s. Ever checked the CPU utilization on a P200 when streaming data over TCP/IP at 100Mb/s? Or even a P3-733? No comparison to fibre channel. nada.