Do-It-Yourself Fibre Channel Array

skarphedin writes "There's an interesting story here on a do-it-yourself fibre channel array. These guys make one for under $250 and it can perform up there with 15k SCSI in some cases." You know you want one.

Comments

Cool

[ExEleven]

This is cool, pity ill never afford it.

Re:Cool

[Anonymous Coward]

Jeezus ... you can't afford $250?? Get a fricken' job!

cool

[strider44]

Wouldn't that make a cool lan setting?

Here's another way

[Anonymous Coward]

1) Buy a lot of fibre
2) Fly to Folkestone, England
3) Rent a sailboat
4) Sail to Calais, France, laying an array of fibre behind you
5) Congratulation on your Do-It-Yourself Fibre Channel Array.

Store already slashdotted

[tliet]

It seems the links to the store selling the goodies is already slashdotted. The $40 a piece FC hostbus adapter page now shows $800 adapters, or a 100 pack for just over $60000. Beowulf anyone?

Re:Store already slashdotted

[tigress]

Try eBay. We got a QLogic 2100 (Copper) for $40. Drives can be found for $99 for a tenpack of 18.2GB Seagate 10k drives.

Re:Store already slashdotted

[Anonymous Coward]

Those $25 drives ain't $25 now, either.

Re:Store already slashdotted

[Anonymous Coward]

Are those drives really that worthless? I mean look at this
http://cgi.ebay.com/ws/eBayISAPI.dll?ViewIte m&item =3406233779&category=11160

Seems like they should cost much more. A small business say a video shop could build one hell of a server with those. I guess its just surprising that they don't cost a lot more than some of the IDE drives I see there. Do you think there just not reliable anymore?

Re:Store already slashdotted

[tigress]

Ask yourself, who buys Fibre Channel drives? Is it home users who need space for their mp3 and warez, or is it major corporations who need multi-terabyte storage for business-critical data?

These big corporations used to buy hundreds, perhaps thousands of FC drives a few years ago. The drives back then were about 9 or 18 gigs in size, meaning that hundreds of drives were required for a sizeable array.

Now (as we also discovered), hundreds of drives have several disadvantages. Not counting the physical space required, you have heat, power usage and the number of drives on each chain to take into consideration. This means, of course, that those big corporations want to replace their old 9 and 18 gig drives for something bigger (Like, a few 72 gig drives).

Many corporations destroy their old drives (for data security among other things), but some sell them off to computer recycling companies who wipe the drives and sell them off, for instance on eBay.

Of course, there's the question of who's buying FC drives. Most home users or even companies can't afford the equipment required for a SAN. Low demand therefore results in lower prices for the drives, which gives us the result that a tenpack of 18-gig drives can go for as low as $99.

Quite nice, if you know what you're looking for. =)

Re:Store already slashdotted

[argus333]

oh yeah, and a buddy of mine just got a QLA2300 (their current top-o-line FC card) for $243.00 US...> linky here

Ewwwww.

[Dr Tom Danger]

Ug, a Compaq case? I've found there just isn't enough "Compaq sucks" propaganda on the web these days.

But seriously, I bet if I wired this in my dorm room I could get some mean negative pings in UT '03. Kinda like a 'spider sense' for the pc.

Re:Ewwwww.

[argus333]

it was cheap and convenient...sorry, i'm not in the habit of being politically correct on case choices...

Re:Ewwwww.

[nc0gnet0]

http://members.aol.com/oldmarbs/images/tower.jpg This is another example

No, I don't want one

[haggar]

I have used FC tech for several years. It's amazing technology, but performance is not it's main advantage. It's advantage is the possibility of stacking up incredible amounts of storage, with rendundant paths, at up to 100 m from the attachment point (one of the servers). This kind of environment is also very mindful of quality, and a self-made solution is not acceptable. Would you stack dozens of these self-made boxes and bet your career that they'll not fail. I know I wouldn't.

On the other hand, if I just want performance, I will do better with SCSI, and even save some money.

In this respect, I don't quite see what kind of niche would the solution in the article cover.

Re:No, I don't want one

[smittyoneeach]

You raise an excellent QA point.
Could become some major FUD, though, Elmer.
You figure that your first outing or two might well have some flaws, and are certainly not ready for enterprise usage.
So you get the hang of it on some little implementations before you go for a big one.
Sort of the way you start up a little software test project and make sure you understand all of
the header, object, and API issues for that shiny new library you just got off of SourceForge
before you try to integrate something new with what you're _really_ working on.
As a business strategy, FUD is great. If we can keep people convinced that they _can't_ do it themselves, they're more likely to hire us at phat consulting rates.

Re:No, I don't want one

[Anonymous Coward]

>>>Would you stack dozens of these self-made boxes and bet your career that they'll not fail

no...I'll let the EMC boxes fail instead.

(and they do...all the time)

Re:No, I don't want one

[nolife]

There is always FUD associated with "Enterprise" class hardware and software. Some things really do need to perform at specific level or a baseline to run in this class, others do not but ride the coat tails of others that due for the exact reason you specify -- people do not want to bet their job on it, remember the old saying, you will never get fired for buying IBM.

At work we decommisioned a large Compaq 2 year old server with a nice raid setup and SAN connectivity running Novell to make room for our MS takeover. The last job this server performed was storage to allow us to boot workstations with a network floppy and create and restore desktop images and for general storage in the IT department. It wasnt to bad but we never really achieved any more then 2-3MBytes/sec when transferring files to and from it. We did not have our new MS server to replace it yet so I took a small footprint Compaq P-III desktop with 512MB ram and loaded RH on it, slapped in an extra 7200rpm Maxtor 160GB IDE drive. Installed Samba, joined it to our W2K domain and it works great. We pull and store multiple images to the thing at roughly 5Mbyte/sec per PC and it has a sustained thoughput of about 10-12Mbyte/sec per HD or per network card (I have two NICs and you can select which one to use from the client boot disk when you connect. It also lets us burn DVD's directly from a Windows workstation at 2.4 speed (3.5Mbytes/sec) which our Novell server could never handle (many buffer underruns or had to transfer image file to the PC first).

It does not have the redundancy of the old Novell server as I have a no raid setup but I back up the files using rsync to my other Linux machine on a daily basis and we have hundreds of other similar desktops I could grab parts from if needed. That desktop coast us about $600. that server was well over $5000.

I guess the point is, it does not have to say "Enterprise", server, or cost a lot of money to perform the work you may need.

The only other small problem is if I get hit by a bus they are screwed as the rest of the department has little interest in the headless Linux thing I have sitting on my desk. I am willing to explain it to anyone but being in a MS driven shop, so far only one person is interested. All they know is it currently works great.

Re:No, I don't want one

[haggar]

In a nutshell:
- the disk setup is not rendundant
- there is no documented disaster recovery plan

Would you be so candid to tell me what kind of enterprise are you working for?

Re:No, I don't want one

[nolife]

You have just commented on the exact point I was trying to make.
Let me rephrase my first paragraph.. Basically it states some things should be enterprise level when required and it is nice to have a CYA backup when you get stumped. Not all things in an enterprise NEED to be at such a level.
In my specific example, if all else fails, anyone in the office can crack out the DVD's and image the damn workstations manually if they wish.

Re:No, I don't want one

[argus333]

dude, it's a "how-to" article, not a "preaching the merits of FC over SCSI." You've gotta understand something: people are going to want to find cheap alternatives for storage. this is simply a 'pointing the way to a watering hole' article, not the end-all, be-all of storage fantasies...

Re:No, I don't want one

[haggar]

Um.. I wasn't "preaching", either. I think you have missunderstood my post, starting from the subject (which was a sort of reply to the comment to the article, not a reply to the article itself). I even addressed the point of cheap storage, and believe SCSI is cheaper than FC. Look, I'm not goin to repeat/explain my points, too late here, gotta sleep. Didn't mean to belittle the article.
Cheers.

Re:No, I don't want one

[argus333]

hey! no worries, dude....I do believe that cost/GB is cheaper via SCSI, but it's mostly the OEMs fault for the way they position it.. :-)

no harm, no foul...

cheers,

dave

I dont quite get it

[watzinaneihm]

Fibre chanels I thought were used because
1) Huge expansivity .
2) Faster speeds, esp. over LAN (Storage area networks)
Why would one want to use it in a home setup?
You probably are not going to buy more than 3 or 4 Harddisks. I say if you want speed use more RAM(*though you wont get much for $250 * results might vary). If you want expansivity(not too much) and relatively fast (depends on a lot of stuff) access speeds and standards based setup, may I suggest iSCSI

Re:I dont quite get it

[tigress]

Why would one want to use it in a home setup?

For the fun of doing it, for one thing. For instance, I have Token Ring, ATM and serial equipment in my home LAN. Why, when there is FE or even GigE? Because it's fun to play around with, and I learn a lot as well.

Second, you might get pretty darn good performance out of a relatively cheap setup. Modern ATA-drives are pretty fast, but the problem is, there's just a single spindle. Random access will kill your drive. A home-built FC array for the price of an ATA-drive will get you perhaps five to ten separate drives. Mostly, these drives will be 10k drives as well (Almost all ATA-drives are 7.2k or less). The slightly higher rotation speed combined with the fact that you've got a large amount of individual spindles gives you much better random access. Also, remember that ATA usually is a huge CPU hog, which adds to the performance bottleneck.

Re:I dont quite get it

[Ryan Amos]

I dunno where you buy RAM, but you can get 512 MB sticks of PC3100 for $120. Crucial, good stuff too. That's right about $250 for a gig of fast, stable RAM.

Re:I dont quite get it

[Anonymous Coward]

There still are limits to the number of devices that can be attach at the OS level. Last time I checked, HPUX's limit was about 124 devices and Sun's limit was 256. Under 11.11, HP may have bumped that up a bit.

And what good is mutliple fibre channels without EMC PowerPath for bandwidth balancing and failover?

Re:I dont quite get it

[zsazsa]

Why would one want to use it in a home setup?

Because slightly obsolete FC hardware is DIRT CHEAP on eBay. That, and it's kind of fun to install some "enterprise level" hardware at home.

Re:I dont quite get it

[LordMyren]

because its cool

oh, and amazing perforance for cheap from ebay. sure, there's huge startup costs, but storage from then on is neigh on free.

Myren

Re:I dont quite get it

[LordMyren]

Ya know, we need to stop mentioning that very badly, because dirt cheap keeps getting less and less cheap as we drag more people over to dark ways.

Supply remaining constant, demand increasing. Not cool.

How long you think...

[red5]

How long do you think till the mac-heads credit apple with bring down the price of FC by including it in the X-raid? Just like they credited apple with bring down the price on SCSI, USB, etc.

Re:How long you think...

[Anonymous Coward]

I think the possibility of getting such cheap FC solution really goes to show how utterly overpriced apple is. Remember when the xserve raid came out, and the apple naysayers were all "You can't build something equivalent from standard parts for the price". That was just over a month ago. In one month, hobbyists have created a working solution that does what apple's does for nearly a tenth the cost. You can't argue with fast hard facts.

Re:How long you think...

[Large Green Mallard]

Apple's solution however doesn't involve blocks of wood next to high speed and therefore hot devices...

It's also comparing Apples to oranges.. the Apple XServe RAID has an FC interface to the host controller, ie the XServe, but only uses ATA HDDs internally. Apple's is expensive, yes, but for 2.4TB, it's a pretty damn good price.

Re:How long you think...

[camh]

How long do you think till the mac-heads credit apple with bring down the price of FC by including it in the X-raid? Just like they credited apple with bring down the price on SCSI, USB, etc.

Have SCSI prices come down? Hot damn. I'm off to get some of that SCSI gear. I've been waiting for this for years.

Thanks Apple.

Been there, almost done that...

[tigress]

We had great plans for building an FC array up until a while ago. For those who think FC is too expensive, take a look at this:

180GB ATA drive: $200

Qlogic FC host adapter: $40
10 18GB 10k drives (eBay): $99
10 T-cards: $50
UTP-cable: $20
--------------
Total: $209

Of course, there's the cost of running the array as well, which is the reason we never finished our project (We did get the hostadapter and built a couple of T-cards though). We calculated that our FC array would cost us an additional $2-300 in electricity every year. After getting hit with a $500 surprise electricity bill for our current equipment, we simply decided it wasn't worth it and got another IDE drive instead. Still, an interresting project. =)

Re:Been there, almost done that...

[Dr Tom Danger]

I'd totally be all over a bit of homework and put something together like this, save for the fact that the last time I started putting components together my processor lit on fire. With that much cabling and harddrives I bet I could light 'em off like a pack of lady-fingers on the fourth of july.

Who says electronics can't be fun...

Re:Been there, almost done that...

[zenst]

noway can you get a host adapter for $40 - just about get a gigabit ethernet adapter for that but a fiberattached iSCSI card, naaa :. Shame the chap never realy check the prices with his pricewatch links/common sence $40, hehehe I wish, I can only wish.

Re:Been there, almost done that...

[tigress]

Uhh, yes we DID get a host adapter for $40. =)

QLogic 2100 copper, purchased on eBay for $32, shipping was about $8.

Check it out yourself, there's some on there right now for less than $30.

Re:Been there, almost done that...

[zenst]

ebay - hehe - I'll take your word for that as I cant find them, must be a new slashdot effect *shudders at prospect of story based ebay parts selling*. I assume all cheap ebay adapters have been purchased in preperation for artile DUP =).

Still with cheap gigabit adapters available for new and the speed of fast IDE discs then I shant worry.

Still your right with ebay part it CAN be done, so I stand corrected upon the question of the price; though still would like to see cheaper off the shelf parts. but it will come.

Re:Been there, almost done that...

[nuxx]

Where did you find T-Cards for $5/ea? I've been looking for just a single one for the longest time, and they always seem to be at least $40/ea. So hard to find, in fact, that I'm working on designing and routing a 4-device FC backplane and just making them myself.

Re:Been there, almost done that...

[tigress]

We make them ourselves. Take a look at http://overclockers.com/tips681/ for a guide. We etched our own PCBs though, which is a lot easier in the long run. =)

Re:Been there, almost done that...

[ivan256]

Done any performance tests with your 2100?

There's a reason why those cards that were $800 originally are only $40 now.

The 2200s and 2300s are still expensive for a reason too.

Re:Been there, almost done that...

[tigress]

No, we postponed our project due to electricity bill reasons. =)

And yes, there is a reason for why those cards are available cheaply. It's called "Upgrading".

The QLA 2100 is 1Gbps. Half duplex if I remember correctly. For modern FC Arrays, this is too slow. The QLA 2200 (1Gbps full duplex) and 2300 (2Gbps full duplex) are much more attractive, which is why they're still expensive. Big Corporations(tm) want to take advantage of the newer technology and upgrade to the newer standards. The older cards are recycled or sold (cheaply), hence the $40 pricetag.

The point isn't performance. The point for us was simply to do something fun that we hadn't done before.

Nice .. but still just for play

[turtle-spin]

Have to be impressed with what was achieved there .. the usual story of a lot of spare parts, spare time and some good old human brainpower :) But the fairly intermittent results and usual dubious quality of the housing etc (due to your own competence) all say to me.. well its nice but why bother.. current ATA suits me just fine for now!

Re:Nice .. but still just for play

[argus333]

*sigh* geez...what do you want from me? a custom-fabricated aluminum housing coupled with an airbrush job that'd make Falcon-Northwest look like newbies? give me a break! It was fun, took me a few weeks, and gave me a better sense of what's out there....

File Locking

[rf0]

Yeah this is cool and such like but what if you want to mount it across two machine using two FCA's? You need software that allows file locking (such as SGI's CXFS) and that costs. Mind you if you only wanted it on one machine why not just buy a load of disks because in honesty when are you going to need such high amounts of bandwidth?

Rus

Re:File Locking

[Anonymous Coward]

insightful, but you didn't really touch upon the more common reason for an array to be mounted on two or more systems.

fail over cluster.

far more common, and the file locking/sgi cxfs stuff doesn't even come into the picture.

Re:File Locking

[matt2413]

You buy Sanergy! :)

Re:RPN example

[GeHa]

Don't you just hate HP for quitting the calculator business? I'm still using my old HP48S, and it still kick butt!

Still, I wouldn't mind learning what your comment has to do with DIY style FC?

Re:I would like one!

[Daengbo]

Optical fiber is not required for shorter distances, however, because Fibre Channel also works using coaxial cable and ordinary telephone twisted pair. Fibre Channel offers point-to-point, switched, and loop interfaces.
I have to say that makes me think of putting one under the stairs for redundant storage. Too bad the prices for the hardware would be doubled in Thailand.

SCA40 Backplanes

[Detritus]

The drive mounting and enclosure was a bit of a kludge. Are there any reasonably priced boxes that you can install the drives in, with the correct mounting hardware and backplane?

Re:SCA40 Backplanes

[mzito]

You might want to look for Netapp FC(xx) disk shelves. They're normal hot-swap fibre channel JBODs. They're also very expensive, but if you get lucky you might see one sans drives and get it for much cheaper.

Thanks,
Matt

Re:SCA40 Backplanes

[NedTheNerd]

card board box wouldnt do too bad considering he used wood to help mount the drives :)

Re:SCA40 Backplanes

[_Sharp'r_]

Plenty of the Netapp shelves out there, with and without drives. Heck, I've got a couple of extra shelves with no drives right now. Anyone want to buy one?

I've been buying shelves filled with drives rather then just drives just because sometimes it's cheaper that way. You can get a shelf with 7 36 GB drives on the market for about $3-4K now. Of course, it helps if you also have a Netapp head... then you get their awesome file system, etc... as well. That's the best argument for not building your own to me... the Netapp filer's file system and utilities are light-years ahead of a "regular" OS's file system for mass storage.

Re:SCA40 Backplanes

[argus333]

LOL! dude, I had it, it was handy and i don't care.... now, I'm going to go back and replace the wood with 5/25" drive rails, but still, I've not heard any complaints from the drives yet.... :-)

Only external?

[larien]

Since Fibre Channel is always found in external drive arrays

Er, no, except perhaps in the Intel world. Sun certainly ships newer servers (280, 480, 880 & 1280) with FC internal disks. Their reason for that (given in their FAQ is that the arbitration for SCSI still takes place at the original 5MB/sec.

Re:Only external?

[hbackert]

Starting at Ultra320, Quick Arbitration is now available.

Given that this is cutting edge of parallel SCSI, I can understand Sun to skip anything older. However I yet have to see a significant performance gain from going from U2W (80 MB/s) to FC. Arbitriation might be slower on U2W, but FC contains routing informations in each packet, which parallel SCSI lacks.

Personally, I am very happy with the good old parallel SCSI. Even cutting edge drives like Cheetah 15k.3 are really fast, even when I cannot push them to their limit.

Article pages / project

[walt-sjc]

Is anyone as annoyed as I am about aticles spreat across 4376245 pages for no fucking reason? Page 2 for example (which is only 2 paragraphs of actual text) uses less than 50% of the page space! You have to scroll WAY past the end of the artlcle to even see all the ad's on the left column.

I guess the fact that there is no normal "print version" link like MOST sites have is the most annoying. There is however a link to a PDF version on the Very Last Page which helps, but html is much prefered.

And what's with the wood blocks that looked like they were cut with a chain saw or hacked apart with an exacto blade? Ever hear of sheet metal? Hell, at LEAST pick up a used DRIVE case instead of a TAPE case. Even NEW they are pretty cheap.

The CONCEPT of the project is interesting, but the implementation leaves MUCH to be desired.

Re:Article pages / project

[ruiner5000]

Yeah, your site is so much better than mine is. You get slashdotted all the time.

Re:Article pages / project

[walt-sjc]

Can you at least give a valid reason for the pagination rather than that a baseless comment (you know NOTHING about sites I run or what they can handle, which is MUCH more than slashdot subscribers could dish out...)

Re:Article pages / project

[ruiner5000]

Yeah, after I digest your 2nd insult. We could put all the content on one page and that would do a great job of dragging the server down as everyone pulls everything. More than one page takes some pressure of the server. Regardless Dave put the article up, and he is new at this. I think you should cut him some slack, and not spout off insults because it is over the internet and not in person and you can.

Re:Article pages / project

[Anonymous Coward]

Banner ads and hit counts. "Hey we served 3,000,000,000 pages this month!"

S'all 'bout the money.

Re:Article pages / project

[walt-sjc]

My turn to digest what you just said. So it's LESS bandwidth to serve ONE request than 12? And how was my response an insult? It's just a fact. You know nothing about the sites I run.

Re:Article pages / project

[jafiwam]

More pages to put ads on I would guess.

The ads totally invalidate any "concerns about bandwith" arguments. Two paragraphs of text or 30 paragraphs of text are going to be way less than even a couple image ads.

So, it's the ads. The pages were broken up to fit more ads.

Re:Article pages / project

[LarsG]

Yeah, after I digest your 2nd insult.

What insult? He merely asked you a question.

We could put all the content on one page and that would do a great job of dragging the server down as everyone pulls everything. More than one page takes some pressure of the server.

Now, *that* is an insult.

Serving 12 pages with lots of ads is supposed to be less pressure than serving 1 modestly larger page?

Re:Article pages / project

[argus333]

ermmm..no? I broke the pages up based on what I thought were logical breaks in the content. I could care less about the ads. I write for the sheer joy of it and I don't get paid for it. So, please, talk to the author (that'd be me) before you go on spouting tripe you know absolutely nothing about... cheers, dave

Re:Article pages / project

[argus333]

hey...as the article's author, I think I can offer my rationale for the pagination...(as I have 15 other places on this thread)... 1.) I'm provided a CMS framework in which to break up my article. The ads are provided by the processes that AMDZone runs, not by my pagination. 2.) I originally wrote the story up in M$ Word (don't shoot me...I'm running openoffice too...) for future placement into PDF format 3.) I tried to decide how best to break the article up...I divided this by thought process... 4.) At the end, as a courtesy, I provided the PDF for the entire article so that users would be able to just look through it that way...If you'll notice, the PDF is approximately 18 pages long...sorry! 5.) I don't make any money on these reviews...I'm not paid at all...so, there's no real motivation for me to break it into as many pages as possible.... does that help any? cheers, dave

from the....

[Anonymous Coward]

...porn storage at the speed of light department

You forgot...

[Ryan Amos]

6) Profit!

Re:You forgot...

[teaserX]

Just like Global Crossing did.

Re:I would like one!

[Anonymous Coward]

>Thanks Mike Harris "...with Unintended Consequences"

I'm trying to figure out if you're talking Mike Harris, ex-Premier of Ontario, Canada, or Mike Harris, some guy you met on the street. Let us know which!

"specialized components"

[Anonymous Coward]

Is it just me or was Fiber Channel made to be as gratuitously incompatible as possible with, well, everything? Six-inch drives? A custom "HSSDC" cable with a plain old DB9 plug on the other end?

fiber, not fibre.

[Anonymous Coward]

fiber.

fibre is pronounced fib-ree.

Re:fiber, not fibre.

[Anonymous Coward]

I agree with this post!

Re:fiber, not fibre.

[Wildcat J]

The AC says:

fiber, not fibre

Maybe someone ought to tell the Fibre Channel Industry Association! They've got it all wrong!

-J

Re:fiber, not fibre.

[conway]

Originally, it was supposed to be "Fiber Channel". But then, to emphasise the fact that you can run it over media other than fiber, the British spelling of "fibre" was chosen instead, so its "Fibre Channel".

Why shouldn't it?

[shoppa]

it can perform up there with 15k SCSI in some cases

Why should this be surprising? FC drives are in every single case SCSI drives with a different, more expensive, interface. Although they tend to be cheaper on the surplus market, which I think is the *real* point.

Wood?!?!

[KingDaveRa]

So, he says the the drives get really hot, so he drills a load of holes for ventilation.

Then he goes and mounts them in the case with wood!! Why? Its an insulator!!! Ok, maybe he didn't have the necessary metal skills (or equipment even) to make a custom bracket, but using wood to mount a drive just seems a bit dangerous to me.

Re:Wood?!?!

[Anonymous Coward]

I always think of mounting when I've got wood....

Re:Wood?!?!

[NedTheNerd]

and then he could have wrapped it in foam rubber because they where "loud" hehe :)

Re:Wood?!?!

[KingDaveRa]

Why not go all the way and lock it in a cupboard, solves all the problems then!

Re:Wood?!?!

[Nihilanth]

why stop there? I had the idea, long ago, of building a computer inside a refrigerator, with sealed holes for the wires to connect to the interface devices (this was before i heard of refrigerated cases)

Re:Wood?!?!

[KingDaveRa]

Nah. Been done. I remember a site, cult of the llama or something they were called, i forget, anyway, they did it. I think it may have been a faked thing just for laughs.

Still, its a good idea.

Re:Wood?!?!

[argus333]

gee, thanks! love your work too... actually, my drives are running rather cool (thanks for asking)...The ventilation holes provide a passive means of cooling. If heat rise, it needs a means to escape. The drives aren't loud and frankly, I could care less... repeat after me: It was a fun project...

Bah.

[Anonymous Coward]

I was interested until this point:

Turn on your computer (yes, without attaching the drives) and wait for Windows to load up. Note: for best results, use Windows 2000. Since the QLA2100 is discontinued, the last driver revision is for W2K only. I'm using .NET Server RC2, so I can attest that it works, although I'm using Microsoft's un-optimized driver.

Re:Bah.

[argus333]

and? I'm not very experienced with Linux, et al. Would you care to come over and give me a tutorial? I can appreciate that you don't like M$. Me either...problem is, until I actually have time to sit down, fool around with any flavour of *nix, you're not going to see any reviews written on that platform... lo siento mucho...

Mmmmm ... Disk Space

[AlabamaMike]

Can you say do-it-yourself enterprise disk array? Anyone want to give EMC a run for their money? Could this be the future of Open Source enterprise hardware? I can see the press report now (from a storage company): "Building your own disk array flies in the face of all capitalistic principles ... In fact, it's down right communistic!"

a Hoax story

[stock]

When i follow the links for the prices of his used components i get the following :

• 18.4GB Fibre Channel 3.5LP 10K RPM 25.4MM (Hitachi) - (DK32CJ18FC) $119.00
• QLA2100/66 64bit PCI FC Host Adpt COPP w/Cab (Qlogic) - (QLA210066) $858.20

So where did he get those goodies? Ohh! he was just good buddies with a ex- Enterprise storage Admin?

Robert

Re:a Hoax story

[AlabamaMike]

I used to work for a Managed Service Provider, and let me tell you, they sold off thousands of pieces of IT gear for pennies on the dollar. I'm sure this is nowhere near a unique situation, as I know someone who has a warehouse full of hardware bought from dead or dying companies (no exaggeration, a whole freekin warehouse.) Maybe someone was having an IT yard sale (or perhaps it was five fingered merchandise.)

Re:a Hoax story

[argus333]

got them off of Ebay...the store that routinely sells the drives for extremely cheap is HDOutlet. (look 'em up...) As for the HBA, I can't remember where I got it, but several on ebay this past weekend were going for 35.00 cheers, dave

Re:I would like one!

[sirwired]

Optical fiber is not required for shorter distances, however, because Fibre Channel also works using coaxial cable and ordinary telephone twisted pair. Fibre Channel offers point-to-point, switched, and loop interfaces.

Too bad whoever wrote that was completely full of it. Fibre Channel does not exist over coax or twisted pair.

Coax really isn't used much for data cabling any more.

Twisted Pair? Who knows. I expect the error rates are too high for the extremely tight specs required by Fibre Channel.

SirWired

Both are acceptable

[donert]

From an english langauge pov:

http://dictionary.reference.com/search?q=fibre

From a google pov:
"fibre channel array" hit count = 2600
"fiber channel array" hit count = 213

What a waste

[Anonymous Coward]

Why put all of 2 disks into an ugly assed old compaq case with those shady adapters when you can go buy a used 11,14 or 22 disk fibre-channel array with redundant power and dual loops made by a certain manufacturer (hey I ain't givin up all my secrets) for well less than $500 empty and around $650 with ~180GB 10K disks in it?! And yes of course they do make FC cards with internal adapters on them too. Here's a hint: SENA. As for FC not having performance, all I can say is 'HUH?!' I'll take a single 1gbps loop over scsi320 parallel or whatever they're calling it any day. Beyond my own benchmarking FC devices, if SCSI were better/faster don't you think people like EMC, HDS and Compaq (believe it or not Compaq makes some pretty kickin arrays) would still use SCSI back ends or even front ends on their storage products? For years EMC has been slammed about using SCSI back ends in their arrays and finally have FC throughout the machines. FC is saweeet and it runs SCSI above the FC layer as well. It can also run other protocols like IP but I've yet to see that implemented well.

Re:What a waste

[RedX]

I'll second those sentiments regarding Compaq's StorageWorks line. They might not yet be able to compete with EMC and HDS on the enterprise level, but their midrange stuff can compete with anybody when comparing best bang for the buck. Their Enterprise Virtual ArrayAnd you're killing me with your SENA hint. I've been dying to find some FC hardware to play with at home, but am not finding any cheap sources for SENA hardware.

Re:What a waste

[argus333]

1.) i put them in the case because I could. I've also got an old CDROM server tower that I'm going to transistion the drives to. 2.) the "shady" adapters you're talking about are actually extremely well fabricated and thought-out. I wouldn't trade mine for the world...

Re:I would like one!

[Anonymous Coward]

Coax really isn't used much for data cabling any more

Tell that to the umpteen thousands of users on my cable broadband ISP :P

SCSI vs. Fiber Channel

[snowtigger]

Where I used to work, we had a few Sun servers with FC disk arrays.

Here's the Sun engineer's explanation of why FC is so interesting for servers:

1) The FC protocol has a 100MByte/s dedicated bandwidth to data. The communication between disks etc. will not interfer with this bandwidth.

2) Modern SCSI has two modes: one for data (burst mode) and one communication mode. The communication mode is a lot slower (first scsi standard) in order to remain compatible with older disks. This means that scsi is a lot more advantageous to users reading large files than small files.

This is where FC becomes interesting: If you have a striped disk array, you will read many small segments from different disks instead of large segments from single disks. In this special case, FC is faster than SCSI, even though it is "slower" by looking at the burst rates in the specs.

Re:SCSI vs. Fiber Channel

[Brento]

If you have a striped disk array, you will read many small segments from different disks instead of large segments from single disks. In this special case, FC is faster than SCSI, even though it is "slower" by looking at the burst rates in the specs.

Uhhh, that's called RAID, and you can do it without FC. You can do RAID striping with IDE or SCSI hard drives that outperform a single SCSI drive.

Re:SCSI vs. Fiber Channel

[snowtigger]

You're missing the point.

Of course you can do raid with ide, scsi or whatever that outperforms a single disk. However, all disks sharing a bus also shares the total bandwidth of that bus.

What I'm trying to say is that when you have a stripe (=raid 0 or 0+1), FC is faster than SCSI because of the way the communication protocols work.

Re:SCSI vs. Fiber Channel

[Anonymous Coward]

If you can afford FC, you can certainly afford a controller channel for each drive. Busses are only appropriate when you have to give up performance in return for cost or space.

A SCSI array that'll beat their FC array

[Anonymous Coward]

From Tony's Liquidators:

Adaptec 29160 - $2
Drive cases - $1
72G 15K SCSI - $1

I get 2 cards, 2 cases, and 8 drives. Cost: $14.

Works with Linux--YAY!--but Tony said something about me going out with his sister...I hope he was talking about Mary instead of Angela, Mia, Theresa, Donna, or Rosalie.

He has a *lot* of cousins too so this is *by no means* a one-time deal.

amdzone.com

[Anonymous Coward]

Too bad this site doesn't render correctly with Mozilla 1.3 for Windows. The pics don't show.

Re:amdzone.com

[argus333]

damn, i'll try to remember that...I love mozilla, but at work, I'm always stuck using IE. thanks for the reminder.... cheers, dave

Gee...

[Lysol]

Has anyone looked at the prices from the story? I could not find a $25 FC drive anywhere. Cheapest was $120. So, yah, $240 for 2 drives, that kinda blows the under $250 out of the water.

Anyway, being a 'mac head' and a 'linux head' and a 'computer head' in general, I don't necessairly get the orig posters issue. Apple does a lot of cool things. And some things just happen to hit the Mac market before the PC market. Big deal. Why not bitch about Billy G. then?

My work has been setting up a FCA for the past three weeks using Linux and there have been some major problems. They have a fat array with 32 15k rpm U320 drives hooked up to a IBM x440 via 4 HBAs. The interesting thing is that no distro they've tried can transfer faster than Windoze due to Linux kernel and driver issues. I was a little shoked. The x440 has 8 Xeons w/hyperthreading. The more cpus that are enabled, the more the performance degrades. The sysadmin says he thinks it has something to do with single-threaded io calls in all Linux kernels - the more cpus try to access io, the more threads that get blocked. Me and the other sysadmin - Gentoo 'heads' - start scratching our heads wondering what all the Linux 'Enterprise' stuff is that everyone is talking about. And yes, these components were all given the 'good to go' stamp by all their manufacturers. Since the prob is with the kernel itself, this is kinda major.

So, these things are cool, but there are definitely reasons for and against having them. I doubt you could build anything useful for under $400-$500 and they're really only going to shine in a server or workstation environment whith sustained io. It's nice to have these things as proof of concept at home (hell, I have all kinds of weird servers and drives and such), but when you start talking about a production environment in the enterprise, it takes on a whole new ball of wax.

I still think the XServe is interesting, however, I'm not too thrilled about ATA drives. Guess that was for cost tho. Go mac heads!

Re:Gee...

[Master Bait]

My work has been setting up a FCA for the past three weeks using Linux and there have been some major problems. They have a fat array with 32 15k rpm U320 drives hooked up to a IBM x440 via 4 HBAs. The interesting thing is that no distro they've tried can transfer faster than Windoze due to Linux kernel and driver issues. I was a little shoked. The x440 has 8 Xeons w/hyperthreading. The more cpus that are enabled, the more the performance degrades. The sysadmin says he thinks it has something to do with single-threaded io calls in all Linux kernels - the more cpus try to access io, the more threads that get blocked. Me and the other sysadmin - Gentoo 'heads' - start scratching our heads wondering what all the Linux 'Enterprise' stuff is that everyone is talking about. And yes, these components were all given the 'good to go' stamp by all their manufacturers. Since the prob is with the kernel itself, this is kinda major.

I went over to IBM's website and it sems to me that all the 6 PCI-X slots on these machines share the same bus, so it isn't going to matter if your kernel has multi-threaded I/O (is there such a thing?) or not.

I think that those 4 separate adapters controlling the drives are going to incurr 4 times the PCI overhead (grab the bus X4, send the data, release the bus X4) as a single adapter controlling all 32 drives would. And I think you would hit your I/O peak with one or two CPUs, assuming you are testing under no-load conditions.

Re:Gee...

[hansendc]

I went over to IBM's website and it sems to me that all the 6 PCI-X slots on these machines share the same bus, so it isn't going to matter if your kernel has multi-threaded I/O (is there such a thing?) or not.

That is utterly wrong. If you look IBM's site, you'll see:

Six Active(TM) PCI-X slots standard

• Allows you to hot-add and hot-swap PCI and PCI-X adapters on the fly
• The latest in PCI-X performance: 2 slots at 133 MHz, 2 slots at 100 MHz, and 2 slots at 66 MHz per chassis

It's pretty obviously 3 busses.

Re:Gee...

[hansendc]

The x440 has 8 Xeons w/hyperthreading. The more cpus that are enabled, the more the performance degrades. The sysadmin says he thinks it has something to do with single-threaded io calls in all Linux kernels - the more cpus try to access io, the more threads that get blocked. Me and the other sysadmin - Gentoo 'heads' - start scratching our heads wondering what all the Linux 'Enterprise' stuff is that everyone is talking about.

The x440 is a NUMA machine. It takes special support added into the kernel to make it work well. This support is being improved daily, and works best when you're running 2.5.

Re:I would like one!

[Anonymous Coward]

Too bad whoever wrote that was completely full of it. Fibre Channel does not exist over coax or twisted pair.

Too bad you're full of shit and your little outburst will make you look like a moron.

Here's one example, here's another and here is yet another.

According to Google, there are about 5,500 pages that disagree with you.

Here's one...

[Anonymous Coward]

Not the exact controller he spoke of, but pretty close - just search for "QLogic" with no other qualifiers.

Here you go!

$250 my ass, try $1000+

[Pharmboy]

Has anyone looked at the prices from the story? I could not find a $25 FC drive anywhere. Cheapest was $120. So, yah, $240 for 2 drives, that kinda blows the under $250 out of the water.

Yea, the $40.99 host is actually $858, pushing the whole project into the 'well over a grand' territory. I will blow 250 to enjoy working with a cool technology, even if i dont have an immediate need, but for a grand i can buy several nice u160 drives and stripe them since i dont 'need' FC.

I found the whole article pretty over simplified (and really bad blinking/flashing ads) Quoteth the comic book guy: Worst article. Ever.

Ok, not that bad, but not that good either.

I think you're right

[Anonymous Coward]

I'm going to start following your excelent example with my Slashdot comments.

Make money fast! Click here now!

I think it will help to improve the quality of my comments, and bring in more readers.

Your computer is not optimised. Optimise now!

However I'm a little concerned that people may find it difficult to follow my posts if I keep breaking them up with adverts and links.

Naked cheerleaders!

I guess it might also be a problem for users on high latency links.

Get your University diploma. Act now!

Who am I kidding? Fuck um, I'll just milk a single post for 6 page impressions per reader and overload it with adverts, animated GIFs and other shit. All I need to do is work out how to make Slashdot accept blink tags and embedded Flash, I can be just as leet as your site is every day!

Adverts got you down? Want content? Well we can't help!

Re:I think you're right

[argus333]

Again, (and again, and again...ad nauseum...) I paginated the article based on what I felt were appropriate breaks in the article. If you have a problem with this, let me know and I'll try to work things out a bit better next time...I just hate looking at 15 screens (in length) and trying to put some coherency together...I'd rather slap a PDF up... cheers, dave

Re:I think you're right

[Anonymous Coward]

I don't get it. Why do people want to mouse to fifteen links/buttons (and wait for fifteen HTTP responses) instead of just tapping the space bar fifteen times?

Re:I think you're right

[argus333]

tell ya what, I'll give you heads up the next time I'm ready to post an article and you can "debug" it for me.... some people DETEST single page reviews...other's like the multi-paged approach... Personally, I'd rather have the PDF... cheers, dave

Nice But ......

[Quietlife2k]

Anyone seen ide to fibre channel convertors ? Before you flame - They DO exist this product has them in it see here :- http://www.axus.com.tw/br1200fc.htm Anyone seen single drive versions of this ?

I built a FC array last year...

[nuxx]

I made a Fibre Channel array like this last year. The only difference being that I used a Mylex eXtremeRAID 3000 (eBay for $200), a 256MB Crucial DIMM for cache, and four Seagate ST39102FC 9GB 10,000 RPM disks.

My whole point to the project was EXTREMELY fast disk access (up to ~160MB/sec sustained transfers, see here) that I could locate at the far end of a REALLY long cable. I've got my machine in my office and the hard drives on the other end of a 30m cable, nestled nicely down in the basement where I cannot hear it.

There are a few basic pictures of the external assembly available here. Works really, really well. It's amazing what hugely fast disk IO does for the rest of a machine.

Shut up, idiot.

[LibertineR]

"Fibre Channel does not exist over coax or twisted pair."

There are hundreds of examples of you being dead wrong on this. You are full of shit.

Re:Shut up, idiot.

[sirwired]

Well I'll be damned... The spec does indeed spell out how to build w/ coax and twinax. I work in storage and I had not ever seen anything but Fibre and HSSDC/DB-9 in the field. (I meant RJ-45 terminated when I said twised pair. I know that HSSDC/DB-9 cables are copper.) I have yet to actually FIND a RJ-45, Coax or Twinax GBIC/SFP though. Where would one find these gems? RJ-45 GBICs are easy to find, but I don't think they are cleared for Fibre Channel, just GigE.

eBay prices

[barryp]

Read the 2nd page of the article again..

Below is the equipment list and prices that I compiled from 2 sources: sca40.com and Ebay

I looked on eBay, and yeah, there is surplus stuff out there at those prices. The brand-new stuff is what you were finding prices for.

Re:eBay prices

[argus333]

geez, ya think if they could get on slashdot, they'd be able to read the first time, eh? :-) You're definitely looking at NEW prices, not ebayed...big difference...

drool.

[uidzer0.org]

i am the only one who drools at the thought of fiber?

WOW

[deadsaijinx*]

Imagine a Beowulf Cluster of these ^^

Sorry, couldn't resist.

FibreChannel is not 1Gb anymore

[britt]

FibreChannel has has a 2Gb mode for at least a year now with products from many vendors.

Re:FibreChannel is not 1Gb anymore

[argus333]

right, and as such, it's still more expensive...I'm working on getting a QLA2300 to test out FC-2 cheers, dave

Not with these drive modules...

[Diesel Dave]

Aside from being a whiny prick that tried to cause trouble for competing fibre channel product vendors on Ebay, Sanden Fuess's products are designed so far out of spec it's sad. They may be cute as a hand assembled hacker novelty, but I'd never put them into any type of production environment.

General rule: If it doesn't AC decouple and doesn't actively terminate, or it does not use 0603 or *smaller* surface mount components, don't buy it. And, if it doesn't use shielded cables, laugh at the vendor! If any of the above are missing you will have retiming issues causing intermittent failures.

I designed the Cinonic FC2's. Alass the market for 'mid-range' FC has all but died, so we never re-uped our stock when we ran low a few months ago. : (

Re:Not with these drive modules...

[Anonymous Coward]

oh, that's right ... sorta how like you founded, not found, cinonics?

Re:Not with these drive modules...

[Diesel Dave]

No, more like I have a clue what I'm doing and he doesn't.

After 10 years of pretty much mass use of BBS'es and Internet systems you'd think spelling flames would have died out by now. Between the spam and dregs like this AC the net got killed. Thank you AOL.

The Internet -Died 1999 - R.I.P

Re:Not with these drive modules...

[argus333]

Hey! I'll be more than willing to test your product out...problem is, the folks that I talked to about it (not Sandin, btw) mentioned that you've had some QC problems in the past... I'm new to the block, so, email me and I'll do a head to head review...fair? cheers, dave

Re:Not with these drive modules...

[Anonymous Coward]

i have had the pleasure of using the cards from Cinonic. Quite frankly they suck. Any attempts at using three or more in a loop fails. Your attempts at impedance matching has left you with some severe reflection issues. This has been duplicated by two others over at 2CPU that I am aware of. not to mention your method of cabling 2 or more together is a major pain in the arse, trying to deal with short sections of shielded cat5e cables and bend them to the next card always results in the tcard being pulled from the drive. As for the tcards listed in this review, I have now a total of 10 in one loop and have not had any issues at all for well over a year. I feel much more secure than with any IDE implementation, and have built an impressive array at a fraction of the cost of SCSI.

From the Trenches

[LordMyren]

Having built my own fiber channel backplanes based of a previously slashdot mentioned cinonic backplane, I agree, there are difficulties in setting up a fibre channel array. I got a db9-db9 cable and a hssdc-hssdc cable orgiinally, and ended up trying to solder them together. I'd never tried soldering shields together, and presumed it would take a while, but I kinda found out its pretty much impossible. So I just pulled on over the other and wire wrapped the hell out of it. I suspect its because of the cabling, but my fibre setup is fairly intollerant to electrical fields. I have to place the drives and cabling as far away from everything; cat 5, computer systems, et al. Kinda not pretty. Before proper positioning, you'd get occasional sometimes fatal SCSI errors. Kernel panics were driving me nuts. I realized the drive was between a switch and a computer, so I spent a while and figured out the path of least interference. Now I can run bonnie++ on my 5 disc soft-raid 5 JBOD for a week straight. And fry eggs as the same time! And no more kernel panics. Thats always a plus. Its a pain, but in the proper linux motto, once you can get it running, you can bet its not gonna stop.

Re:From the Trenches

[Anonymous Coward]

Now I can run bonnie++ on my 5 disc soft-raid 5 JBOD for a week straight.

For a week? How does that sync with "Its a pain, but in the proper linux motto, once you can get it running, you can bet its not gonna stop."?
Gosh, you Linux admins have such lofty goals for stability.

Re:From the Trenches

[LordMyren]

it was only under the very infrequent heavy duty usage that the system used to fail. a week of pure heavy usage is enough for me to say good enough.

Solutions

[LordMyren]

Are there any free solutions for file locking?

From the Trenches

[LordMyren]

Having built my own fiber channel backplanes based of a previously slashdot mentioned cinonic backplane, I agree, there are difficulties in setting up a fibre channel array.

I got a db9-db9 cable and a hssdc-hssdc cable orgiinally, and ended up trying to solder them together. I'd never tried soldering shields together, and presumed it would take a while, but I kinda found out its pretty much impossible. So I just pulled on over the other and wire wrapped the hell out of it.

I suspect its because of the cabling, but my fibre setup is fairly intollerant to electrical fields. I have to place the drives and cabling as far away from everything; cat 5, computer systems, et al. Kinda not pretty.

Before proper positioning, you'd get occasional sometimes fatal SCSI errors. Kernel panics were driving me nuts. I realized the drive was between a switch and a computer, so I spent a while and figured out the path of least interference. Now I can run bonnie++ on my 5 disc soft-raid 5 JBOD for a week straight. And fry eggs as the same time! And no more kernel panics. Thats always a plus.

Its a pain, but in the proper linux motto, once you can get it running, you can bet its not gonna stop.

[properly formatted repost from a moron. sorry]

Buddy, you need to be put in your place

[I Am The Owl]

You have no idea what a high traffic website is. None. I work on the IT staff of a certain Fortune 100 company, and I can tell you that

1. Slashdot traffic (forget about just link clickthroughs) is piddly compared to what we see on a regular basis (you should've seen the logs on Superbowl Sunday!). Slashdot is not special. You are not special.
2. We have a business model more concrete and certainly less offensive than "throw up 20 obnoxious banners and make readers look at things 2 paragraphs at a time".
3. Your site is ugly.
4. You are ugly.
5. It takes more bandwidth to serve up 14 requests than it does to serve up one moderately larger one.

But such is the life of a hardware fanboy, eh?

Re:I would like one!

[britt]

FibreChannel over copper is quite common.

The usual connector types are HDSC (?) and DB9 (though this only uses 4 pins)

Copper makes a lot of sense for many fibrechannel uses. Connect multiple disk shelves together is the most common, since this only requires 1 foot cables, and copper is much more sturdy than optical, and lower cost for the connecters/interfaces.

Obviously there are limits to what copper can do, the most noticable is the 30M limit.

Test Cards

[Anonymous Coward]

I'd love to make a FC array but the test cards are way too expensive. Does anyone happen to havea schematic for how to make one?

Re:I would like one!

[sirwired]

Oops... Yes, I know about Fibre over copper. When I said "Twisted pair" I meant RJ-45 terminated TP. I know all about HSSDC and DB-9 connectors.

Re:I would like one!

[sirwired]

Yeah, and there is a spec for CSMA (i.e. passive hubs) for Gigabit Ethernet. Does it exist in actual products? No. Coax is specified in the Fibre Channel Specs, yet I have yet to see a single Coax or Twinax GBIC/SFP or HBA. If you find one, let me know.

When I said Twisted Pair, I meant RJ-45 terminated. Oops. I do know about the DB-9 and HSSDC copper cables.

Oh so offtopic...

[Darth Hubris]

Fiber. Fibre. Whatever. Fibre looks cooler. So do a lot of the Queen's English spellings, colour, flavour, armour, etc.

As long as we get Cathy Rogers to say knickers, bum, wibble, wobble, semprini...

Last Post!

[alpg]

THE LESSER-KNOWN PROGRAMMING LANGUAGES #14 -- VALGOL

VALGOL is enjoying a dramatic surge of popularity across the
industry. VALGOL commands include REALLY, LIKE, WELL, and Y*KNOW.
Variables are assigned with the =LIKE and =TOTALLY operators. Other
operators include the "California booleans", AX and NOWAY. Loops are
accomplished with the FOR SURE construct. A simple example:

LIKE, Y*KNOW(I MEAN)START
IF PIZZA =LIKE BITCHEN AND
GUY =LIKE TUBULAR AND
VALLEY GIRL =LIKE GRODY**MAX(FERSURE)**2
THEN
FOR I =LIKE 1 TO OH*MAYBE 100
DO*WAH - (DITTY**2); BARF(I)=TOTALLY GROSS(OUT)
SURE
LIKE, BAG THIS PROGRAM; REALLY; LIKE TOTALLY(Y*KNOW); IM*SURE
GOTO THE MALL

VALGOL is also characterized by its unfriendly error messages. For
example, when the user makes a syntax error, the interpreter displays the
message GAG ME WITH A SPOON! A successful compile may be termed MAXIMALLY
AWESOME!

- this post brought to you by the Automated Last Post Generator...