I totally understand. I realize that the IT-people-are-power-drunk-jerks stereotype didn't appear out of thin air. I try my hardest to surprise my fellow employees by actually being nice, helpful, and non-autocratic.
As for attaching spreadsheets and presentations to emails and then forwarding them to fifty people instead of copying them to a public directory, I'm completely with you.
Also, any company that is in any way interested in reducing energy costs would ask their employees to turn off computers when not in use!!
We do. Many of them don't. And I'm not so draconian as to force the PCs to shutdown at a certain time.
This means that any data you store on these would probably only be sporadically available between 9AM and 4PM (or whenever a quorum of computers would be operational.
Everyone assumes I wanted to use this pool for live, high-availability storage. What about using it for a versioning backup system? Archiving? There are dozens of other uses which don't require the pool to be available 24/7/265. The other issues I've addressed elsewhere. They're all surmountable with relatively common distributed computing approaches. The problem is, I don't have time to write the software myself, but I guessed other people had thought of this and might suggest a product that already exists. Some did just that. Most just complained about how it didn't make sense and can't be done.
I think the biggest point everyone is missing here is how do you upgrade the desktops once their part of your storage network ? This is the type of added complexity that makes this solution just not worth it.
The reason I mentioned fault-tolerance in TFQ is because I am not so ignorant as to think you'd just add all this space together and present it as one big share to put files on. Since all this is currently wasted space, there's little penalty for high-order redundancy. I should be able to pull three or four PCs right out of the wall and still see a healthy pool that has started some peer-to-peer activity to bring the duplicity of any data lost on those nodes back to some safe level.
I am shocked at utter lack of imagination among many of the posters, and how many of them assume I'm an idiot who hasn't thought about even the simplest aspects of this (not yourself, specifically, but just in the comments as a whole).
Trust me, I would love nothing more than to have thin clients on everybody's desk. It would be hard enough to convince upper management to let me replace PCs with thin clients if thin clients cost less to purchase, but when a cheap HP business PC is $375 and their thin client costs just as much if not more, it's impossible. Add that to the licensing costs for Terminal Server or Citrix, and forget it.
No, not as in "Product XYZ was designed to do just that! Just install it on each machine, set up your pools, and forget about it". Lots of links to projects that sorta-do-that-kind-of-distributed-something-or-other-but-on-linux-or-on-Windows2K3Server-or-, but like you said, no straight-forward answer.
It's rather baffling how many replies are "why would you want to do that?", "That's stupid.", "Buy an iSCSI SAN thing you idiot!". They just don't get it, and it seems so simple to me. The peer-to-peer nature of BitTorrent, combined with on-the-fly encryption, combined with the virtualized and redundant approach to storage of ZFS, and you've got all the pieces. Just nobody seems to have put it together in one project or product. I'd write it myself if I had six months of free, my-bills-are-paid time.
The most obvious application for it (at least in a corporate setting) is online backups and highly static data - both of which essentially circumvent the writing problem by very infrequent modification and only allowing writing under carefully controlled circumstances. EXACTLY! Finally somebody who gets why this might be useful. Thank you.
You know, I'm no tree-hugging environmentalist and I'm as guilty as the next guy of buying all kinds of stuff I don't really need, but even I realize there comes a point when "buying more" isn't the best answer. If we could use all this space, maybe I wouldn't need to buy a fancy storage array. And the power to run it. And the drives to stick in it. And the place to put it. And all the garbage generated when they built it. And the room it's gonna take in the landfill when its time is done.
What if I said, "I have a found way to magically extract all the gasoline sitting in gas tanks in junked cars, and now I can give every one with a car a free tank of gas." You'd surely raise your hand to get your tank filled. Sure, doing so is nearly impossible and completely impractical. But in the case of pooling unused desktop PC storage to use on the network, I know it's far from impossible, and with the right software, could even be practical. Otherwise I wouldn't have asked the question.
It's a company. There's nothing we desperately need the space for, but I was thinking about how to expand our present backup solution once we've exceeded its capacity. Here I am pricing out off-site backup services, tape robots, SAN stuff, more drives for the servers, RAID boxes, etc., and it just dawned on me that we'd purchased 40 new PCs with big hard drives that won't ever have more than a few gigabytes of data on them. I've been encouraged by some of the positive, helpful replies here, but I am overall discouraged at the thought processes of the typical Slashdotter involved with IT. It's always "not worth it", "buy a SAN solution", "stupid idea". These are some of the most think-inside-the box people I've ever encountered.
The following is not specifically aimed at you, but at all the naysayers:
YES, I know each client can be turned off, burn up, be hacked.
YES, I know this isn't a good idea for high-availability, high-bandwidth storage.
YES, I know they should all be thin client terminals running Linux and booting off a server. Welcome to the real world.
YES, I know it sounds crazy when you can just buy yet ANOTHER terabyte drive from Seagate when you need more storage. But why should I do that when I know full well the know-how exists to combine all this unused space in a highly-redundant, fault-tolerant manner? Hell, I could write the software to do it myself if I had six months of nothing else to do.
Thus, the reason I asked the question, figuring I couldn't be (and some comments prove me right) the only one to have thought of how to use all the wasted storage out there. And if others have thought of it, maybe they've found solutions. I thought that's what "Ask Slashdot" was all about.
I know that for versioning, backup, and archiving uses, all this space is available and could be made useful with the right software--software that combines the approaches of BitTorrent, ZFS pooling, encryption, etc. I know it's possible and I know it would be useful to lots of companies with the same setups.
You can hardly buy smaller drives if you wanted to. That's really what started me thinking about this. I was worried about how to provide backups once our present solution is at capacity, and I was thinking "We just bought 40 new PCs with 80GB drives and I'm here pricing out ways to add another dozen gigabytes to our backup system. We don't have a budget (nor really a need for) big fancy SAN solutions and iSCSI and tape robots and all that junk. Reading some of the comments in this thread make me realize how stuck in the '80s and '90s most IT folks are as far as philosophies and approaches to problems.
We sure don't have much faith in software do we? I don't have time on my hands, which is why I am envisioning a bulletproof "load it, configure it once, and let 'er rip" solution. If you consider what's accomplished with BitTorrent and other peer-to-peer architectures, along with the fairly hardware transparent storage pooling provided by ZFS, and encryption, all the pieces exist to make this work. This would not be for high-availability data. More like a way to have the network back up to itself, or provide versioning as another poster suggested.
You're telling me that if you had a service you could load on each client machine that, with a few minutes of setup, would provide a pool of highly-redundant storage, you wouldn't use it?
Way to jump to conclusions about me and how I manage a network. I honestly didn't ask the question as a "control freak", I don't spy on the employees, and I don't play Internet cop. I try to get them the tools they need to do their jobs, help them when things don't work, and otherwise stay out of their way. I also didn't imply the pool would be for me to do with as I please; I can see several ways in which that storage would benefit our business were it not spread out in small chunks. The users have all that space, and they simply DO NOT use it. In our business, they don't have much call for large files like photos, movies, etc. It's mostly spreadsheets and OpenOffice Writer documents. But thanks for being an ass.
Switchfoot makes better music than Korn, too, but such opinion is no more revolutionary than the one in the article. Ethanol IS NOT the cure for our energy disease.
the Spurious Engine! And maybe now malware purveyors will learn how to program for the Cell (since nobody else has), leaving your Core Duo free to run the browser that's being hijacked and a few UAC dialogs.
A steam plant on the ground actually spins, in the US, at a constant 60rpms when all is said done, that's how we get power at 60hz. Actually, US utility steam turbines usually spin at either 1800 or 3600 RPM. 60 RPM (Revolutions per MINUTE) would not yield 60 Hz (Cycles per SECOND) alternating current.
Your shameless plug just sold at least one of your CDs. And ignore the AC's comment--they obviously didn't give your music a listen. You are most certainly a musician. I, however, am a drummer, which just means I hang out with musicians. *BADDA BOOM*
Slashdot Mirror
I totally understand. I realize that the IT-people-are-power-drunk-jerks stereotype didn't appear out of thin air. I try my hardest to surprise my fellow employees by actually being nice, helpful, and non-autocratic. As for attaching spreadsheets and presentations to emails and then forwarding them to fifty people instead of copying them to a public directory, I'm completely with you.
We do. Many of them don't. And I'm not so draconian as to force the PCs to shutdown at a certain time.
This means that any data you store on these would probably only be sporadically available between 9AM and 4PM (or whenever a quorum of computers would be operational.Everyone assumes I wanted to use this pool for live, high-availability storage. What about using it for a versioning backup system? Archiving? There are dozens of other uses which don't require the pool to be available 24/7/265. The other issues I've addressed elsewhere. They're all surmountable with relatively common distributed computing approaches. The problem is, I don't have time to write the software myself, but I guessed other people had thought of this and might suggest a product that already exists. Some did just that. Most just complained about how it didn't make sense and can't be done.
192.168.1.100, .101, .102, .103... And port 546 is open on 'em all. (+1 You Rock if you get why that's funny.)
Cheers!
The reason I mentioned fault-tolerance in TFQ is because I am not so ignorant as to think you'd just add all this space together and present it as one big share to put files on. Since all this is currently wasted space, there's little penalty for high-order redundancy. I should be able to pull three or four PCs right out of the wall and still see a healthy pool that has started some peer-to-peer activity to bring the duplicity of any data lost on those nodes back to some safe level.
I am shocked at utter lack of imagination among many of the posters, and how many of them assume I'm an idiot who hasn't thought about even the simplest aspects of this (not yourself, specifically, but just in the comments as a whole).
Trust me, I would love nothing more than to have thin clients on everybody's desk. It would be hard enough to convince upper management to let me replace PCs with thin clients if thin clients cost less to purchase, but when a cheap HP business PC is $375 and their thin client costs just as much if not more, it's impossible. Add that to the licensing costs for Terminal Server or Citrix, and forget it.
No, not as in "Product XYZ was designed to do just that! Just install it on each machine, set up your pools, and forget about it". Lots of links to projects that sorta-do-that-kind-of-distributed-something-or-other-but-on-linux-or-on-Windows2K3Server-or-, but like you said, no straight-forward answer.
It's rather baffling how many replies are "why would you want to do that?", "That's stupid.", "Buy an iSCSI SAN thing you idiot!". They just don't get it, and it seems so simple to me. The peer-to-peer nature of BitTorrent, combined with on-the-fly encryption, combined with the virtualized and redundant approach to storage of ZFS, and you've got all the pieces. Just nobody seems to have put it together in one project or product. I'd write it myself if I had six months of free, my-bills-are-paid time.
You know, I'm no tree-hugging environmentalist and I'm as guilty as the next guy of buying all kinds of stuff I don't really need, but even I realize there comes a point when "buying more" isn't the best answer. If we could use all this space, maybe I wouldn't need to buy a fancy storage array. And the power to run it. And the drives to stick in it. And the place to put it. And all the garbage generated when they built it. And the room it's gonna take in the landfill when its time is done.
What if I said, "I have a found way to magically extract all the gasoline sitting in gas tanks in junked cars, and now I can give every one with a car a free tank of gas." You'd surely raise your hand to get your tank filled. Sure, doing so is nearly impossible and completely impractical. But in the case of pooling unused desktop PC storage to use on the network, I know it's far from impossible, and with the right software, could even be practical. Otherwise I wouldn't have asked the question.
It's a company. There's nothing we desperately need the space for, but I was thinking about how to expand our present backup solution once we've exceeded its capacity. Here I am pricing out off-site backup services, tape robots, SAN stuff, more drives for the servers, RAID boxes, etc., and it just dawned on me that we'd purchased 40 new PCs with big hard drives that won't ever have more than a few gigabytes of data on them. I've been encouraged by some of the positive, helpful replies here, but I am overall discouraged at the thought processes of the typical Slashdotter involved with IT. It's always "not worth it", "buy a SAN solution", "stupid idea". These are some of the most think-inside-the box people I've ever encountered.
The following is not specifically aimed at you, but at all the naysayers:
YES, I know each client can be turned off, burn up, be hacked.
YES, I know this isn't a good idea for high-availability, high-bandwidth storage.
YES, I know they should all be thin client terminals running Linux and booting off a server. Welcome to the real world.
YES, I know it sounds crazy when you can just buy yet ANOTHER terabyte drive from Seagate when you need more storage. But why should I do that when I know full well the know-how exists to combine all this unused space in a highly-redundant, fault-tolerant manner? Hell, I could write the software to do it myself if I had six months of nothing else to do.
Thus, the reason I asked the question, figuring I couldn't be (and some comments prove me right) the only one to have thought of how to use all the wasted storage out there. And if others have thought of it, maybe they've found solutions. I thought that's what "Ask Slashdot" was all about.
I know that for versioning, backup, and archiving uses, all this space is available and could be made useful with the right software--software that combines the approaches of BitTorrent, ZFS pooling, encryption, etc. I know it's possible and I know it would be useful to lots of companies with the same setups.
Fin.
You can hardly buy smaller drives if you wanted to. That's really what started me thinking about this. I was worried about how to provide backups once our present solution is at capacity, and I was thinking "We just bought 40 new PCs with 80GB drives and I'm here pricing out ways to add another dozen gigabytes to our backup system. We don't have a budget (nor really a need for) big fancy SAN solutions and iSCSI and tape robots and all that junk. Reading some of the comments in this thread make me realize how stuck in the '80s and '90s most IT folks are as far as philosophies and approaches to problems.
We sure don't have much faith in software do we? I don't have time on my hands, which is why I am envisioning a bulletproof "load it, configure it once, and let 'er rip" solution. If you consider what's accomplished with BitTorrent and other peer-to-peer architectures, along with the fairly hardware transparent storage pooling provided by ZFS, and encryption, all the pieces exist to make this work. This would not be for high-availability data. More like a way to have the network back up to itself, or provide versioning as another poster suggested.
You're telling me that if you had a service you could load on each client machine that, with a few minutes of setup, would provide a pool of highly-redundant storage, you wouldn't use it?
Way to jump to conclusions about me and how I manage a network. I honestly didn't ask the question as a "control freak", I don't spy on the employees, and I don't play Internet cop. I try to get them the tools they need to do their jobs, help them when things don't work, and otherwise stay out of their way. I also didn't imply the pool would be for me to do with as I please; I can see several ways in which that storage would benefit our business were it not spread out in small chunks. The users have all that space, and they simply DO NOT use it. In our business, they don't have much call for large files like photos, movies, etc. It's mostly spreadsheets and OpenOffice Writer documents. But thanks for being an ass.
Switchfoot makes better music than Korn, too, but such opinion is no more revolutionary than the one in the article. Ethanol IS NOT the cure for our energy disease.
Obviously, I'm in dire need of this new laptop and its awesome computational capability.
the Spurious Engine! And maybe now malware purveyors will learn how to program for the Cell (since nobody else has), leaving your Core Duo free to run the browser that's being hijacked and a few UAC dialogs.
Your shameless plug just sold at least one of your CDs. And ignore the AC's comment--they obviously didn't give your music a listen. You are most certainly a musician. I, however, am a drummer, which just means I hang out with musicians. *BADDA BOOM*
What's wrong with our light switches?
...in N.Ireland and large parts of Scotland a 'tube' is an idiot...
Ah, so that's what Sen. Stevens meant when he said the Internet is a "series of tubes." So true.
Good post, but what's wrong with the way iCal handles close vs. quit? I just closed its window but iCal (v1.5.5) continued running, as you'd expect.