Slashdot Mirror
Making Use of Terabytes of Unused Storage
kernspaltung writes "I manage a network of roughly a hundred Windows boxes, all of them with hard drives of at least 40GB — many have 80GB drives and larger. Other than what's used by the OS, a few applications, and a smattering of small documents, this space is idle. What would be a productive use for these terabytes of wasted space? Does any software exist that would enable pooling this extra space into one or more large virtual networked drives? Something that could offer the fault-tolerance and ease-of-use of ZFS across a network of PCs would be great for small-to-medium organizations."
It's the obvious choice.
install vista on them, that would fill up that space and give you something to manage your time a little better than wondering about what you could manage..
Does any software exist that would enable pooling this extra space into one or more large virtual networked drives?
Absolutely! Just hook them up directly to the internet before you update the machines, wait a few minutes, and voila! They'll be filled up with extra files in no time! Hey, you didn't say anything about wanting to be in control of what gets put on the machines...
If you have a very robust local network with plenty of spare capacity, and can accept a performance hit on the client computers, I am sure some kind of linked filesystem would be possible. In most practical situations, I think this idea would be a non-starter.
If they're in a computer room, then such a scheme might work. But, if they're on user's desks, you don't really have control. They're subject to filling up, being shut off, being knocked about, crashing, etc. I don't think in this case you would really get the reliability that the diversity and independence would suggest.
--Marc
...just in case your connection fails.
Is this a company, college, or just a random collection of boxes in your mom's basement? What function does your organization want to do that it can't because of a lack of a few terabytes? What does the actual owner of these boxes have to say about your little enterprise?
You could try to use something like "Localhost Azureus" for distributed data storage. The only problem will be that it will cost you in terms of processor and network hogging.
Is it cost effective to reclaim that (small) space? Probably not. My suggestion is to realize that no-one tries to save clock cycles any more and maybe this is the way disk storage is probably heading that way.
It's a very interesting question, but from my point of view, hard drive space is so ridiculously cheap nowadays that it is utterly pointless to look for a useful application that will fill it up.
Let's assume that the average computer has 80 GB of storage. Multiply that by 100 and you get 8 TB of space. That's what you can get into one or two computers nowadays without plunging out too much cash.
What's more interesting is how much processing power you have as well as how fast the internet connection is.
Full Tilt
Check out GlusterFS. (http://www.gluster.org)
You definitely can't run Windows in order to utilize this, but it should be a minimal effort to setup a quick netboot lab to test it with.
Cheers.
I had a drive fail on me last year and I wanted to take my frustration out on it so naturally I did what any good American would do. I shot the shit out of it. Surprisingly it seemed to make for a pretty good piece of bullet proof armor. It stopped multiple rounds of full metal jacket 9mm rounds and managed to get a couple rounds lodged inside the casing. (None appeared to penetrate fully)
If I can not smoke in heaven, then I shall not go. -- Mark Twain
Datacore offers software called Sanmelody to turner servers into a cheap storage network and there are other vendor solutions as well. http://infiniteadmin.com/
OpenAFS is a distributed file system. It seems to fit your bill. No personal experience, so don't know how well it actually works.
Doolittle :
Bomb no.20 : To explode of course.
There's project dedicated to this on Linux, http://nbd.sourceforge.net/.
If there's nothing similar for windows, you might be able to run it through cygwin.
Actually, this claims to run on Windows: http://www.vanheusden.com/Loose/nbdsrvr/
I tried to tout the merits something like this could have for non-critical regular user backups, but as previous posters mention, it was shot down.
:)
/doc" or whatever. NetDrive (and I'm sure there are others) help take away the learning curve and hassle of "here, use this internal ftp for backups, not a network drive" as it will map the actual FTP to a network drive and appear like normal.
I was suggesting to run DrFTPD as a backend with NetDrive as an access medium. It looks good on paper, but I've never had the chance to apply it so widescale
With DrFTPD it's easy to setup whatever kind of redundancy you would want, ie: "at least 3 nodes will mirror all files in
Just my 2c.
It could be that the only purpose of your life is to serve as a warning to others.
http://www.dcache.org/ You will need a system to act as a master, but otherwise your normal nodes should work great.
I am that much more enlightened and proportionally disillusioned
Please stop typing words like "utilization" when you mean "use". You sound like a PHB trying to sound smarter than he really is and you make it a pain for people to read what you write, especially non-Anglophones. Read George Orwell's essay on this topic.
If you don't already have a backup mechanism for the data that may be on these systems, one way to use all the available storage is for backup. Vembu StoreGrid a solution designed specifically for this problem. Get more info @ http://www.vembu.com./
You might want to ask yourself why, after more than a decade of research and countless papers and prototypes that address this problem, your PCs storage are still underutilized...
It's harder than it looks to get something reliable. Your PCs have extra capacity because it's cheap, but mining that capacity is not cheap. As other posters have pointed out, putting together (or just purchasing) a server with a few TB of storage is simpler and cheaper, less prone to getting wiped out by a virus, easier to manage and backup.
Am I part of the core demographic for Swedish Fish?
While I was in college, I worked in the IT department. In my experience, your end-users will have a proverbial shit-fit if their computer's HD starts spooling up when they aren't doing anything. While it would be nice to use the spare space for data storage, I'm not sure it would be worth the headache. The volume of user complaints would skyrocket, you'd have to train them to leave the things on all the time, and you'd have a distributed data pool to manage. Changing user behavior is like teaching a two-year-old to say "thank you" (It's possible, but not fun) and your electrical and manpower expenses would probably outstrip the savings.
is a project at the University of Virginia that tries to do exactly what you describe: take unused storage on a bunch of machines and turn it into a file system. http://vcgr.cs.virginia.edu/storage_at_desk/index.html
You know, make fun of Microsoft all you want, but they actually have something for this - DFS - Distributed File System. Just create a share with each of these and POOL IT with a DFS system. Then use and manage it to your hearts content with all the midget-donkey-goatse crap you want.
Hrmm... Funny, he didn't come across that way to me at all. You, however, come across as a pompous linguistic Nazi, much like Orwell. If you compose sentences for people who don't have command of the language, then you are really quite delusional.
As is my understanding, resources are utilised, while tools are used. He was correct in its usage.
Great, let's all dumb down to the lowest common denominator. English is a rich language and all the better for it. If you're too lazy to learn it, your choice. I'm a non-native speaker but prefer a vibrant, expressive language to some "for-dummies" international pidgin.
I'm sorry if I haven't offended anyone
Acutally, this sounds nothing like that thing you link to.
More like your post being a slashvertisement.
HI O WISE PRINCE. WHT TOOK U SO DAM LONG?
Well, you sound like a troll. I seriously doubt anybody misunderstood what he meant because he used the word "utilization". Or, should I say he utilized it? UTILIZE UTILIZE UTILIZE UTILIZE UTILIZE UTILIZE UTILIZE UTILIZE Does it hurt yet?
The solution is obvious. We need to think outside the box and raise the bar when it comes to language... someone needs to step up to the plate and bring something new to the table. I'm thinking of someone I have synergy with, not just the type that goes for the low-hanging fruit.
Ooh.... he's spinning nicely. Another couple of Orwells and we'll have enough electricity to power the world
"Slashdot - News and Chat Sites Deviant". (Click "homepage" link above for details).
This is why SAN manufacturers have come up with "thin provisioning". NetApp is quite good it, read more here.
Obviously computers will crash or be turned off. We have this wonderful concept in architecture design called "redundancy" which we can use to address problems like that:
Assume the probability of any computer being offline is d(c_n). For some computers you will have d(c) very low, such as user out of town often, other will have d(c) quite high, either the user leaves it on all the time or it has background processing to do.
Computing and updating d() is fairly easy given any modern management tool. Then create clusters of computers with a required availability so that you stripe data across the componenet computers taking into account d() of each computer. Availability of the cluster would be a function of your modified striping algorithm. When you save data, you just choose what availability you would settle for, and the right cluster is chosen.
Let me answer your next question in advance: if this is so obvious why is no one producing a product that's cheap and easy to implement? Because you'd have about 25 patent trolls lined up at the courthouse - too many teeth, not enough ass.
Support microSD: in a post 9/11 world, it is unwise to carry your data on media that you cannot comfortably swallow.
What would be a productive use for these terabytes of wasted space?
Well, I had this idea when I read about some Open Source software that allowed distributed storage (sorry, forgot what that was, but by now I am sure it has already been mentioned in this discussion). The idea was this - suppose we have such software for unlimited distributed storage, so that people can download it and volunteer some unused space on their HD for a storage pool. Then suppose we have some software for distributed computing like we have for the SETI program. Now we have ziggabytes of storage and googleplexflops of processing power, what can we do with that? How about, for one thing, storing the entire internet (using compression, of course) on that endless distributed storage, and then running a decentralized, independent internet via P2P software? The distributed database could be constantly updated from the original sources, and the distributed storage then becomes in effect a giant cache that contains the entire internet. Now we could employ the distributed computing software to datamine that cache and we could have searching independent of Google or Yahoo or M$FT. Beyond that we could develop some AI that uses all that computing power and all that data to do... what? - I'm not sure yet. Just thought I would throw this out there to perhaps maybe get stepped on, or who knows, inspire further thought.
I think what you're saying is we need to leverage a new paradigm in order to take things to the next level. Am I right?
Running DFS (to serve files) on Windows XP clients? What are you smoking?
From Microsoft TechNet:
The servers that will participate in DFS Replication must run Windows Server 2003 R2.
It is possible to use DFS Namespaces when domain controllers and namespace servers run a mix of Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 without SP1, and Windows 2000 Server, but some functionality is disabled or available inconsistently, depending on the operating systems on the servers.
From: http://technet2.microsoft.com/WindowsServer/en/library/1aa249c0-40f3-4974-b67f-e650b602415e1033.mspx?mfr=true
Even a blind squirrel will find a nut every once in a while.
Since the Romans invaded Britain, English speakers have used latinate phrasing to appear scholarly. Anglo Saxon words were short and pithy, like "home", "pig", "horse", "cat". But scholars learn latin, so it's "domicile", "porcine", "equine", "feline". In modern English, the choice gives you a palette of moods - like colors on a web page.
today's "businessspeak" (mindless repetition of words and phrases that have long since been driven into the ground by thoughtless, banal, stupid repetition)
Kids! That word, meaning "trite" or "unoriginal", is pronounced "ba-NAHL". If you say it the wrong way like I did in an interview, it sounds naughty and you sound stupid.
Detailed knowledge of any technological artifact will make you better at using it, maintaining it, knowing when to use it, whether it's an automobile or an AK-47. Yes, some people find guns interesting to a greater degree than others (I don't, personally, nor do I own one) but whatever floats one's boat. Let me ask: do you find someone that has an advanced knowledge of computers creepy? Probably not, if you're on Slashdot ... but there are many that do, until they need him.
When the time comes that I need a brain to pick, it's those "creepy" nerd types that I seek out. They're the ones most like to be able to help. Maybe you're anti-gun, and the fact that some people are not is offensive to you, I don't know. Regardless, you should look at people who know much more than you about a given subject as a potentially valuable resource, not an object of scorn.
The higher the technology, the sharper that two-edged sword.
ZFS ftw http://www.cuddletech.com/blog/pivot/entry.php?id=729
Way to jump to conclusions about me and how I manage a network. I honestly didn't ask the question as a "control freak", I don't spy on the employees, and I don't play Internet cop. I try to get them the tools they need to do their jobs, help them when things don't work, and otherwise stay out of their way. I also didn't imply the pool would be for me to do with as I please; I can see several ways in which that storage would benefit our business were it not spread out in small chunks. The users have all that space, and they simply DO NOT use it. In our business, they don't have much call for large files like photos, movies, etc. It's mostly spreadsheets and OpenOffice Writer documents. But thanks for being an ass.
Having tried this in college, I can tell you a couple things.
1. You will noticeably reduce the lifespan of the discs. (Which can anger cost conscious supervisors)
2. Doing ongoing hardware maintenance, because of this reduced lifespan, on closed, used by others, boxes is a *serious* pain.
Storage setups make hot swapping discs easy, trying to do this with full blown systems just gets tiresome. The solution I eventually came up with was the following.
Implement a two tiered hardware replacement cycle where you reduce the time a user is allowed to keep any hard drive in their box before replacement. Then using the still reasonably good drives, create a centralized storage solution in which the drives can live out the rest of their useful spans. Data security, user happiness, and redundancy are all good selling points of this system. You still have to deal with monkeying around in user boxes but if it's on a schedule and it nets you more drives, it's not so bad.
-Ian
You make it sound like it's a bad policy keeping all business data somewhere properly managed. It won't mitigate any damage done to your company or your career because you told them to be careful. People will store data in the most convenient location, thats not stupidity - just human nature.
You've toolized the language!
Fascism trolls keeping me up every night. When I starts a preachin', he HITS ME WITH HIS REICH!
Project Celeste is basically what the OP is talking about. It's a distributed filesystem with automatic replication, handles rogue nodes via voting and also exports the "filesystem" as CIFS. It's essentially a distributed object store, which can be used to implement a filesystem on top of it. I saw a demo of it last year and I was pretty surprised, it seems to work quite well for a research project.
Jean-Francois Im's blog
Been there also and I disagree on the "just works" part. We're using it successfully but we've run into issues with losing files from some of the computers. Let's say that the admin changes permissions on somebodies directory so that they can write to it also. DFS will think that the file with the changed permissions is the newer one and blow the other one away. I hesitate to call that "just working".
Don't forget that at those sizes, a .45 is nearly 30% larger in diameter, and has far more mass. A 9mm will normally have a 124 grain bullet with a velocity of 1150 ft/s, 364 foot-pounds of energy. A .45 can be shooting 230 grain rounds at 900ft/s for 414 ft-lbs of energy.
.45 was designed for FMJ ammunition from the outset. The larger and slower .45 round will use more of it's energy in a body, causing more damage. A 9mm HP will out stop a .45FMJ - but US soldiers are forbidden expanding ammunition. A .45HP will stop more often than a .45FMJ, but the difference is nowhere near as large as the difference between a 9mm HP & FMJ.
.223/5.56 round our military uses in most of it's rifles. 1300 ft-pds of energy in a 60-70 grain bullet traveling at over 3k ft/s. Sufficient velocity that the round will often fragment when it strikes a target.
Despite all this, I think that when it comes down to the army, it's mostly because of ammunition selection. Troops are issued non-expanding FMJ ammunition, which leads to 9mm over penetrating and under performing. The 1911, chambered in
As for the rifle comment, I have to agree. Consider the 'poodle-shooter', the
I don't read AC A human right
The poster asked how to use the wasted space on all the Desktops in his business by pooling them as one big hard drive. So yes, we are in fact looking for ways to make 1 big hard drive, not just share files, and yes, we're pretty sure he's not running a Windows Server Family Operating System (tm).
So you can count DFS as a big NOGO.
"Not to mention all the idiots who use words like boxen."
Anonymous Coward on Monday August 04, @06:49PM
Please do not use the space for anything else. Do not try to actively use the space.
The reason is the obscenely large amount of power required to use the space given a few gigabytes requires the whole machine to be running, and uses it's CPU which can't be less than 21Watts itself.
It's actually cheaper to get a 1TB drive and use it elsewhere than use the power on so many desktops (or worse, servers). Even with the desktops in use by active users.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
limitations?
And, if you're claiming some kind of market race, you might want to check for relevant dates concerning ZFS
Of course, if you're just trolling, ignore me.
Computer memory is just fancy paper, CPUs just fancy pens with fancy erasers; the 'net is just a fancy backyard fence.
You could put a Hadoop Distributed File System (HDFS) on them. HDFS allows you to use the storage as a single file system that is stable and reliable. We have multiple 2000 node clusters with petabytes of user data on them. Because the blocks are each replicated to 3 hosts, if a node goes down, your data on that node is not lost.
I do some work for Allmydata, which an online storage provider. Their next-gen storage technology is open source and nearly perfect for this application. It's a bit green at this point, but coming along nicely. http://www.allmydata.org/
You have violated Robot's Rules of Order and will be asked to leave the future immediately.