Slashdot Mirror
Large Scale Solid State Memory Storage?
spacechicken asks: "I am doing a theoretical study of an extremely secure large scale data storage concept. Due to the nature of the (theoretical) location of the (theoretical) warehouse some of our constraints include very few (if any) service visits, complete remote administration and no moving parts. Does anyone have any experience using or information on large scale (on the order of 10^12 - 10^15 bytes) deployment of solid state storage? And to preempt those who will say it - I have Googled."
I believe that this might be what you are looking for.
The technology is quite mature.
You want almost a THOUSAND TERRABYTES (10^15 bytes is 909.5 terrabytes)... of SOLID STATE storage?!? Are you insane?
That's about a quarter of a billion dollars in PC133 ram. I'm sure wholesale prices would bring that down, but you're still talking many millions of dollars.
http://www.google.com/search?q=Large+Scale+Solid+S tate+Memory+Storage&ie=ISO-8859-1&hl=en&btnG=Googl e+Search
Just go the the place where you got your 10^6 processor smp server. I'm sure they deal with 10^15 bytes of solid state storage all the time and will be happy to get you what you are looking for.
Hope that helps.
Since when has Uncle Sam worried about cost? :)
3dretreat
*pulls out a calculator* Hmm, 10**12 bytes is approximately 931 TB.
WHATEVER FOR?!
And that's on the low end, too. Since you're ranging a couple orders of magnitude, you might be pushing the limits of recognized abbreviations. All in solid state, too? That could be costly.
TANSTAAFI: There Ain't No Such Thing As A Free iPod.
How about a combination of commercial IDE/SCSI RAID technologies chaining many individual SDRAM based solid state units along with some sort of SAN?
For example
http://www.storagesearch.com/ssd-3.html can hold up to 8gb per unit, and at 15 per SCSI card, 6 cards per system, you'll end up with 720GB per 'server'.
Figure gigabit fiber (at the minimum) between machines and you'll end up with quite a bit of storage.
The interesting part about your question is the complete lack of background on why you would possibly need all this storage.
So I'll make a guess. 10^12--10^15 bytes is a large range. And I can only think of a few ways to generate that much data. The most probable is video cameras, but I can't think of any reason why you would need it secure in that fashion.
Secure without human intervention is interesting. I mean, if all you want is security, the easy way is distributed networks and encryption. And really, that would be more secure in the event of nuclear war or other similar events.
So I have two guesses:
A) Given the similarity of your numbers to the 10^11 neurons in the human brain (and each neuron has as many as 1000 connections to its neighbors) this is some sort of screwy immortality thought experiment.
B) Given the security requirements, this is some screwy thought experiment involving the preservation of the sum of human knowledge over a vast stretch of time without human presence. That could be interstellar travel, say, or large disasters wiping out the human race.
We're on you and your nefarious plans. You'll get no help from us! *makes sign of the devil*
Assuming slow, but commodity CompactFlash cards of 1Gigabyte each (currently $800 retail, your price may vary). You'd need 1000 to 10^6 of these puppies for approximately 800,000 to 800,000,000 $US (Retail). It would be fairly compact, fairly reliable, and fairly slow.
So, with a price of $800,000 today for the low end, in 3 years (more or less) the price for 1 Terabyte of CompactFlash will be $100,000. This drops further to the point where I can afford it, in about 2 more orders of magnitude (7 years?).
Bottom line, it's feasable, would be $1Million to $1Billion to implement 1 of (at retail, buy the fab, prices WILL drop dramatically). I'll be able to afford the same thing 10 years later.
--Mike--
There is a serious topic behind this question. However, it usually requires a lot of explanation.
Some of your guesses are correct - this does have space applications, but not in the way you think.
Suffice to say that the storage is not planned (theoretically) to be used as anything more than a commercial repository.
The reason for the vast amounts (as I keep saying) is because there will probably be only one visit to the site to install it all - meaning no upgrades.
Using solid state drives you could probobly do it on the order of about a half billion dollars. Possible indeed. But there are other things you haven't thought of. You didn't say if you need to have this thing on 24/7, but I'm going to assume you do.
1: Power. Solid state drives tend to forget stuff when shut off, so you'll need a UPS in the data center to handle it. No biggie, except when you realize that the batteries are going to need maintence. They do go bad after a while, I know of no batteries that don't. In theory you could do flash memory instead of volitile for about 3 times the cost (512meg ATA flash storage is $300 on pricewatch, add raid, san, etc, pricey but possible)
2: Cooling. Massive amounts of solid state chips are going to generate massive amounts of heat. This means water chillers (most likely) and fans. Both involve moters. Moters go bad. You can build redundent, but eventually both cooling systems will go out.
3: The hardware it's self. CPUs go bad, controllers blow up, ram chips go out, power supplies blow, etc. You can only leave a redundant system alone for so long until it's no longer redundant.
4: Acts of god. Floods, fire, lack of fuel for power, emi. These things happen. You can build two data centers in seperate locations and write one off when something bad happens, but then you're back to no redundancy.
5: Murphy's law. Don't forget, Murphy always wins.
Having worked on a LARGE scale redundent system (Think uncle sam), I can tell you these things do require maintence. Building a system that large without bugs that creep up in a few years time is going to be next to impossible.
That said, it sounds neat. Let me know when you guys need an engineer to build it, it'd be fun.
Remotely administering solid-state storage is no different than remotely adminstering spinning storage, just less chance of failure, so less visits. The data size you require can easily fit in a single rackmounted disk array. Vendors like E-Disk sell highly reliable solid state drives in IDE and SCSI, up to at least the high 10's of gigs in size per disk, 2.5 and 3.5 in form factor.
11*43+456^2
If you're planning to shoot this into space, I'd think again. You're going to have trouble finding solid state storage that both holds a petabyte AND is radiation-hardened.
The only other practical place I can think of right now where hard drives wouldn't survive the trip is some sort of undersea base. Sealand already has an offshore data haven, and they can visit theirs.
Tim
Omnia vestra castrorum habetur nobis.
So it is sort of a data haven in space kinda thing?
Heh, screw HavenCo, let's see the feds get a search warrant for an orbital datadump.
On second thought, one would have less social protections against getting the crap nuked out of it...
*LAUGH* Don't forget the latency to Geosynch, unless you're talking LEO, in which case you'd be in range of ASAT systems.
You're the closest one yet.
There is a way to protect it from radiation though - both natural and "man-made".
SARCASM
/SARCASM
Look at me Mom, I got my question posted on Slashdot by makin up some bogus and totally illogical question. Then I made tons of vauge references and quasi-logical posts to make people wonder. Ooh ooh ooh look at me!
Rule of Life Number 2: Remember, it can all go to hell at any minute. --Jimmy Buffet
That means:
A)You are sending it up in a big lead box.
B)You are burying it on the moon or another spacebone body.
C)If ISS has proper shielding, it's going into ISS.
A would probably be insanely expensive. Costs a lot to shoot lead into space. That's why NASA buys radiation-hardened equipment instead of buying some P4s and shooting them up in a big lead box.
B would be huge news. It would be extremely likely to fail if you send a robotic digger to the moon or an asteroid, so you'd have to do it with people, which would be a historic event.
C makes the most sense, but I don't know if ISS has enough shielding. Presumably there's enough to keep the people from overdosing on bad particles, but I'd imagine the computers would still have problems.
Tim
Omnia vestra castrorum habetur nobis.
sithkhan, it's called HTML and you do it like this:
<a href="http://www.yourstuffhere.com"> Fancy Word Description</a>
You have an "Allowed HTML" reference below you, I won't go to deeply into it, the B tag is for Bold, I for Italics, P for Paragraph, try them out, but do remember that all tags delimit an area of text and so they have to have an end tag. For instance to make a word bold you do <B> this </B> and it should look like this.
Any more questions feel free to ask.
Is the data changing where you will have hundreds of gigs being replaced each day? Or is this a write once read many application? If it is mainly a WORM then what about using proms? Or use a form of optical storage that will have a minimum of moving parts, perhaps using crystals instead of rotating mirrors or media.
There is a way to protect it from radiation though
;-)
:-)
Actualy, all solid state memory experiences errors due to cosmic ray particles, against which you CAN'T shield- eventually, some of these high-energy suckers will get through- and the problem gets worse the higher you go.
The chance for a given memory to fail due to this reason is called MTBF- Mean Time between Failures (actually, there's a broader definition, but I'm using the one related specifically to memory).
In addition, the more memory you have, the more errors you will have for the same MTBF- for example, if the MTBF is 1000 years for a single MB of your ultra-shielded memory. For 1000 MB, that means almost certain failure once a year! and you are talking about MUCH larger memory sizes!!
To conclude- in space, no one can hear you scream...
To Probe further:
Cosmic Rays
An article called "Can Hardware Be Trusted"
Despite everything I said above, there has been research on fault-tolerance in space, which might help you. You can look at the homepage of the Stanford REE project for more details
You might also be interested in these slides (PDF document) of a research project called Fault-Tolerant Computing for Radiation Environments.
Hope this helps
Astromage
HAHA this is a joke man. Even if you dont think it is, it is.
...
First of all, do you know the costs envolved with placing one pound into orbit. Do you think something this size will fit in one rocket/shuttle load? Well you wouldnt really have to deal with cooling, but power generation. I dont know what the hell you expect something like this to be powered from, but sorry we dont have fusion reactors yet.
Until going into space is a regular thing, and it isnt anywhere close to that now, this is completly pointless.
And a data-haven is pointless too. If some government wants they can just oh not launch your verhicle or go after those using your services
Yea no kidding. Who the hell are they going to get for customers? This thing would be way to expensive for anyone person or most any company to afford. The only groups that could really afford this would be governments. This is going to need a lot money to create. If some company is hiring this guy to research into this, they got duped. If you have to come to /. for this information, you do NOT belong in that position period.
You're going to have to pull your submarine up to that underwater fiber optic cable much more frequently than you planned to. Why not just run another fiber from the underwater tap to a secure listening facility on land?
(If it's not clear to most people reading this, the questioner wants to build a device that can listen in on a fiber optic tap and record a lot of data for a long time. Since fiber is vulnerable where it lies on the bottom of the ocean, and that position would necessitate the other requirements, that's the best guess.)
This is America, damnit. Speak Spanish!
on the CONRIS effect. Basically it's a 3D core memory, but with no cores. You get a chunk of specially prepared cryogenic crystal, some sort of carcinomer, a metal atom trapped in an ammonia molecule for example. .1 degree of absolute zero and then you use the same kind of magnetic selection system as core memory. Except it's in 3D, and each bit is the size of the crystal grain, which you can get quite small.
Freeze it to within
It'd be kind of big, slow and power hungry, but basically unlimited as long as you can get the crystal to grow.
Hey, it was the 70s.
I assume this is for a space mission from previous comments, so talk to the people with the most space experience. For this type of application the only system I would use would be a raid 1 utilizing IBM's shipkill technology. Basically you would have RAID 5 at the dimm level and raid 1 at the system level. Would it be cheap, hell no. Would it be the most likely to survive radiation hazards, yep. IBM has nearly a half century of studying the effects of radiation on computer components, so if you want to put a complex computer system into space talk to them. For some information to digest before talking to them see the chipkill whitepaper here.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Did anyone think in terms of *real* solutions, rather than just pricing out parts and assuming this guy could magically make them work together?
Anyway, at least one solution to this request appeared on Slashdot just a few days ago: The solid-state RocketDrive.
Perhaps not the ideal solution (I honestly can't say if OEM solid-state storage exists on a much bigger scale), but something that you can concretely say "this would work". Proof-of-concept, if nothing else.
Granted, for the size you want, at $5k/4G, This would cost USD $1.3 billion just for the storage itself (not counting the array of 32k+ PCI slots you'd need to hold all these and the hellacious network to RAID them), but this sounds like a gub'mint project anyway, so cost presumeably forms the *last* of your concerns. If cost *does* matter, you can get the unpopulated controller boards for $800 each, and certainly a *much* better bulk deal on RAM then what Cenatek offers (basically they charge $1k/1G? Perhaps 10 years ago!).
Checking Pricewatch, the average non-volume-buyer can get 1G of PC133 for around $100). That would lower the storage-only cost to only USD $315 million, before considering volume discounts.
- Power and heat: either you don't have enough juice, or you melt down - ram also produces heat;
- Errors: the more parts, the more errors, so a project this size would be the worlds' biggest random error/number/data generator;
- MTBF: you have parts, you have failures. For example, Maxtor drives may have a million hour MTBF, but the company recently reduced their warranty period from 3 years to 1.
If this was real, and your boss was even half-way competent, you'd be out a job immediately.The only storage that would begin to meet SOME of the requirements would be molecular/biological. Try again in a decade.
Have you thought about perhaps a custom many million mile long "spool" of fiber optic cable? What I am thinking of is something akin to a nickel wire delay line - except using fiber and laser pulses, photomultiplier tubes to pick up pulses on the other end, and high-power lasers to pump the thing - basically with the data going around in a very long circle, so that the data takes a half minute or so to make the trip - you could pack a lot of data in that length, much more with a multi-mode system (I think that is the right term). Of course, I am talking out my ass here, so it may not be that good of an idea...
Reason is the Path to God - Anon
And you need to be protected. Shoving will protect you.
Pushing will protect you.
Do you have stairs in your house.