Slashdot Mirror
Google Prefers DRAM to Hard Disks
KP writes: "I came across this interview with Google's CEO. A very interesting
read." It's interesting in part becase that CEO (Eric Schmidt) claims that for Google's purposes, "it costs less money and it is more efficient to use DRAM as storage as opposed to hard disks." "I still cannot figure out how he says storing data on DRAM is
cheaper than storing it on hard-disks. Maybe, if you buy in bulk?"
In the hallowed halls of Google... Row upon row of uber-boxen with a Bagillion megabytes of ram...
Then someone trips over the power chord...
-- Dan =)
How often do you see DRAM fail compared to Hard Disks? A bit more reliability IMHO.
If google has something like 10,000 linux PC's, I would definately think that using RAM and a ramdisk for the rootpartition would be cheaper than putting a hard drive in every PC. I would imagine that the hard drives would be the first to go if something failed.
Obviously, if they used DRAM for their HUGE central databases, it would not be a cheaper solution.
But, I'm talking out of my ass, because I don't know how their datacenter works.. anyone anyone?
-metric
They make their money on hits served so speed is far more cost effective than cost of storage medium. If they can speed up serviing hits, they're ahead of the game.
I still cannot figure out how he says storing data on DRAM is cheaper than storing it on hard-disks. Maybe, if you buy in bulk?
When you pay for DRAM, you get read latency measured in nanoseconds rather than milliseconds, which lets you get more queries done faster with less processing hardware. The key metric here is seeks per second. From the article:
With a rotating disk, if you wanted to access a million different pieces of data, you would have to either wait for a million seeks or set up a 1,000-way mirror and wait for 1,000 seeks. Because DRAM seeks several orders of magnitude more quickly, you don't need as many mirrors of the data to get the same number of seeks per second.
Will I retire or break 10K?
Err. No.
I maintain a tiny search engine (some 5000 sites), with the data cached locally, just like Google. It takes ~250Gb of disk space for that miniscule cache. The one at Google must be of the order of a few hundred Terabytes, not Gigabytes.
On that basis, I echo the original query about how it can be economical to use RAM...
Simon
Physicists get Hadrons!
AFAIK Linux and Open BSD cannot do this either. It seems amazing to me that people have missed this idea.
Google reads all the newspapers on the Web every hour and constructs a newspaper for the world by computer--no humans are involved.
Now if only Google could go out and do its own fact-checking, it wouldn't need to rely on other newspapers at all. Mark my words, by 2010 google will be the only place you go when you need information. Forget askjeeves, try listentogoogle. No humans will be involved. Scary.
By the way, this guy can't speak for beans.
The speech I give everyday is: "This is what we do. Is what you are doing consistent with that, and does it change the world?"
I often see comments from this from people who have little experience in business.
What you pay for the initial product is not what it "costs" in the long-term. Businesses have a term for this called TCO or Total Cost of Ownership. It includes all the other time and materials needed to keep the item in use.
I would imagine in this case that the simple reason is that why DRAM is more expensive to purchase it is a *lot* less expensive to run, the primary cost being power.
Also consider that if speed is of essence, as it with Google, it's not 50GB or RAM vs a 50GB cheap-n-cheerful IDE drive. A 50GB Ultra160 drive costs considerably more than an IDE and still won't come near the DRAM for speed.
[)amien
Huh? I would have thought it would have been between 10x to 100x that much. Especially if they cache most pages. (Maybe they just use dram for the indexes, and hd's for the cache?)
I still don't understand that claim. $300 will get me a 160G drive, and I can load four of them in a cheap PC case or 1U rackmount case, 640G per unit. That's under $2K for
RAM prices vary wide, but say on the low side I can get 256M for $20. I'd need 2560 sticks of 256M to equal 640G, or $51,200 for the equivalent storage. And that doesn't take into account that most reasonably priced PC motherboards only handle 2G or 4G of memory these days. You'd need 160 motherboards in the best case, adding $80,000 to the cost, assuming you could get 4G per unit, and $500 per motherboard/chassis. Let's, see $51K+80K = $131K, versus $2K.
RAM, as I figure it, is at least 65 times more expensive (that's not 65% more, it's 6500% more).
Either their archive is a lot smaller than I assumed, or they're talking performance/price tradeoffs, where speed has a high premium.
-me
Love many, trust a few, do harm to none.
That it can handle many clients with little latency... You'd have to duplicate the data across a huge number of disks to provide similar response time to clients. Sure, if you were the only client, you couldn't tell the difference but with thousands upon thousands of clients all seeking data that would be stored in different locations on a disk things would quickly grind to a halt. Because so much unrelated data is being requested, seek time is the key. Sure, memory is more expensive per meg but its ability to serve so many more clients makes it less expensive overall.
A few 100gb to cache the entire internet?
I had an opportunity to play with one on a 20 CPU Starfire domain and it was pretty impressive. The unit I was using had 8 wide SCSI ports on it, which were all connected. Interestingly, when the system was pegged, it was off the scale in system time. There's probably a locking problem in the Solaris kernel that's the real bottleneck.
This just shows how limited the lifespan is of 32-bit 4GB architecture, especially for servers.
At my dad's work, they use a type of chip, but it's not dram. They use E^2prom. True, you do take a performance hit, but they have 10 "gig ethernet ports" on the thing. The last price quote I got was $12000 for a terabyte of this stuff. Don't forget to compare price/performance ratios to the best chipsets of IDE (or if you're a scsi bigot, SCSI). Pulling random data is very easy for chips, but HD's of ANY speed and quality are still slower.
Josh Crawley
If they made a 2GB RAM Drive in each of their 10,000 machines then that would be 20 TB of storage. This seems sufficient to me for most storage needs.
You would still need to be able to direct searches to the machines that have the part of the data you need. This would take a high speed network and some clever programming. But it is doable.
I always was amazed at the speed of googles search engine, now I have a little more clue as to why it is so fast.
Sounds to me like they might be able to sell their database software as a money making product at some point. Oracle, watch out!
-- Never make a general statement.
See The Five-Minute Rule, ten years later (Word Doc) or it's HTML-ified Google Cache
Reasonably priced DRAM goes for about $250/gig; a reasonably priced SCSI RAID setup goes for about $10/gig.
In order to say that the DRAM option is cheaper than the hard drive option, the performance of the DRAM option would have to exceed the performance of the DRAM option by a factor of greater than 25. If you do the math, it's possible.
Years ago, I worked in a VAX shop that used RAM drives for some installed/shared images that required high concurrency. The performance was impressive - and was factored into the overall cost analysis of the purchase.
AFAIK, Google does not cache images, only HTML text. The web size is estimated around 5-10 Terabytes, and text size as percentage of the web is between 12-30% depending on whose paper you read.
Hence the size of the cache is somewhere between 500GB and 3TB, plus the index would be another 40% of that.
My best guess is that the google archive is somewhere around a 2-3 terabytes, and that the total amount of DRAM available at google at the present time is somewhere between 5-10 terabytes.
It's important to note, though, that he states DRAM is more efficient (cost-wise? speed-wise? whatever) when it comes to storing seekable data. I wonder if that means they're using DRAM for their search indices and plain old disk for their cached content. DRAM is ideal for completely random access to multiple pieces of data, whereas disk does okay for serial access to data, the location of which is well known.
Maybe he's talking in terms of TCO (total cost of ownership). Over its lifetime, RAM costs less than its hard drive counterpart?
Another point... as long as you don't store you METADATA 100% in RAM, you can store at least your data (cached web pages) in RAM. What happens if it gets dumped? Simple. Just respider the pages you lost and go on. Small amounts of data loss can be covered.
Okay. It may sound like I'm talking out of my ass because I am. It is really hard to cover for a statement like that. But lets talk again on the performance angle that has been covered (but with a little more emphasis on RAID disks).
You *may* be able to get better cost/performance with LOCAL memory (not ram-based drives) than you could with a RAID array. And a raid array could never equal the performance you get with local memory. Of course, local memory could never reach the storage you achieve with a raid array. So these two paths seem to diverge (bulk storage vs speed) when comparing local DRAM to RAID'd disks.
His statement MAY make sense, but it would have to be put into a larger context. (RAM is better than disk in X circumstances.)
I think he (Eric Schmidt) spoke of storing the indices.
Traditionally, they are only stored partially in RAM due to their size.
Certainly, the unprocessed pages are still stored on HDs as one doesn't gain
anything from storing them in RAM.
"Between strong and weak, between rich and poor [...], it is freedom which oppresses and the law which sets free"
Look at PDAs / handheld PCs. They use flash memory, albeit out of necessity (price, power consumption, size, etc)... but we're already beginning to see laptops incorporate solid state storage technologies. It's only a matter of time.
... ;)
Now, if we could just get around that pesky limited-write lifetime
...has an article on this very subject. The listed article "How to hack from a RAM disk" is what you're looking for.
The simple truth is that interstellar distances will not fit into the human imagination
- Douglas Adams
DRAM is probably much cheaper than hard drives in the sense of their electricity bill. Think of how many nodes their clusters have and then imagine each of them each having at least two hard drive motors spinning 24/7.
this makes more sence then:
/.ed site)
PC World: What are Google's biggest challenges?
Schmidt: Managing the growth. Our servers are overloaded. There is a DRAM shortage. We're building more computers. We are adding more-sophisticated products to the advertising side of Google. Our problems at the moment are growth problems.
If you have computers where 4 GB is not very much memory, but use the amount we use on out HD for memory i would have a dram shortage too.
And i bet they store only the most frequest used part of the index in memory.
Did you notice when you access the google cache this very slow compared to a search? Even if that cache was accessed frequently (because it references a
More often than not with a database your bottleneck is I/O. When you run a database you cannot have enough disks, and you cannot have enough FAST disks. In order to accomplish the kind of I/O bandwidth that a place like google is going to need you're going to need the best EMC arrays (or perhaps an IBM Shark) money can buy. And guess what? They run you megabucks. You can't just take a bunch of SCSI disks and expect them to perform as well as Fibre channel arrays. You gotta have controllers with multiple caches. Everyone who's never dealt with databases think that SCSI is the beginning and the end of hard drives, and its so far from being the truth its not funny.
I've really no idea how complex the queries are or whether or not they use a relational database but that being said its still has to hit the disk to retrieve the data and that's where every decently designed database's bottleneck is. Besides google caches all its pages. Egads! Do you have any idea how much RAM they must need for just that alone? Yes RAM is faster. Oracle even teaches you to try to keep your frequently used tables in cache anyhow, because its fastest, of course they qualify that with the word small realizing that most people don't have the gobs of memory needed to cache large tables.
Although it's not mentioned in the Slashdot writeup, I think that probably the most important part of this interview was the discussion of Google's business model and future. It's good to see that they're committed to not getting in over their heads with extraneous services. They've found a business model that works and they're sticking to it, rather than getting greedy and adding dumb new services that have nothing to do with searching, or "search," as he put it.
A lot of technology companies would do very well to follow Google's example, it seems to me. They're proving that Internet services are a perfectly sound venture if the company has a sensible business model and always keeps focused on providing quality technology and services in the area that they know best.
Lots of other posters have mentioned pieces of the puzzle, so I risk being redundant here. But, it seems the whole equation goes something like this:
1. If each box only handles a part of the web, it is possible that most of the space on it's drive (or drives) are wasted anyway.
2. If disk latency means that cpus spend idle time, eliminating that latency means more throughput per box, hence fewer boxes. More money spent on DRAM, less money spent on CPU, power supplies, etc.
3. Even with same number of boxes, lower power draw, smaller and/or fewer UPS(s) required. With fewer boxes, even more reduction.
4. Which leads, of course, to lower A/C bills during the warm weather.
5. Fewer boxes, fewer pieces, whatever, means fewer things breaking. The impact of a single outage may be greater, but, from the cost standpoint, you need fewer man-hours to manage the outages, fewer spare-parts, etc.
6. Lower medical expenses from sysadmins going insane due to the noise from all those drives and the associated larger power supplies and extra cooling fans.
OK, that last item is a stretch, but how many sysadmins are more than a step from insanity anyway?
Another service that takes advantage of recency is something we just added called Overview of Today's Headlines. Google reads all the newspapers on the Web every hour and constructs a newspaper for the world by computer--no humans are involved.
This is a pretty cool idea. I only hope they make a RSS feed out of it so that I can use it in my companies new Portal environment. That would be really great! I love Google!
Check it out here.
KangarooBox - We make IT simple!
DRAM requires little electricity and produces almost no heat.
Hard disks consume large amounts of electricity, and produce large amounts of heat, since they consist of pieces of metal spinning at 7200rpm.
Using DRAM upfront costs quite a bit more, but uses less electricity and requires fewer chillers, condensors, etc to keep cool.
Conformity is the jailer of freedom and enemy of growth. -JFK
Individually, the mean time betweeen failure for a brick isn't that bad, but when you get enough of them, it's a constant drain on the pocket and on person-hours.
-Eldurbarn
Google does so cache images [google.com]. :)
Cute, but not quite correct. They cache post-stamp sized copies. If you want the full image you have to go to the original web site.
Granted, this does increase somwhat my original estimate of the amount of DRAM required.
I really think people under-estimate the size of the web, and this only becomes apparent when you try to cache large sites. Sure the majority of websites are pretty small, but more often than not now, government and business websites are used for real data-access solutions.
As I mentioned above, I look after a small but targetted search engine (http://www.financewise.com/) which looks at only financially-orientated sites. Take for example the European union site http://europa.eu.int. This is a fairly innocuous site, but if I do:
That's a 7.7Gb website, and that's just the text (in fact I only search for
I just think that your estimate for the cache size is a long way short of the real figure...
Simon
Physicists get Hadrons!
...but they'll get a million times better as soon as they'll allow boolean searches. Man sometimes it's frustrating!!
Its not a fair comparrison to put 1GB worth of DRAM on one side of the scale, and 1GB worth of physical storage on the other. The hard disk will obviously come out to be the cheaper of the two. However, to a company like Google who undoubtedly uses RAID technology for storage, you're effectively not getting the same "bang for your buck" as you would with a JBOD array. In order to have 1TB worth of DRAM on a scale next to 1TB of physical storage, you're going to have to amass like 2TB of storage on the plate in order to have just the 1TB worth of usable free space.
Mind you, thats not to say that RAID is a bad technology..heh, hardly. Its just that you cant make a 1 to 1 comparrison from DRAM to physical without taking into account the storage methods employed by each.
Cheers
Bowie J. Poag
The sound a Mac makes when you turn it on.
Your single box for 2000$ doesn't take into consideration the fact Google needs to make their tons of information available to everyone at once. With a search engine like Google it is going to be rare information is just going to sit around and never be used. This means that by conventional database architecture logic you keep it cached in RAM. Hard drives are useful when you're cutting power to a computer, how often does Google reboot?
I'm a loner Dottie, a Rebel.
See that "mature content filter"?
How about a "mature content ONLY search"?
********* sig: If you don't like the law, get filthy stinking rich, and buy a better one.
I really think people under-estimate the size of the web, and this only becomes apparent when you try to cache large sites. Sure the majority of websites are pretty small, but more often than not now, government and business websites are used for real data-access solutions.
Indeed, this has been a hot area of debate for the last 7 years or so, when the first paper with a substantially larger web than that indexed by search engines came out.
Usually search engines estimate the web size to be about 15-30% of that claimed by statistical measurements.
One would have expected /. nerds could to better at price comparisons than what we have seen so far.
Quick, what is a better price a 1994 Ford Fiesta at $10,000 or a brand new Ferrari at $12,000?
Clearly the Ferrari is a better deal. To do a proper price comparison you have to look beyond the sticker price alone.
What is the performance you get? resale value? maintenance cost? operation costs?
If all you wanted to buy is megabytes of storage you would be better of buying backup tapes. They are hard to beat price wise.
But in all likelihood you need to store that data for some purpose, so depending on frequency of access, latency, total cost of operation (tapes are operator/robot mounted), alternative solutions with higher sticker price, might well end up being cheaper.
What Eric Schmidt claims is that if you have a ton of data and you are accessing it all the time DRAM is more cost effective than (a) a large mirrored RAID array server or (b) a zillion tapes being mounted by operators.
Hrm...
So this is why SDRAM prices have been going up and not down lately...
Bastards...
~z
sig?
Recently I was fortunate enough to be able to play with (test) some RAMdisk products from a company called Platypus Technologies (do a Google search for platypus linux) on Solaris workstations and servers. And of course I just had to try them out on the Slackware boxes too.
These Platypus drives are PCI cards and have dual power source ability; they plug into the wall as a secondary supply and get power off the PCI bus as primary. Very cool to be able to shut down the machine to do whatever and still have your RAMdrive ready to go upon boot. Feature wise, they use expensive RAM and the manufacturer strongly suggests you not just grab any ole ECC to stick in the card but order from them (probably has to do with the grade of RAM they use in their cards.)
Performance was absolutely unreal: more than twice the speed of SCSI, in fact, practically as fast as the PCI bus in the machine will allow. I used the cards briefly while doing a a small database conversion project and was totally bummed when I had to send the RAMdrives home. *sniff*
If you have to do anything requiring lots of I/O (like database,) you _really_ do want one of these things or something like it.
Cost-wise they are a little spendy up front (even when compared to a SCSI setup with controller and drives) but if you are at all measuring time, then everything else looses the comparison; if you are measuring lost data on dead drives, the time required to make many redundant backups to avoid lost data on dead drives, the time required to shut down and swap out dead drives, etc. -- RAM wins! Just be sure to factor in the cost of quality UPS units because they truely are part of the cost (read necessary.)
Hook up a Qikdrive2 with one GB RAM, plug it into your UPS, make sure it gets backed up to the hard drive regularly (plenty of tools to do that) and I promise you that you will not want to be without one. If you have the resources, get one of the big ones (6 or 8 GB RAM, I forget.) Look on CDW, search Platypus for prices. The Platypus site has links to purchasing sites.
As always, be sure drivers/modules are available which will work for you. Ack, I'm rambling.
Everything in the Universe sucks: It's the law!
Just a thought:
when is it worthwhile to trade off cpu for storage? In your case, I suspect that the website has a degree of redundancy in its 7 gigs of data; there is likely much duplication. Both at the page level (duplicated ccs info), and at the snippet level (duplicated copyright disclaimers).
It is quite straight forward to discover this sharing (IIRC exactly how lzw compression works, but w/ a smaller window) and significantly cut down your storage costs. Of course, now you have a CPU hit, where storing new data becomes expensive, and just reading the data requires some pointer chasing.
The interesting issue is that the CPU hit isn't guaranteed to be a Bad Thing: your higher cache hit rate (indeed, your data may fit in ram entirely now) will possibly (likely?) result in significant speedups.
...
"I love my job, but I hate talking to people like you" (Freddie Mercury)
According to "The Anatomy of Large-Scale Hypertextual Web Search Engine" by Segey Brind and Lawrence Page, the inverted index ("inverted barrels") was about 47.2Gb large (Total data without repository 55.2Gb, Repository 53.5Gb). It had about 24 Million web pages indexed. Assuming a linear increase this amounts to about 5Tb.
But, to quote from the paper:
Which is surely slightly exaggerated, but shows that they considered that there is room for improvement. (E.g using varying length index instead of fixed width)
>I dont think Linux can do it
At least they think it can do it, since they are using Linux boxes, at least accoring to
The Technology Behind Google, by Jim Reese CEO.
More than 10,000 Linux boxes, that is.
"Between strong and weak, between rich and poor [...], it is freedom which oppresses and the law which sets free"
That's a great calculation, but just figures the space needed for caching the raw data.
What about the indexes required to actually access that data in a timley manner? Once you factor in the extra stuff needed to actually make it a viable search engine, you could easily imagine a PB or more of storage was required.
As for the other poster going on about comrpessing the data - I doubt they'd want to compress the data when all they are concerned about is raw speed of processing requests!
.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Fixed head hard drives have no seek time, since tracks have a many to many relationship to heads. That's also why you can't get them at compusa. ( expensive )
Also, Google's searchable data is considerably smaller than the total size of the pages searched, even excluding the images. Read their white papers. And I doubt that they store the cached pages and images in DRAM. Those don't get hit that often.
Google doesn't cache images google doesn't index or cache dynamic (scripted) content google caches PDFs as Plaintext.
However they are definitely on the scale of terrabytes. "Searched the web for a.
Results 1 - 10 of about 1,470,000,000. Search took 0.31 seconds." Assuming an average of ~25k cached per link 1.4 billion links would leave a cache of about 37,632,000,000,000 bytes, However The Cache doesn't necisarily need to be stored on RAMDISKs. He clearly states that it's 200,000 times more efficient for _seekable_ data. This means not the 'cached' data but rather the stuff that the search alagorythm looks at to show you appropriate hits. So the heart of the 'search' engine is using RAM exclusively, but 'cached' data would almost certainly still be stored on HDs, unless of course someone has built google a bunch of 120GB DRAM disks that use conventional HD interfaces (sorta like the Flash memory Drives, only on steroids when it comes to speed).
It could even be misleading Google could have meant flash memory HDs were cheaper but mistakenly refered to them as DRAM.
https://www.gnu.org/philosophy/free-sw.html
The data isn't just sitting there static, though: It's being searched. To switch to hard drives and maintain their current performance level, they would have to increase the parallelism of the search, by having many more copies of the index. One copy of the index on disk is not really equivalent to one copy of the index in DRAM, because the DRAM index can be searched many times in the period it takes to search the HD index once.
The quantity they're trying to minimize is not dollars per megabyte, but rather dollars per (megabytes searchable per second).
Sorry, I wasn't being clear - I forgot to point out that these files are already compressed (using gzip), but only on an individual file basis. The real site is significantly larger than this 7.7Gb, and I should have mentioned that.
....
Whereas I agree that we're getting close (or maybe have passed) the point where it would make sense to do something better, since I don't have much of a budget, and disk is cheap
ATB,
Simon.
Physicists get Hadrons!
Consider other things though. While the initial cost is high, electrical power for 1GB of RAM is lower than that of 1GB of hard drive, and since RAM is solid state, maintenence costs would be tiny. Imagine the costs of keeping a few hundred hard drives, each rattling away 24/7 from dying out!
It's been a long time.
a least completely.
I had the same question myself over the years. Especially recently, as memory prices dropped through the floor.
Linux has the option of loading itself into a ramdrive, and that's great. But why not Windows 98 or ME? Is it because it was technically hard, or was it instead tht the concept was too alien to the developers? (One ALWAYS uses disk! Don't bother me!)
RAM is faster -- always. I realize you that you can't live off of RAM alone, but at the very least the swap file shouldn't be on disk. I've spent too much time in the past ten years listening to hard drives slice meat as I waited for Windows to move pages off of and into RAM.
Well, if XP provides the option, fine. But I won't use XP. Don't like subscription OSes. Maybe the 2K version permits it. I'll try.
Wonder how much of computing is just bad habits?
In many clusters today like KLAT2 they only use hard drives for the root nodes, and the other 98% of nodes use 2GIG of ram.
This saves you at least $150 per slave node by not buying a hard drive, thousands for having to deal with less hard drive failures, and acess times are orders of magnitutes better.
Lets do the math. 512MB of PC133 on pricewatch today was $67. For 2GIG of ram that comes out to $268 per node. For a terabyte(2modules*$67*1000GB)=$134,000.
That blows my mind. A small research lab can now own a terabyte of PC133 for under $150,000. Man, do I feel old.
bash-2.04$
bash-2.04$yes "Don't you hate dialup connections?"| write USERNAME
Google doesn't create content. They are a search engine. Nor are they in the business of archiving the net for posterity. If they lose data, it's out there to be recollected or if not, then there's no point in them saving it anyway.
The cost differential between RAM and disk has been eroding for some time, particularly if you compare RAM with SCSI disk. While the price of IDE had dropped, SCSI is still premium priced for the business market, even though there is no reason why a SCSI controller should cost a cent more than IDE.
A 80Gb SCSI-160 drive costs $800, RAM costs $150 for a 512Mb DIMM. So Disk costs $10 per Gb compared to $300.
The problem with the raw comparison is that you still need a lot of RAM to service a large disk, caching etc. There is also a limit on the amount of disk data one CPU can effectively manage. From experience I can asure folk that that limit is certainly less than 80Gb if the lokups are frequent!
So when you add the cost of a CPU and box into the equation the RAM solution is gong to look much better. I doubt that a single CPU could effectively manage more than 4Gb of disk data, but 4Gb of RAM data is quite viable. And you probably need at least 1Gb of RAM to support the disk data in any case so the all RAM solution looks good.
For most database applications RAM wins hands down. On top of the cost of the disk you have to count on
The main problem for the RAM route is getting persistence on transactions. So you need some secondary storage in case of power failure or disaster. This could be tape, but ironically disk is cheaper to run these days than almost all tape systems. A 40Gb cartridge for a tape drive can easily cost $150, which is more than an IDE disk drive that outperforms on practically every level (probably even longevity).
The key is that you use your secondary storage to write out the transaction log, you don't attempt to maintain the data structure on disk like SQL databases do. For high reliability you use a complete duplicate of the system to provide your first level backup with disaster recovery at a remote site.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
Google will also likely break their technology into three components:
spidering and indexing
searching
caching
::Colz Grigor
Each of the financial analysts for the business groups responsible for each asepct of Google's technology may calculate the value of DRAM vs. HD differently. For searching, latency is extremely critical, but it's not so critical for caching, and there may be some physical problems with solely using DRAM for indexing.
That being said, I would expect Google to use HDs for spidering and indexing, DRAM for searching, and HDs for caching. Mr. Schmidt was probably only discussing technology on the most visable component of Google's technologies: searching.
Assuming 80GB drives each drawing 40 watts of power, and electricity rates at $0.20/kWh, you're looking at an annual power cost of less than $1 per gigabyte of spinning disk storage. That hardly accounts for the difference.
"Biped! Good cranial development. Evidently considerable human ancestry."
You guys crack me up some times.
I'll lay it out. Obviously Google is not storing the master copy of the full multi-terrabyte database in ram, but they are certainly storing as big a chunk in ram as they can, and the cost model ought to be easy for anyone to understand if you sit down and think about it.
Consider the cost difference between the following EQUAL amounts of hard disk storage:
* A 160GB IDE drive
* A 160GB SCSI drive
* Four 40GB drives in an external RAID system
* The cost of a small medium-performance RAID
system.
* The cost of a larger high-performance RAID
system scaleability to a terrabyte.
* The cost of an *EXTREMELY* high performance RAID
system scaleability to multiple terrabytes.
Now consider the cost of building, say, a 40 terrabyte data store (lets not worry about backups for this experiment). If you build it out of a bunch of huge SCSI drives connected to a bunch of PC's it can be fairly cheap. But if you build out of, say, high performance EMC arrays it could cost millions of dollars more to get the same theoretical performance.
So when you consider the cost of storage, you always have to consider the cost of the PERFORMANCE you want to get out of that storage. All the Google CEO is saying is that, Doh! It's a hellofalot cheaper to improve the performance aspects of the system by buying DRAM in a distributed-PC environment in order to be able to avoid having to purchase extremely-high performance (and extremely expensive) disk subsystems. The cost of purchasing the DRAM to make up for the lower-performing disk subsystem is actually LOWER then the cost of purchasing an equivalent higher-performance disk subsystem.
The same is true in the ISP world. When RAM was expensive we had to rely on big whopping HD systems to scale machines up. But when RAM became cheap it turned out that you could simply throw in a very high density drive with 1/4 the performance that four smaller drives would give you, and the operating system's RAM cache would take care of the problem. Suddenly we no longer needed to purchase big whopping disk arrays.
Think about it.
-Matt
I liked it, even though somebody apparently thought it was redundant. It doesn't directly apply to Google, but the principles of trading off speed and cost are still relevant even though the problem's a bit different. One thing I'd find interesting is knowing how much of Google's index data is replicated - one master copy (which might be backed up on disk) kept on N search engine boxes - vs. how much do queries get spread across multiple boxes? Does it make sense to cache the spidering on disk (probably, because rerunning spidering takes a long time, and because the article caches probably don't get hit as often, and don't need the same response speed as the indexing.)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
This sounds plausible, since you can use fewer machines. But the problem I see is, where do you find a machine that can address 80+ gigabytes of memory? Otherwise, you have to but just as many commodity boxes to hold the ram, which ruins the cost benefit.
Does anyone have any insight on what machines you would use to support this scheme? Does a SAN-type device for RAM exist? Some network-attached box that holds tens or hundreds of gigabytes of RAM?
I mean, obviously you have some kind of grudge against it, to abuse it that way.
Take the RAM out of your computer and throw it at your workmate/housemate/mum. He or she will say 'Ow!', and it's not because he or she was hit by electrons!
This would, indeed, be the use of RAM as a mechanical object but this type of use is not characteristic. You appear to be claiming with this example that any solid object (and possibly any matter) is a "mechanical component," which is wrong and would be harmful to meaningful communication if accepted.
Any solid object's atoms move in relation to each other. This does not mean it can be said to have "moving parts" (this useful phrase would be rendered meaningless, otherwise), or make it a "mechanical device" (ditto).
Every electrical device is utterly reliant on its physical structure to function properly, and will cease to function properly if its structure is altered beyond certain limits. A broken connection is not a mechanical failure.
Sure, the clip that holds it in place is mechanical, and can suffer mechanical failure, but that is not part of the RAM. To note Telstra's odd problem as evidence of RAM being subject to mechanical failure is like talking about a wind-up alarm clock being struck by lightning as evidence of such clocks being subject to electrical failure (this would, of course, actually be an electrical event causing a mechanical failure).
It's a risk, but the problem is that other sites will intermingle .html and .php/.asp depending on whether there is any customisation or even just for headers and footers.
:-)
In this case, almost all the documents are in fact dumps of pdf files also on the original site. I chose it because I knew it was big
Besides, for a search engine, getting the catalogue can be a useful thing - in the sort of targetted search engine that I'm maintaining, anyway. A lot of the searches are for particular mathematical models (mainly excel spreadsheets at exorbitant cost). These tend to be catalogued just like any other online shop...
ATB,
Simon
Physicists get Hadrons!
the catalog part is still in beta, but it's really amazing.
I hadn't really looked at that part of Google until your post. Based on a couple of searches I did, didn't seem all that amazing to me. More like white knuckle frightening!!
This must be that level of technology that is too easily taken for magic. There are just too many perfectly rational reasons why this "shouldn't" work at all!
The line must be drawn here. This far. No further.
It does. But it doesn't help much and measn you have to reload the whole RAMdrive (generally over a LAN) when the box dies. Admittedly, it is a more efficient use of RAM than just handing it to Windows, since Windows (particularly the 9X stream) is a hopelessly inefficient user of RAM.
You must really have spent a lot of time and looked hard before saying that... )-:
``And death and hell were cast into the lake of RAM. Diskless Windows is the second death.'' -- Revelation 20:14, Geek Modified Version
Got time? Spend some of it coding or testing
Solid state everyting would be great (wasn't there an article on solid state cooling fans a while back?), but it may take a while for RAM drives to bridge that big a gap, especially given the volatility problem. One big step is the drastic increase in RAM speeds, compared to hard drives which have increased only slightly in that regard.
As someone else said, it is only a matter of time.
Dyolf Knip
You can learn a bit more about these results from our short paper (PDF) just presented at FAST, or wait for the June Usenix conference to see a longer paper.
They're not going to lose refresh because of power failure. No matter what the storage technology is, you don't leave a server farm like Google's at the mercy of the local power grid, you have some sort of generators for backup.
They _will_ lose bits of data. DRAM chips fail. Motherboards fail (taking out perhaps 2G at a time). Cosmic rays flip individual bits. It's much less lost at a time than HD fails, but probably the flipped bits occur far more often. But Google never guaranteed 100% accuracy...
Actually, I thought I heard that Google uses single IDE drives, in a whack of distributed generic PC's. No SCSI involved.
:-
And as several other posters commented, *YES*, I AGREE, if speed vs. cost is a factor, then the 65x caculation is less relevant. But it'd take a heck of a lot of requirement for speed to overcome a 65x cost savings (put 30x more machines in place at half the cost, and get the performance you need, with the right architecture).
And one of the most popular (my favorite) search engines might just mandate speed to the point that a 65x cost penality is *well* worth it.
Man, I wish people would *read* the posts in detail before posting. (Not that *I've* ever been guilty of that
-me
Love many, trust a few, do harm to none.
The idea that all this is on DRAM is staggering.
:)
I remember when AltaVista (back in 1996) was boasting that they had 1GB of RAM for their search engine.
But RAM was so cheap up until recently and Google uses so many servers, that I think it probably would be cheaper for them to just work out of RAM. No disk or LAN medium can match RAM for access time, transfer rate and life span and these things are probably most important to Google.
Trying to have extremely fast disk sub systems in each server in the Google farms would probably incur very high expense space, yeilding much more space than required and much slower space of that which is actually used.
I don't think this comes down to the typical MB/$ comparison between disks and RAM because Google might only have a gig or so in each server, with lots of servers.
If you're comparing a gig of really fast memory between RAM and disk, it is easy to see which is cheaper. A gig of RAM would have cost me a few months ago in Sydney ~$300.
Whereas a gig of the fastest disk I could possibly get might cost me tens of thousands for a load of 15k RPM SCSI disks and a few 64bit PCI hardware RAID-0 cards so that I could only get a meazly 528MB/s transfer rate out of, probably half that of the RAM speed and access times for the RAM would be astonishingly faster than any disk, resulting also in many hundreds of gigs that will probably mostly not be required. Far too expensive, far too ineffective. Google needs fast access times and transfer rates the most, but the fastest of SCSI systems will have their transfer rates killed by zillions of very poor access times. Random access does'nt hurt transfer rates with RAM, the way it does for disk.
In the end, these machines would probably just end up being configured to each serve what they could cache with RAM so as to keep up with the demand, so why not just boot all these machines off a little flash disk and then just work the engine out of RAM?
This does'nt just come down to needing RAM or disks that can transfer as fast as the network interfaces, since this is not a simple cache or file server. These servers need to search through their whole index a fast as possible, and doing this in RAM at super high speeds is going to be much more economical in RAM than disk. I doubt Google could even be feasible working out of disks.
War crimes, torture, lies, illegal spying... Would someone give Bush a blowjob, already, so he can be impeached?