Slashdot Mirror
Where Does Google's Hardware Go to Die?
An anonymous reader asks: "I was talking with a co-worker today about how Google is so big, and how they make such great use of commodity hardware to do their business, and one of the topics that came up is what Google does with its old hardware. Google has been around for many years now, they have more machines than any sane person would own, and they are continually expanding. At some stage they have to push out old equipment, either when it starts entering into its MTBF limits or it's been depreciated down. Searching (using Google of course) wasn't particularly fruitful. Has anyone seen where Google's hardware goes when it dies?"
/dev/null
I wonder if I use bold in my signature, people will notice my posts.
I'm guessing that if its actually *DEAD*, they throw it out [read: some lucky employee gets a dead server to putz with], otherwise they probably keep using it in some form or another.
Video Production Support
Seriously I'm sure it would depend on what dies. A disk drive would get thrown and replaced, a motherboard frying is more serious (but of course you can rescue the memory and media). The nice thing about scaling out rather than up is that as newer hardware comes along you don't need to replace the old stuff; so why through something out unless it's dead beyond recovery, in which case it's useless to anyone and off to the recycler it goes.
When companies are so big such as Google they can't simply throw away their equipment. Probably they're donating the working hardware to schools or communities. Donations are tax deductible in the US, so they're actually saving money while getting free positive publicity.
As for the broken machines, there are companies that make money off getting old hardware from businesses and recycling raw materials, so I think Google is doing the same. Here in europe there are high fines if you're caught throwing polluting stuff (electronic device are filled with polluting materials) without disposing of them properly, and I don't think in the US the law is very different from here.
Most of the old hardware gets sold to MSN and Yahoo. Really old hardware gets sold to MS for use on .Net dev boxes. The newer stuff gets sold to MS for MSN search, but that's only if they have 640k of memory.
In the words of a googler: We buy crap.
I'm willing to bet that once the hardware is too crappy for Google, that it's completely useless for anyone remotely sane.
Look for completely broken hardware at recycling places.
-- To dream a dream is grand, but to live it is divine. -- Leto ][
Probably the most annoying issue they have with failing hardware, is new versions of the used components, causing incompatility with the old hardware components and the operating system. I imagine they try to keep it running as long as possible by shipping broken machines to a central location and use parts of those to keep the datacenters as coherent as possible. So a setup like one datacenter runs generation-5 years, 2 run generation-4years, 3 run current. This way you can at one moment decide to update a complete datacenter instead of hundreds of machines spread across several datacenters.
My wife's sketchblog Blob[p]: Gastrono-me
as opposed to slashdot? One may have a clue, the other can make beowulf cluster jokes. Which do you prefer?
Monstar L
There's no such place as silicon heaven! You don't get little calculators with wings playing harps .. etc
The guys from the Wayback Machine come round and archive it.
One line blog. I hear that they're called Twitters now.
Seriously, think about it. As long as it doesnt break down, the machine can still get search/indexing/crawling jobs, or take over part of the distributed storage network. Its already installed, its working, and even if its slower than the others, the system google is using doesnt really depend on individual machine speed.
And after it broke down, they are going to dispose them, i guess.
HI O WISE PRINCE. WHT TOOK U SO DAM LONG?
I think it is an absolutely horrid thing of current American society that so many people always run after new stuff and never even bother to think about others when dumping old stuff. I've seen companies trash hundreds of computers (yes, actually trash them... because the HDD has sensitive data and because of taxes). I think that this should somehow be stopped. One way would be to heavily charge for the disposal of things containing lead, and remove all taxes for donations of educational supplies to needy institutions within the US or abroad.
That way, companies can do a good zeroing of their hard drive and then send off the PC to an organization that will take it to Africa, India, China, or somewhere else with a shortage of computers. Seriously, kids in Africa who have never touched a computer before would really be able to make use of a lot of thrown-out pentium-1, 90mhz systems. It's not funny that US society just trashes this stuff. It's such a wasteful thing to do in this world.
My supervisor (remaining unnamed) had a laboratory cleanup and hesitated throwing away anything - he almost put a cordless phone in the trash before I had to grab it out of his hands! The thing has lead in it, and for gods sake works! Some poor kid could use that thing in this world, and not everyone is as rich as he is to be throwing away a working phone! They also threw out this giant heavy "Communications Biophysics" plastic poster. I had to yank that out of facilities because it is recyclable. (Welcome to MIT. I wish people recycled more and thought about the world a little more here before they ran around inventing stuff.)
You have to watch what you donate and give away to employees. You want to give away equipment that can still be maintained in a usable state. Once, I had to get rid of a bunch of obsolete monitors, and a group of employees were actively requesting them. Instead, I donated them to an Electrical Engineering professor at a local college. He tested each of them before doing anything with them. One caught on fire and caused a serious mess. Big problem! The Electrical Engineering professor was skilled (and ready) for this sort of thing, so it was okay in the end. If I gave these monitors to employees, their houses could have burned down!!!
Lesson: If you give untrained employees or volunteer organizations equipment, make sure it works! Sure you can give the stuff away with a "no guarantees" label. However, your employees are still expecting "safe" equipment that reasonably works. Unless you are confident that you are giving away "good" kit, only send the equipment to trained professionals.
Like this early rackmounted array of Google servers which was displayed at the Vintage Computer Festival in 2005 and now is (I believe) on display at the Computer History Museum (which is worth the effort to tour if you are near the Palo Alto/Mountain View California Area.
From my write-up about the rack: The rack in the picture holds 4 standard Pentium II Motherboards per level and has a total of 80 Linux (2.0) servers per rack. Since they were standard MBs they had to get creative with things such as wiring and insulation (which was, in this case, cork-board.) The panel shows the server room as well as talks about the fire dangers of doing such a design. (Google is a neighbor to the Computer History Museum BTW). (closeup)
"Enjoy what you're doing! If it becomes drudgery, you're doing it wrong!" - Jim Butterfield
Of course there is. In fact, all the calculators are female and they fly upside-down with the number "58008" on their screens.
Godless heathen.
I was fortunate enough to get a tour of the google campus once and i asked them about thier server and what they run, etc. One of the most surpising things i found out is that most of thier servers dont have cases. They do this so its super easy to swap out hardware if something dies and to perform any upgrade that may be needed. If a part dies, they can yank it out and replace it without even having to take the server out of the rack. The dead part simply get tossed in the trash and the server continues serving its merry a** off. To me that means thier servers never 'die', they just constanly get new parts and the parts that do die are so dead you wouldnt want them.
There is nothing to sell.
This whole discussion is moot as Google simply does not have computers per se, just components. There are no cases, no monitors, just motherboards, CPUs, RAM and, maybe, storage. There's no reason for video, sound, peripherals either.
All of the components would be run until they fail. They would be not usable at that point. Nor would they be counted as a "computer," only an aggregate of parts. The failed component would be replaced and the rest kept on working until some other component fails. Effectively the "computers" would be immortal, but the components would be perpetually replaced.
-- Dan Jenkins, Rastech Inc.
"CPUs, RAM and, maybe, storage. "
Oh, I imagine there is at least some storage involved...
Unless they cache the internet in ram
Google has been transitioning to RAM-based storage. With redundant servers around the world, power outages are no problem. RAM is way faster; a disk can only do about 100 to 400 (commonly 200) seeks per second.
I've seen toasters with wings though... I wonder if they were running Java?
On a good day we could expect to completely shred or rehabilitate 20,000 pounds of electronics, the vast majority being CRT glass, with quite a bit of metal and plastics as well, so I suspect Google's output is not beyond a single company's capabilities. The company philosophy frowned on the idea of sending semi-obsolete equipment to third-world companies for a 'lease on life' as those companies are not really equipped for repair or disposal of dead electronics.
I may make you feel, but I can't make you think.
Given their existing parts-swapping policies, and the level of hardware they're using, I'd guess Google doesn't buy much NEW hardware, but rather, probably buys motherboards etc. from the various e-waste recycling outfits -- which charge about 16 cents a pound for sorted motherboards, RAM, HDs, etc. if you buy it in bulk. It only takes one entry-level employee to test such stuff to the "does it power on and boot up? Good enough!" level that is all Google really needs, given their massively-redundant setup.
~REZ~ #43301. Who'd fake being me anyway?
I bet there's a big landfill site somewhere near Google HQ, but its probably really low resolution in Google Earth...
B
In google search, type, "Where Does Google's Hardware Go to Die?" and click the "I'm feeling lucky" button. No, seriously, do it. You will get the right answer.