A Glimpse Inside Google's South Carolina Data Center

miller60 writes "Google today released a video showcasing the security and data protection practices in its data centers. Filmed at the company's South Carolina data center, it provides a look at Google's wiping of data and (literal) shredding of hard drives."

Comments

Amazon's East Coast Data Center

[0100010001010011]

Amazon also released photos of their East Coast datacenter today.

Re:Amazon's East Coast Data Center

[bev_tech_rob]

That looks more like Arkansas than the East Coast! (and I am FROM Arkansas)!!! LOL

I want a video of Amazon's data center

[jbplou]

I want a video of Amazon's data center about 36 hours ago instead.

Ultimate Security

[ThePromenader]

...of course there's no better way to protect your data - my basement door is securely locked, and I shred my HD's daily. And mom rarely lets anyone past the front door.

Re:Ultimate Security

[Anonymous Coward]

You don't know yo mama like I do.

Re:Ultimate Security

[eln]

That may be true, but from what I hear, your mom's back door is wide open.

Shredding hard drives is a pointless waste.

[Anonymous Coward]

Call every data recovery company you can find and ask them the following:

"I have a hard drive which was zeroed out, with one pass, accidentally. Can you recover the data for me?"

You will not find a single "yes" answer. It's impossible. It's a myth, or a theoretical attack.

Maybe the CIA should worry about stuff like this, but you shouldn't, and Google really shouldn't. Those hard drives could be reused or recycled.

Re:Shredding hard drives is a pointless waste.

[chebucto]

They're only being discarded because they've started to fail. So giving them away would be a bit of a dick move, regardless of whether it's a privacy threat or not.

As for the shredding, my bet would be that they're just following a data-destruction spec from 10-20 years ago, when wiping really wasn't a surefire way to destroy data.

Re:Shredding hard drives is a pointless waste.

[nacturation]

Call every data recovery company you can find and ask them the following: "I have a hard drive which was zeroed out, with one pass, accidentally. Can you recover the data for me?"

You will not find a single "yes" answer. It's impossible. It's a myth, or a theoretical attack.

If the hard drive had any bad sectors which were automatically reallocated from the pool of spare sectors, your "accidental" zeroing of the sectors would not have cleared those. Therefore, there is the potential for some data recovery even if it's only a few kilobytes at a time. Additionally, it's impossible to visually tell the difference between a drive with all data intact and a drive that had been zeroed out. Shredding the drive removes all doubt as to its status.

Re:Shredding hard drives is a pointless waste.

[Anonymous Coward]

IMO, they should have invented a robot to disassemble the drives and chuck the metal parts into a large recycling bin, and run the platters through the shredder instead. Much slower, but the largest chunk of metal in a drive is the case itself, and contaminating the metal with all the PCB parts is just makes it all wasteful.

Or maybe I'm missing something and there's a way to separate all the rare metals in the PCB chips from the recycleable metals when they're all shredded together without having to re-smelt everything.

Re:Shredding hard drives is a pointless waste.

[fuzzyfuzzyfungus]

I'd be curious to know if (once a drive is dead or failing) shredding reduces its value, or whether any recycling procedure would just start with shredding anyway. A pile of shredded drive chunks should be substantially richer in copper, nickel, rare earths, aluminum, and iron(and possibly gold) than many ores considered to be commercially viable. I imagine that it comes down to whether it is cheaper to get a cleaner separation at the cost more labor, or just grind 'em up and let the refining process sort it out...

Re:Shredding hard drives is a pointless waste.

[Seumas]

Yeah, sure. Google will just invent a hard-drive disassembling, sorting, and recycling robot. Are you fucking nuts? What's next, Google will just invent some self-driving robot car?!

Re:Shredding hard drives is a pointless waste.

[Anonymous Coward]

A very, very common failure mode for a hard drive is that it continues working until either the electronics or the mechanics of the drive fails. At this point, it's too late to zero it out.

Now that it has failed, how does one erase it? Well, one can either try to put the platters in a new enclosure with fresh mechanics and fresh electronics...

or one can destroy it.

Guess which one is cheaper. :-)

Re:Shredding hard drives is a pointless waste.

[Animats]

Google probably shreds them so that they don't get bought by some low-rent operator and show up in "new" machines.

They're low-end drives, incidentally. Google uses cheap parts and redundancy, accepting that hardware will fail regularly. I'm surprised they even bother to test failed drives.

Re:Shredding hard drives is a pointless waste.

[heypete]

They're only being discarded because they've started to fail. So giving them away would be a bit of a dick move, regardless of whether it's a privacy threat or not.

As for the shredding, my bet would be that they're just following a data-destruction spec from 10-20 years ago, when wiping really wasn't a surefire way to destroy data.

I would think that they'd be shredding (and crushing) the hard disks because it's faster than sitting around and waiting to overwrite disks, especially with larger hard disks. Shredding also works on disks that are damaged and unable to function.

Re:Shredding hard drives is a pointless waste.

[JWSmythe]

    I knew someone in the IT recycling business. They had some big customers interested in security, such as the DoD. They had a machine much larger than the one shown, which would shred anything put into it. The guarantee was that every piece that came out would have no dimension larger than 0.25 inches.

    They sold this mixed scrap metal to other companies who had methods for sorting the various metals out, and then they were paid based on the total metals. This included sending off electronics for their precious metals. They made some good money from the precious metals contained within the electronics, although they were impractical to separate on a small scale.

    I didn't quite get the Google demonstration on their destruction of a drive. First they wipe it, verify it's wiped, bend the plates, and then shred it? Why? It would save a lot of time and manpower to just shred them.

    I work for a company now that is under strict guidelines, both by contracts with 3rd parties and federal law. There are stacks of old unserviceable hard drives (don't work, and are generally under 40GB). are currently kept in a safe. The agreement I've come to with everyone is that we will be melting them down. A little (say 5 pounds) of home-brew thermite, and we'll have a nice chunk of molten metal to give to a recycler. If done properly, you wouldn't recognize any piece of it as being a drive when we're done. Good luck trying to recover from *that*. :)

Re:Shredding hard drives is a pointless waste.

[AngryNick]

If the datacenter is really where the video shows it to be (I have doubts), then it would be about 1/2 a mile from steel and metal recycling facility.

Recycling raw materials has been common practice in manufacturing for decades. And if you can't reuse that material in-house, then you do what you must to get the best price/lowest cost to get it the hell off site. My guess is that they get a better return on pre-shredded metal and get the assurance that some red-neck isn't going to take a truck load of their 'specially manufactured' drives, write "Samsung" on them with a Sharpie, and then try to sell them at the Charleston Market for $5 each.

Re:Shredding hard drives is a pointless waste.

[DaveGod]

I didn't quite get the Google demonstration on their destruction of a drive. First they wipe it, verify it's wiped, bend the plates, and then shred it? Why? It would save a lot of time and manpower to just shred them.

Shredding requires extremely noisy machinery and therefore it would not be practical to hold it in the most secure area where the drives are kept. The shredding is probably a redundant step, partially to catch any screwups and partially just to allow customers to tick off the "shreds drive" requirement box.

I'd expect the scrap would also be worth a lot more, just having run it through a fragmenter can double the value per ton.

If you thermite your drives I'd assume you'll end up with a not-so-nice chunk of all the drive materials melted together. Shredding the drives like in the Google video is essentially putting it through a fragmenter, stage 1 of standard recycling processing. Magnets can then be used to separate out the ferrous metals, and so on.

Re:Shredding hard drives is a pointless waste.

[swillden]

Shredding hard drives is not pointless, and neither are the other steps taken.

It may seem redundant to first wipe the drives, then shred them, but if you think about it both steps are necessary. Wiping them is the best method to ensure that no data is recoverable, but remember that drives are pulled from service when they're failing. Can you trust a failing drive to successfully zero itself? Even verifying that you can successfully read all of the zeros from the disk after writing them doesn't prove that all of the data is gone, because a failing sector may have been "relocated" to a spare sector -- but that doesn't mean the failing sector is necessarily completely unreadable. And, of course, many failing drives won't operate well enough for the wipe/verify process to complete successfully.

So, given that after the wipe operation data may still remain, it's necessary to physically destroy the data. So why not just shred the drives, rather than bothering with the wipe? Because a small piece of a hard drive platter may contain a significant amount of data. Actually recovering that data is probably quite difficult, but it is possible.

I'm somewhat surprised they don't go a step further and use an acid bath, but I suppose they figure after wiping has gotten rid of most of the data and shredding has made whatever's left very hard to retrieve, the drive fragments are shipped to a trustworthy recycling operation and any remaining data will quickly be destroyed by the recycler. Of course, the recycler could do the shredding, but since it has to be done anyway Google may as well do it up front.

Plus some customers undoubtedly have security policies that require wiping and shredding.

If you want to argue that the shredding is unnecessary, what you should point to is the fact that all of the customer data on the drives is already encrypted, with keys that are not on the drives. That arguably makes all of the destruction steps unnecessary.

(Full disclosure: I work for Google but have nothing whatsoever to do with data centers or hard drive lifecycle management, and no information about how this stuff is done beyond what I saw in the video.)

Re:Shredding hard drives is a pointless waste.

[justthinkit]

When I worked at the U. of B.C. I was told to take a way old computer to SERF, the campus recycling center for them to re-purpose. A short time later I got a call that it had been "stolen". I never bothered to take anything to them again.

If I am a PHB at Google, I would not want people trying to get their hands on discarded hard drives as it would be happening on company time but for attempted private gain. As soon as a drive is shredded, geek interest (and lost work productivity) ends.

Re:Shredding hard drives is a pointless waste.

[bill_mcgonigle]

They're only being discarded because they've started to fail.

This is surprising - they must pay very low electric rates. I've consolidated a bunch of 300GB drives onto a pair of 2TB drives and the power pay-back is on the order of a year.

when wiping really wasn't a surefire way to destroy data.

It still isn't. Drive manufacturers won't tell you if their drives are certified to correctly implement ATA Secure Erase. Without it, all your re-allocated sectors still have the raw data.

I use block-level encryption since learning this.

Re:Shredding hard drives is a pointless waste.

[JWSmythe]

Shredding the drives like in the Google video is essentially putting it through a fragmenter, stage 1 of standard recycling processing. Magnets can then be used to separate out the ferrous metals, and so on.

Thank you. Now it makes more sense. Well, at least the shredding versus melting. Since the unit they're using did not appear to be too large, all it would require is a soundproof room, or a room separate from the offices. One site I did work at had us dispose of our trash ourselves. So we'd tote all the shipping boxes and miscellaneous trash down to the basement, where they had 3 compactors. One was for paper and cardboard. Another was for plastics. The third was for other trash. They were big and noisy, but being in the basement, the noise didn't disturb anyone. That was contained in the "secure" area of the datacenter, behind the mantraps, and ID checkpoints at the entrances.

Re:Shredding hard drives is a pointless waste.

[Bengie]

The magnetic fields are stored as analog waves which take different forms based on the previous data stored. If all you did was single pass zero a HD, the bits that use to be ones will look different than bits that use to be zeros because of the shape of the wave. The problem is getting access to this low level data. You need custom hardware that doesn't return ones and zeros but the shape of the magnetic fields.

No one but powerful or governments could get access to this equipment... or anyone who owns a company that makes harddrives.

You're not going to get a 3rd party private company that has the equipment that can do this, you're going to have to ask the FBI.

So, I guess it does come down to "unless you're trying to hide information form the government, just dev/zero the HD."

Re:Shredding hard drives is a pointless waste.

[teslafreak]

My interpretation was that the zero and zero verify was actually not necessarily related to the destruction. I thought they were saying they zero'ed it, and then ran a test to see if the drive was perhaps still viable (after a clean wipe). Failing that test, they then destroyed it. I could be wrong on that though.

Re:Shredding hard drives is a pointless waste.

[Cramer]

Agreed. The usual answer will be no, unless you have very deep pockets. The effort to recover data is almost always not worth what you're trying to recover.

The whole reason for them being discarded is they'd started to fail or past their safe usable lifetime. Anyone willing to buy them is simply burning money.

Google servers

[Compaqt]

Anybody know if the "Google web server" at the same website as in the article is actually real?

I mean, do they really have a 6-inch battery contraption hanging off the side of every one of their web servers?

Re:Google servers

[Anonymous Coward]

yes. each webserver has a builtin PSU+UPS.

Re:Google servers

[Lennie]

As the other reply says, it has a UPS+PSU builtin, it is actually a lot more efficient. Because a normal UPS converts the power coming into the UPS for the batteries, then it gets converted again when it is sent to the PSU, the PSU converts it again...

All of that is a waste.

If you have the UPS next to the PSU, you convert the power ones coming into the PSU, it sends power to the board/CPU and so on and also to the UPS if it needs to be charged. When the PSU does not get power, the UPS delivers power to the board/CPU, no conversion needed.

From an older article:
"This design provides Google with UPS efficiency of 99.9 percent, compared to a top end of 92 to 95 percent for the most efficient facilities using a central UPS."

http://www.datacenterknowledge.com/archives/2009/04/01/efficient-ups-aids-googles-extreme-pue/

That's nice and all.

[fuzzyfuzzyfungus]

So, hands up anyone whose privacy concerns RE:Google had to do with people stealing hard drives or breaking into datacenters, rather than Google mining them...

Anybody, anybody? Bueller?

Sure, the fact that the datacenter isn't a shack with no access controls is nice; but mostly from an uptime and efficiency perspective. When it comes to large web players, Google definitely among them, physical attackers are so far down the list of information security concerns that they might as well not rate(for the users, that is. Obviously the operators would face significant costs if people were breaking in and grabbing stuff all the time).

Re:That's nice and all.

[Jeremi]

So, hands up anyone whose privacy concerns RE:Google had to do with people stealing hard drives or breaking into datacenters, rather than Google mining them...

You and I might not worry about that, but keep in mind Google is trying to convince government and industry to outsource much of their internal email and other IT operations to Google's servers. I'd imagine they would like to be reassured that nobody will walk in and grab their confidential data.

Re:That's nice and all.

[Guspaz]

Data center robberies are actually rather common, so physical attackers should definitely be pretty high up on the list. A google search for "data center robbery" turns up tons of results. One particularly bad offender is C I Host, who had their data center broken into four times in three years. At least one of those times, someone cut through the wall of the datacenter to gain access. Other times, well, it turns out that pointing a gun at someone is a rather good way to get around all that fancy security.

Re:That's nice and all.

[mlts]

People don't think of data center robberies, but with the economy in the skids, the guys who would rob banks are starting to wise up to data centers.

Until recently, the most security a data center would have on staff would be a guard in front, and maybe another to run rounds. Data center locks are intended to keep geeks and skulkers out. Most places do not factor in people who will be more than happy to blow the brains out of the secretary at the desk to get her badge and keys, so they can get access to the server room.

Having the data center be at an unmarked location was a great help too. However, this is something being wised up to, and more sophisticated thieves are starting to find ways to locate them.

This is why that IBM, EMC, and other storage makers are making encrypted hard disks for the basis of their SANs, where the array checks with a server on the LAN segment, gets a key from that to unlock the individual drives. Pull a drive or drawer out, and there will be no useful data (even barring the fact that the drives are RAID members which requires a full quorum for access.) This is also why DAR encryption on the server end is becoming an important selling point for US Federal contracts.

As of now, a lot of data center robberies tend to be inside or well-skilled jobs (like the people who ripped off Peter Gabriel's webserver.) However, the bad guys are getting smarter. I wouldn't be surprised that data center robberies become a common plague, especially due to the fact that the other ways of criminal activity (bank robberies, drug sales) are either "owned" by a more well-armed organization, or the risk/reward is not worth it (car theft). Couple this with the the knowledge that a criminal can make money not just selling the stolen server hardware, but money from selling data on the drives, and even holding the drives for ransom [1].

Realistically, physical robberies are low on the list compared to network intrusions, compromised/disgruntled employees, and laptop theft. However, this is something a data center planner needs to factor in, having some sort of duress/holdup alarm, possibly coupled with remote monitoring of the area via CCTV.

[1]: A lot of companies have little or no backup policies. A lot of businesses would pony up if handed a ransom notice for stolen drives because the data is so vital for their day to day operation.

Re:That's nice and all.

[jroysdon]

I'm not sure why anyone needs to be where they can physically access the guard. Why not use a man trap where a card access badge is used to enter the trap, the outer door secured (locking the person in), the ID of the person is verified (local scanner and camera), and then the inner door released? All of this can be done without physical staff anywhere near the location. If you have many of these such data centers it makes more sense to centralize your security staff anyway.

Cameras, motion sensors, and other methods can keep the physical building able to alert, and law enforcement called in to deal with issues since any rent-a-cop guard isn't going to be shooting anyway.

Re:That's nice and all.

[Nethead]

The man trap is SOP for Switch and Data, er, Equinix data centers. Amazon security is on par with Google. Go into a data center and you'll note the Amazon cage right away... it's the one that looks like a prison camp.

Re:Oh... Moncks Corner SC... not Coroner

[fuzzyfuzzyfungus]

Most of the time it's just an oxygen-displacement fire suppression system accident. Occasionally, it's Sergei quietly suffocating a suspected mole behind an out-of-the-way row of racks.... The two look pretty similar after the fact; so you have to call in the professionals.

And be quick about it. A body left in the hot aisle will be a bloated, putrid mass alarmingly quickly.

Hard drives need upgraded

[drmacinyasha]

Did anyone else notice in the video at 00:53 that the guy is assembling the server... With an IDE hard drive?

Re:Hard drives need upgraded

[Anonymous Coward]

Hey, we need some hardware to shoot a video.
  Here's some shitty old stuff we don't care about anymore.

Re:Hard drives need upgraded

[Mashiki]

What's wrong with IDE? You can pick up crates of 500gb drives for dirt cheap these days, about half the cost of what a SATA will cost, about a 1/3 of the cost of scsi, and about a 1/10th the cost of fibre channel.

Re:Hard drives need upgraded

[InfiniteZero]

Low speed, non-hot-pluggable, and a mess that is the IDE cable.

Re:Hard drives need upgraded

[vikisonline]

Wow, people always find something wrong with everything. Those IDE drives are for booting only. Since they have uptime in years, it doesn't matter that they are slow, but they are significantly cheaper when you are buying them by the thousands. It would be a fail to think they would store anything needed on such servers, other than os. The servers are probably linked to a harddrive farm by network or fiber-channel.

Re:Hard drives need upgraded

[proxima]

It would be a fail to think they would store anything needed on such servers, other than os. The servers are probably linked to a harddrive farm by network or fiber-channel.

Wrong. Google stores its data all over the place, including on each individual server. They designed their own networked filesystem for the purpose. If they really didn't store data locally, they'd almost certainly PXE boot and avoid drives on each server altogether. I suspect the video just used some dated footage (from a training or other internal video perhaps?), as this article clearly shows SATA drives. Every server has two drives, and since no one node is critical for anything they also wouldn't bother with RAID1 for an OS boot drive as you suggest.

Re:Hard drives need upgraded

[Anonymous Coward]

What a pity Google didn't realize this, right? Quick, apply for a position with them so you can help them out, since you're obviously so much smarter than all the Google employees in charge of hardware. ;)

Re:Hard drives need upgraded

[hanshotfirst]

None of which matter in Google's general architecture. It's not like they're running a bunch of W2k8 RAID5 servers. Everything is massively redundant and replaceable.

Re:Hard drives need upgraded

[LVSlushdat]

What's wrong with IDE? You can pick up crates of 500gb drives for dirt cheap these days, about half the cost of what a SATA will cost .....

Not necessarily so, at least in the case of laptop IDE vs laptop SATA.. I have a sideline "business" buying up broken Dell laptops, fixing/cleaning them up, installing Linux, and reselling. Since 99% of these systems come with no drive, I have to factor in a new drive for the system, and for the older IDE models, I'm having trouble finding 80-120GB IDE drives at a competitive price, while the equivalent SATA drives are dirt cheap.

Re:Hard drives need upgraded

[lucm]

> It's not like they're running a bunch of W2k8 RAID5 servers. Everything is massively redundant and replaceable.

I love those RAID5 servers, too bad they are not redundant and replaceable.

Re:Hard drives need upgraded

[Bing Tsher E]

Laptop 2.5" drives are a whole different category.

Re:Hard drives need upgraded

[bill_mcgonigle]

about half the cost of what a SATA will cost

How's the power cost? Are these old inventory or are drive manufacturers charging a premium for SATA (when it should be the cheaper option by this point)?

Re:Hard drives need upgraded

[Mashiki]

If you're pulling the entire rack when it fails, non-hotpluggable doesn't matter too much especially in terms of redundancy.

Re:Hard drives need upgraded

[Mashiki]

They're still making new ide drives, and probably will for another 5 years. But you can get 4200-5200rpm drives in the choice of your brand pretty easily.

Re:Hard drives need upgraded

[matty619]

Looks like they're also installing ECC PC2100 Memory. Wow. That file footage has been collecting dust.

Re:Oh... Moncks Corner SC... not Coroner

[Seumas]

Well, when *was* the last time you saw Chris DiBona, huh?

Re:Skip this blog spam

[Jello B.]

WARNING: Not a goatse. Goatse seekers don't waste your time on this legit link.

Re:Google is so racist.

[ShooterNeo]

Hey, google hires everyone. They just put the people in the jobs they are good at.

Re: surprise--

[Ganty]

Sometimes these spammers have no clue.

Ganty

Serious question:

[cyberfin]

Why did they do a zoom in to Finland during the few last seconds?

Re:Serious question:

[moonbender]

Revealed: Google’s new mega data center in Finland

Re:Serious question:

[gedw99]

http://www.youtube.com/watch?v=aFhlGgcU0To

security guards

[fragfoo]

Imagine a conversation those security guards probably have: Random person: So... where do you work? / Security guard: Well i work at google. / Random person: Whoaa, lucky you! So what do you do at google, are you a programmer, a security expert? / Security guard: Hmmm, something like that. /

Speaking of Google security

[e9th]

A woman who claims Google is "inside her head and making her do things" followed a visually-impaired worker into their headquarters.

Vera Svechina, a self-described filmmaker and former stripper, walked undetected into Google's main offices on March 14 and spent several minutes there, Mountain View police spokeswoman Liz Wylie said.

"An administrative staff member returned to her desk and found a book in Russian as well as a letter addressed to the two founders," Wylie told Reuters, referring to Google co-founders Sergey Brin and Larry Page.

Re:Speaking of Google security

[e9th]

You do realize that an empty desk outside of one of the co-founder's offices isn't exactly "a random Google office," right?

Re:Speaking of Google security

[mjwalshe]

unfortunately this is not uncommon I once had a chat with a very senior Guy in BT whose first job was opening the CEO (well the postmaster general at the time) mail. One chap kept writing to the Postmaster General about the evil organization that was bent on taking over the country - which in his eyes was the BBC

Re:Speaking of Google security

[e9th]

Well, if you're going to send rants to the Postmaster General, it's only fair that you mail them rather than personally dropping them off outside his office.

Re:Sexy tape libraries.

[Anonymous Coward]

I love those 8500's, but I can't believe they only have like four drives per robot.

In the video you see five libraries with between 12 and 24 drives per library. But you cannot see how many storage cells and how many robots are in each library. That would put the drive to robots ratio between 1.5 and 6 for the libraries you see there. Maybe the number of drives was not chosen to match a desired robot to drive ratio, but a desired storage cell to drive ratio. Those libraries only support up to about 10k storage cells. If they need 1000 storage cells per drive, they cannot put more drives in each.

That gives us at least three different guesses at why they have so few drives:

1. They need many robots per drive for fast mounts
2. They need many storage cells per drive for storing data for 3-4 months
3. The libraries are new, and will get more drives later

It looks like the libraries on the left have 24 drives each. Maybe those on the right will get that many drives as well.

7 out of 10 could do better

[mjwalshe]

Not if they are pitching for government contracts - and that site well the face is a bit pussy it should be 5 foot higher and should curve inwards. They should have cut down the woods as on 2 sides of that facility the woods came up to the fence. And out in the country like that they should have just dug a moat it looks like they build a pond for cooling water any how

mining inside (c)

[enosdan]

So, hands up anyone whose privacy concerns RE:Google had to do with people stealing hard drives or breaking into datacenters, rather than Google mining them

It's a wild world but you're safely locked inside a cage - together with Dr. Evil himslef.

Re:Encryption wont protect you

[swillden]

Nope... predictions are that mainstream encryption (eg. AES256) will be trivially crackable within 5-10 years

Cite?