Pleasing Google's Tech-Savvy Staff

An anonymous reader writes "Douglas Merrill, Google Inc.'s chief information officer, is charged with answering that question. His job is to give Google workers the technology they need, and to keep them safe — without imposing too many restrictions on how they do their job. So the 37-year-old has taken an unorthodox approach. Unlike many IT departments that try to control the technology their workers use, Mr. Merrill's group lets Google employees download software on their own, choose between several types of computers and operating systems, and use internal software built by the company's engineers. Lately, he has also spent time evangelizing to outside clients about Google's own enterprise-software products — such as Google Apps, an enterprise version of Google's Web-based services including e-mail, word processing and a calendar."

Comments

All Credit to Him

[Avohir]

I've had to do IT work for tech companies before, and it's like being the caterer at a chef's convention, they always think they could do it better. That he's managed to do it with a relative degree of success at a place as eclectic and high profile as google is impressive. I think the approach is novel too, although I'm not sure how well it would apply outside of their unique company culture.

Re:All Credit to Him

[zappepcs]

It always applies to other companies. The thought process it takes to create software services is what I believe should be the approach to network services. If each little group of employees is walled off the basic network, and their access outside that playpen restricted to what they need, any major error inside the playpen is less likely to corrupt the whole network. Much like a city's services are configured. Everyone needs water, electric, sewage, trash service, roads etc. If you trip the breaker in your office, the next office building is unaffected just as they are normally unaffected if your toilet overflows. In that way each can do pretty much whatever they like and all remain unharmed. I'm not saying that your hobby of cultivating anthrax is going to fly for very long, but short of that... well, you can (more or less) grow what you want in your window-box garden. You can walk down the street to the park, just not through everyone's backyards.

The idea is not to restrict people, but restrict damaging elements from hopping around your network.

Re:All Credit to Him

[Kelbear]

I think the kinds of people Google hires are less likely to run executables and install toolbars from seedy and irreputable niches of the internet. Other companies probably can't assume the same of their employees.

Even smart people can make errors of ignorance or naivetè with regards to their computers. It's nice that they've cordoned off the system to prevent them from torpedoing the whole network at once, but you still have a mess on the other side of the wall to clean up. Most of the important stuff is probably saved where they're regularly backed up(Google sure as hell isn't going to have problems with storage space) but there's definitely going to be downtime involved.

It's probably not worth the cost and risk for most companies. If someone wants or needs something on their system, just having them ask first is a reasonable approach.

Re:All Credit to Him

[TheLink]

That's fine if the walls are 100%.

If you allow some employees access through those walls to other networks, and a hacker manages to get their credentials it can start to get quite nasty.

Even if the isolation between networks is good there's also the possibility of _work_ being secretly tampered with. I'm sure there are hacker who would want to tamper with GMail or Google Desktop.

Or confidential information leaking out.

Re:All Credit to Him

[Anonymous Coward]

I think the kinds of people Google hires are less likely to run executables and install toolbars from seedy and irreputable niches of the internet. Other companies probably can't assume the same of their employees.

Exactly. IT security at most companies is designed around the belief that the average clueless user will find a way to screw something up if given too much freedom. So we lock them down in order to minimize the damage that they can do.

That's less of a problem with more technically inclined users. At my organization, we keep most of our users locked down but give our development group freedom similar to what is described in the article. They're a competent lot, fairly trustworthy and they're right across the hall. So we let them do whatever they want on their workstations, within reasonable limits.

Re:All Credit to Him

[nschubach]

Considering myself a technically inclined user (being a senior developer) I lock my machine down myself. I know it sounds backwards, but I don't want rogue applications running on my machine when I'm testing. Not even the ones used by my company to keep the system "inventoried."

Nice approach

[the computer guy nex]

Unfortunately it will take only one mistake by one employee to ruin it for everyone.

Re:Nice approach

[SanityInAnarchy]

I'm not really sure how that works.

Other than leaking source code onto the Internet, I don't really see what problems this could cause. I work at a small company with a similar philosophy -- the company buys your hardware, and certain software if you need it, but you can use whatever you want so long as you're not fighting with it on the clock.

But think about it: Spam botnets can be blocked by killing port 25 outbound. Data loss can be managed by the fact that everything's on version control, which is backed up. Traditional spyware and viruses will at worst take a machine down, at which point, it's the responsibility of whoever owns that machine to fix it -- or maybe they try to spread over the local network, at which point, staying patched and/or running a personal firewall will pretty much stop it.

The only real danger would be if we got big enough to be a target for deliberate attacks, and someone stole our source code. Google is arguably this big, but I've never heard of a leak from them. TFA does mention a possible strategy:

We have antivirus and antispyware running on people's machines, but we also have those things on our mail server. We have programs in our infrastructure to watch for strange behavior. This means I don't have to worry about the endpoint as much.

So what mistake could one employee make to ruin it for everyone?

Re:Nice approach

[TheLink]

"So what mistake could one employee make to ruin it for everyone"

Get pwn3d and:
a) Commit GMail/etc code secretly backdoored by a hacker.
b) Leak out the search ranking and antisearch spam methods/algorithm google uses. Google's search results are already not as good as they were years ago.

Re:Nice approach

[bishiraver]

a) I'm fairly certain google employees would review each others code before commits. TFA mentions they have automated scripts that check security of code.
b) I got nothin', though I'm willing to bet the search algorithm is one of those things that not many people get to see/tinker with.

Re:Nice approach

[jd142]

Well, let's say that an employee downloads a piece of software with a license agreement that allows the software manufacturer to monitor all the data the users produces, what websites the user visits, and gives the software manufacturer the right to keep that information in perpetuity. By installing the software on google computers as an employee of google, google is now bound by that license. So sensitive company information ends up being stored on the software manufacturer's computers in perpetuity. And if the license gives the software manufacturer the right to read the information you've got a really nasty can of worms.

Or how about an employee who downloads a piece of software that is only to be installed on the employee's personal computer. The employee installs it on a work computer, thinking that it is the employee's computer and is only using it for personal use. That's wrong and suddenly Google gets audited and sued for illegal software usage.

Or even better, the software manufacturer makes the legal venue the laws of Lichtensteinavania, where the user has no rights at all.

I know, I know, the slashdot response is switch to gpl, but that isn't always an option.

I've actually run into all of these software licensing issues at my job.

Re:Nice approach

[somersault]

The first hacker to take down their network, either internal or external facing, would be infamous. He'd also be killed in less than 24 hours by an army of angry geeks who want their porn back

Re:Nice approach

[forgotten_my_nick]

I think it has less to do with a hacker and more to do with litigation. IBM for example is extremely anal about what developers are allowed access when creating applications and have to account for everything they do. Because when your a large multinational with lots of money people will try to get it from you.

Re:Nice approach

[element-o.p.]

Data loss can be managed by the fact that everything's on version control, which is backed up. Traditional spyware and viruses will at worst take a machine down, at which point, it's the responsibility of whoever owns that machine to fix it -- or maybe they try to spread over the local network, at which point, staying patched and/or running a personal firewall will pretty much stop it.
That's a great theory, but more often than not, that *isn't* the way things really work. I've seen sys admins really bork config files that were using RCS. I've seen a virus take a network down for two days despite updated and running A/V and firewalls. Anyone who has worked in IT for very long is forced to admit that you can make it really, really difficult for your users to shoot themselves in the foot, but nothing you can do can guarantee security. The best firewall, the best anti-virus and the best revision/version control will give you some measure of protection, but it won't be 100% effective. Ever.

Re:Nice approach

[SanityInAnarchy]

Your logic is faulty.

Show me how.

The first hacker to take down their network, either internal or external facing, would be infamous.

And traditional viruses/spyware won't do that.

The trouble is, modern OSes are reasonably secure at this point, and you can bet the external-facing IPs are going to be locked down. Same with internal services -- some random developer's desktop might be open, but the service is going to be secure. So what you're talking about is someone actively making a "hacking" attempt at something that, to my mind, looks pretty much impenetrable.

The only other option is something more insidious -- set up a website which exploits some browser flaw, then hope someone at Google sees it. Or sit around a wifi hotspot, praying that someone logs on with a laptop that's vulnerable, infect it, set it to phone home, then pray it is actually able to phone home, and that Google doesn't take a peek at exactly where it's phoning home to.

And a successful variation of this is still just going to give you the one insecure machine. It's not going to give you the entire network. It's probably got less of a chance of doing that than if they were extremely anal-retentive in their security policy (and refused local-admin rights, etc), because it's going to be a heterogenious network.

But then, you did just provide the perfect counterargument: The first hacker to take down their network, either internal or external facing, would be infamous. Therefore, people are trying. It's not working. Therefore, whatever Google is doing for security is working.

Re:Nice approach

[V for Vendetta]

Have software clickwrap licences even been tested in court yet?

Yes. At least in Germany. Here, you, the purchaser, need to able to reed the EULA/ToS before even buying the software.

Re:NO TFA

[orclevegam]

Reload the page, it worked for me. Looks like their server is having a minor case of slashdotting.

I wish our IT was like this.

[dangerz]

With all the restrictions on tools and languages, it seems like our IT holds us back more often than pushing us forward.

I recently built an application for my group that started off in PHP/MySQL. The customers were using it and loving it, but IT said they're not interested in supporting PHP and we weren't allowed to stand up a server. After months of talk with them and compromising, it was rewritten into JSP/Oracle. Then they said we're not allowed to do that either, so we agreed on C#.net/MS SQL. I rewrote it to that and after a month, they again came back and said no way. Getting ever more frustrated (I now had the same program in several languages), I ended up in C# Desktop Application instead of web/MySQL. They've been complaining again, but we have more leverage there in that my entire group was stood up to build desktop apps. I'll probably have to switch it to Oracle, but that shouldn't be a big hit.

We wasted lots of time and money rewriting what was already done all because of politics. I always thought IT was meant to *support* rather than hinder.

Re:I wish our IT was like this.

[pongo000]

With all the restrictions on tools and languages, it seems like our IT holds us back more often than pushing us forward.

Beware of any job where IT support calls the shots. That is an incredibly inane and inefficient business model. IT support is exactly that: They are there to support development efforts, not to hinder them with brain-damaged policies usually written and enforced by CTOs that don't have a clue and administered by low-paying drones who substitute authority for what they lack on the pay scale.

Why even bother working for a company like that? With the upswing in IT, you sound like you've got way more than enough experience to find a job elsewhere.

Re:I wish our IT was like this.

[filterban]

Wow. Did you bother asking them what they would support before writing the application? That seems like the better approach to me.

If they're only willing to support a specific language, then you need to work in their requirement (generally speaking).

Re:I wish our IT was like this.

[mc900ftjesus]

Does IT make the company money? No, not a dime, they're a money sink-hole like electricity and phones. They don't call the shots just like the maintenance man doesn't call the shots. IT departments need to be enablers. When IT crosses the line from preventing you from installing tons of crap on your desktop to killing the rollout of a platform that generates revenue, someone in management should have been fired on the spot, no questions asked. IT should never dictate a product, only internal policy.

Re:I wish our IT was like this.

[dangerz]

This app started in PHP before I was here. When I came in, I rewrote it in PHP to make it more efficient and strip out some of the fat. There were emails with IT on it and they didn't seem to care. It wasn't until the app got popular and used that it became an issue.

My management did their best to fight it, but IT has a strong pull here I guess.

Re:I wish our IT was like this.

[houghi]

I feel with you. The several IT departments I wored with have the same attidute of not wanting to change anything and forbid everything that could hinder them.

The worst I have seen was where I requested an email to be send from a a system. I knew it was possible. What was even worse was the fact that they had bought the CRM package for a LOT of money, because it was able to do so.

So when I asked if it would be possible to implement it, the answer was that I needed to fill out a request. I told them I could only fill out the request if I knew how much money it would cost.

Catch 22. The procedure on how to do things was written and nothing could change that.

I have seen IT departments that were unable to remove certain rights from people if they would not need them anymore, because there was no procedure for it.

I myself had, due to human error, access to each and every place in the building. More then anybody else. When I mentioned this, they told me that because I got it, somebody must have OKed it so I have the right to it.

IT departments just LOVE procedures. Basicaly because they are so easy to put in logical yes and no questions and answers. They should start with some debugging of their procedures and realise that the real world is more then if, then, else.

It seems that the person at Google has done just that.

Re:I wish our IT was like this.

[Anonymous Coward]

I'm in finance/IT and I'd just like to say: *all* large financial companies are like the one described by pongo000.

Why not switch to a company like google ?
Simple: they pay me so much money that this form of light torture / kafkaesque work environment is still more attractive to me. The banks I work for pay me approx 4 times more than google would - this way, I can retire when I'm 40 years old (and spend time doing interesting/creative IT stuff instead of having to be chained to a corporate entity).

I work to live - I don't live to work. As hordes of clueless MBA's have had 20 years to surgically remove all the fun and creativity out of corporate IT, I have decided that I prefer to take the route that enables me to be *truly* free once I'm 40.

Re:I wish our IT was like this.

[VENONA]

Users v Admins is yet another category of religious war, and has been for at least 30 years. It's further complicated by the fact that the role of IT can (and does) vary from org to org. Sometimes it follows a role somewhat like you'd find described in a college's curricula listing, but they sometimes absorb more MIS-like functions, etc.

One large factor that keeps the war burning brightly is that the relative skills between various user communities and an administration community is also all over the map. I've seen developer groups who were purely code-monkeys, and made some very bad calls on software that they would then have thrown over the fence for an admin group to support, no matter the (large) impact on that support group, if someone from an admin group hadn't been able to do some basic sanity checking. OTOH, I've seen groups of users thrashing about trying to accomplish even the simplest thing, because some bit of software they needed had been wedged in the IT approval loop for several months.

Another factor is that admins often have little concept of what the developer has to deal with on a daily basis, and vice-versa. In my experience, this one doesn't get enough attention, and it often leads to people from different groups talking past each other, instead of helping each other.

Better communications, and a bit of experience on both sides of the fence, often helps people find some commonality of experience. I know I've usually had buddies (and people I didn't were too clueful) in both broad groups, in any org I've worked with. If nothing else, you can always band together with admins in mutual hatred of Roving Bands of Managers, thereby moving the religious wars to a different level.

I don't mean to deprive anyone of the pleasures of a religious war. If the two groups could somehow band together, but somehow not against Roving Bands of Managers, all is not lost. Developers can always fight other developers in the language wars, etc. Well, actually *both* sides can do that, so never mind. But admins can always fight the MTA wars, or similar, amongst themselves, while developers can argue about the One True Way to do IPC, etc.

As far as I can tell, it's turtles all the way down.

The question is...

[adpsimpson]

From the article:

"How do you run the information-technology department at a company whose employees are considered among the world's most tech-savvy?"

Mostly fluff

[orclevegam]

Not much to this article but there are a few interesting tidbits. A lot is in the summary, so not much need to go to the actual article, but something interesting not in the summary is when he talks about googles security environment, and why it's not really a security risk to let people install whatever they want. What it boils down to, is that the old style security of locking down the endpoints (that is, peoples workstations) makes people sleep better, but doesn't actually provide much in the way of security. Instead they focused on securing the infrastructure, such as running AV software on the mail server, and intrusion detection software that monitors the networks and servers, plus one would assume properly configured firewalls. He also mentions that being a search company they already had really tight security in place and that few people had access to customer data, so adding security to support outside enterprise data wasn't a big leap.

Re:Not actually a big deal

[Danny Rathjens]

But do they have a PR guy savvy enough to advertise that fact and the related "enterprise" products by getting a mention in the WSJ and submitting the story to /. anonymously?

Re:How?

[orclevegam]

Okay... Sounds interesting, but how exactly security and proper licensing is maintained? Could other companies emulate it? Maybe. Depends a lot on the company I imagine. Part of the reason it flies at google is because of something mentioned in the article. Almost everyone is an engineer of some type, and they all have security training. The security bit isn't as important, but as far as licenses go, most of them should understand you can't for instance bring your copy of MS Word in from home and install it on your company system. At companies with less technically inclined individuals, they may not see the problem with installing whatever software they can find on their company systems (talking from a purely licensing standpoint here, not talking about security). Essentially if Google got raided by the BSA they'd probably fair pretty well, but some other non-IT centric company might not fair as well with a similar IT policy. Of course, there's no reason for any company not to implement a similar policy for all their technical users at least.

Re:It all comes down to this....

[ccguy]

So who writes these 'automated tools' and who checks those?

Most likely they use those tools to check themselves, pretty much as you compile (most of) a compiler with itself, debug a debugger, and so on.

If you are interested in how these recursive tools work, check valgrind's documentation (interesting because it relates a bit how some design decisions were made so that valgrind could be used on itself) for example.

Not uncommon in tech-savvy organisations

[Bertie]

I also worked at a very big company which let us do this. Not company-wide, just the couple of thousand people that worked where I did, which was probably very similar to Google in terms of the sort of people who would work there. We were considered to be bright enough to stand on our own two feet. We weren't the sort to bother tech support unless it was a problem with, say, networking - applications we'd installed were our problem, and besides that we'd be more likely to know what we were doing with those applications than the average techie. It meant that if we needed a particular piece of software or equipment, we didn't have to wait weeks to get sign-off from God Himself - we went and downloaded it and our manager found the money for it if it had to be paid for. We were trusted not to buy stuff we didn't need, and by and large it worked. Treat people like adults and they'll behave like adults, mostly.

More than once I got hold of an oldish spare computer and installed Gentoo Linux on it, and the only justification I had for doing so was that Windows got on my nerves. Not much of a business case, but as far as they were concerned I was a big boy and could look after myself, and it was no skin off their nose as long as it didn't take up tech support's time.

The only thing that made us different from the tied-down masses elsewhere in the company was our level of knowledge about what we were working with. I maintain that the best security system is user education. Obviously that's not to suggest that you should throw caution to the wind, but clued-up people generally won't get you in trouble. So clue them up.

Right now I'm in a much more locked-down environment and it's incredibly frustrating. Something as simple as connecting to a printer is a nightmare because I have to go through some tech support clown who invariably knows a lot less than I do and bumbles around randomly prodding things till it works. I don't have admin rights to my own machine, and useful things like the command line are blocked. It drives me mad, and it holds me back in my work, but hey, some IT goon has an easier life because of it, so it's all fair enough, right?

Google is full of smart people, and the people in charge are clearly smart enough to treat them as such. I wish more companies would follow this example.

Re:Not uncommon in tech-savvy organisations

[KiltedKnight]

Not for nothing, but back in its heyday at AOL, you supposedly had some of the best, brightest, and most innovative developers... yet a lot of them were NOT email savvy at all. People would just download and open attachments from random, unknown people without performing a virus scan or anything like that.

Just because you have some brilliant techies doesn't mean they are all security conscious as well.

Re:NO TFA

[Coraon]

...I was looking for pictures...I'm in lust with the google building.

standards-compliance

[PigleT]

The reason this works is because he's a sensible fellow who knows standards-compliance. both in network protocols and data formats, is more important than the mere name of the OS or application issuing them.

Quick Story

[Cytlid]

I've actually experienced this type of thing in the last two jobs I've had. Allow me to explain.

  I moved from my job in NY as a System Admin for an ISP. I won't name names, but our major tech we used was Cisco, Solaris, Linux and VMware ESX.

  My family and I moved to SC for the nicer weather ... I landed a job as Sr Network Engineer for an ASP. I thought, ASP, can't be too different. Well 800 miles away, some things are the same, some are different. I'm a command-line, CLI type guy. The ASP is an MS Gold Partner and takes advantage of Citrix. All the network gear is Cisco (which is where me and my team come in). I thought, oh great ... I don't belong here (except for the Cisco stuff). For the record, we do have *some* Linux hosting and colo.

  But I setup a few smallish vmware servers and I'm happy. I have my Linux-in-a-box. I've done a bunch of grepping and typing and scripting and such this morning, and I found some new issues that I didn't see before without seeing the "big picture".

  So back to my point. I'm very picky about the apps I use and whatnot, so it's hard for me to "conform" to an IT ruleset about what can and cannot be run on company machines. The ISP I worked at was very flexible in this manner, for some reason I expect this out of the new job.

  Our business model is we sell these published apps and hosting to our customers. We run a large private MPLS network and connect many smaller places to us. They can run Office 2007 from a website.

  Then it hit me. Things have been getting really optimized in the last year or two, so we're using our own stuff. My office apps "live" in a website. The revelation came that now, when it comes to my laptop (or desktop), I can do whatever I want. Notice this is typically a nightmare for common IT shops, but many of our smaller customers think IT is a pain and will be happy with published apps and thinclients. For someone like me, who is tech-savvy, I can format my machine and install Linux (some of the other guys have already done so). Because there's a Citrix web client for Linux (I use it at home). Involve virtualization in the mix, and our datacenter becomes one giant network, one giant machine that we manage and the apps are just floating around inside. We manage all the security and whatnot, and keep it running.

  So in a way, you really can have it both ways. We're not a Web 2.0 shop, but our method is definitely Another Way to Do It.

Re:How?

[bishiraver]

I'm willing to bet that any licensed software is freely available from internal google downloads, along with the legal license to said software. Google has the money to, after all.

Last Adopter

[salesgeek]

IT departments are typically the last adopters of anything. They typically roll up to the CIO, who typically is not a real C level executive. The CIO typically works for the CFO and is an advisory member of the executive committee in most companies. Information Technology generally has two crucial corporate functions: automating accounting functions and managing corporate communication platforms like phones and email. Everything else that happens on a computer - i.e. productivity applications, intranets, etc... are side effects of putting general purpose computers on desks and are secondary functionality. IT Departments have generally claimed fiefdoms over all things computerized so they can have bigger budgets, more resources and are harder to fire and outsource. It's ugly. But true. Most IT innovation starts in some department, and goes like this:

• Kid in sales writes really cool web app that sells product automagically on MySpace.
• IT finds out about it, can't integrate it with accounting, tries to kill it.
• Kid freaks out because someone who is three managers over him is calling him asking what he's doing.
• Kid's boss freaks out because CIO is calling his employee.
• Project is killed when Bosses Boss finds out about it because it doesn't make sense to him OR - Bosses Boss intervenes and tells IT to stuff it, and counts money from sales from web app.
• IT is forced to support web app because CFO now needs to book revenues for month or quarter.
• Kid is transfered from sales to IT and leaves company one year later to start company that sells MySpace widgets and goes on to become millionaire.