What about Shingle recording ? There are prototype drives already built that can near 100TB on a single device. It has limitations, of course, like 2GB+ block sizes, but it'll get interesting. Give it 3-5 years (as usual, for mythical tech that might ship someday).
So, this is bullshit. You can keep secrets as long as the people involved think secrecy is warranted.
Google have an astonishing track record of not leaking projects to the press. They've worked on some incredible stuff, and the vast majority don't get leaked at all, or get leaked accidentally. Huge numbers of internal/infrastructure projects never get told about outside the company. Sure, some projects are pre-announced because by working with outside companies they assume there will be leaks (ChromeOS, Android).
Internally people get told "Please don't leak unannounced projects. A leak could cause your co-workers to have to launch an unfinished or unpolished project ahead of time, reducing the impact of months or years of their time".
The problem with Apple is that they work with a lot of outside agents, all of whom can leak without thinking of the personal consequences to friends, just financial/legal ones (which can be avoided). Their own engineers have a pretty good track record of keeping quiet about 'important' things.
Yeah, we had problems in Google with these too; we have large networks of machines that used to use multiple different NTP servers (for resilience). Turns out not all NTP servers implemented leap seconds the same way, and many cluster based applications get upset when they aren't synchronized to within 100ms.
Now, we run a dry-run of a fake leap-second with all software a few weeks before the leap-second failover. It's the only way to be 100% sure that applications changed since the last leap second won't have problems. Though, most unittest frameworks now have the ability to implement second skewing, since the suffering caused by the 2005 leap second.
The main problem is that the POSIX description of how to do a leap second is retarded; you basically go from 00:00:00 to 00:00:59, some apps also get upset when they see the same time twice.
Um...not as such. There are many parts of Google Apps that are better off running on a cluster than a 'server farm'.
Especially searching. If you need to search through your 100GB of indexed documents, you want to be able to transparently break up that search query over multiple machines.
Obviously, a cluster is just a fabric running on top of a server farm. But having a fabric that spots slow/loaded machines and repairs or unloads them, and gives a rich API to things like GFS/Bigtable/Chubby/Mapreduce etc. means a big difference to Apps developers.
Upgrades are planned to have no impact on availability. Check the article; they have no 'unplanned' outages. I don't think Google have ever suffered a user-visible data loss; when they talk about downtime, it means user connections can't get to the backends for some reason. If your network connection dies, Google Docs retries till you get back in coverage. Hopefully that'll happen sometime.
If you plan to take out a service with an upgrade, you are doing it wrong. Seriously.
Google certainly don't use the stock linux kernel.
They have a team of kernel programmers that rework the VM slightly, every time another team comes across a problem; usually the index guys. That said, I think the 2.4 kernel needed an order of magnitude more patches than the 2.6 one.
Given that most cluster machines run only a few processes at once, OS overhead isn't that significant.
Since then, they've double their engineering staff every year, and rolled out a lot of new purpose-build datacenters. I've seen public guesstimates as high as 400,000 machines. With SATA, it's not unreasonable to imagine 5 or 10 disks in every machine. That's a lot of storage. Of course, having one 'SAN' makes no sense; you need it distributed, so latency to each user is as low as possible.
Bigtable in particular is interesting; a database designed to handle many petabytes of data in a low-latency environment.
Sigh. I really really wish we could be completely candid about all this stuff. Maybe over time...
Thankfully, archeologists will have loads to work on pre-20thC stuff, so it's not quite the end for Archeology. However, in 10000 years, after we no longer use digital computers, we'll need very advanced archeologists to recover data from defunct machines, buried hard drives etc.
Not all civilisations last forever. Even global ones.
The 2.6 VM is an order of magnitude better, as far as I can see, based on some stats I've gathered on a few thousand hardworking 2.4 machines and a few thousand 2.6 machines. Only reason to go 2.4 is...less stuff to change.
Google and Apple have wildly different management styles.
Google is run for the engineers. The engineering managers are interviewed by engineers. If they don't know as much about engineering as the guys they manage, they don't get hired. So, the end result is engineers running at 100% efficiency, giving everything they have to every crazy project they come up with. The public then choose which products they like - its not like you have to splash out 1000 bucks to try a new google product out. And most hires come from friends of engineers. Much more important than a HR dept. that knows the difference between someone with a unix certification and the ability to create.
Apple is run by media people. Really good media people & industrial designers who are in those great social networks across the industry. Engineers produce stuff that the media people want. They may only be running at 70% efficiency, but they are told exactly what to do, and because the managers are usually right, the engineers are happy to go alone with that that.
I'd a chat with some Google engineers that used to work for Microsoft. One guy was proud of the fact that not one line of code or one bugfix he'd put in over seven years ever made it into a product that shipped. What do you think his morale & efficiency was running at
Everytime you buy something made in China, you help the chinese government. You'd hold google to a higher standard ? It's not like they are selling them censorship equipment or anything, just giving people the option of uncensored search, or censored search, and letting them know the censored stuff is censored...which is more than anyone else does.
They give a lot more than lipservice to open source. 77 million to Mozilla last year, through advertising. GFS and MapReduce is their core business. You want them to give that away ?
In Ireland, householders go crazy if they move into a new house to find that power lines haven't been buried yet. Of course, it's likely that all the talk of EM from power lines causing cancer is complete bullshit, but its certainly unsightly, and doesn't cost that much to bury them, if you have telecoms cables that need to be buried anyway.
It just needs a little planning and cooperation; something local governments force on Irish utility companies.
A guy I know in Sunnyvale, CA bought a bottling machine for his meadery.
The day he got it, he spent all night pricking around with the interface to it, you know, where you set speeds, cork or screwcap, how much to put in each bottle etc.
And, then just for the hell of it, he realised you could reprogram it. He added a LCARS type interface, for the main screen. Funny as hell. It really confused the engineer who turned up the next day to show him how to turn it on...
I'd rather risk losing someone good than be lumped with someone weak who got through a light the recruitment process.
I don't know what the ratio to hire-nohire is, but some engineers do a hundred 45 min interviews a year, on top of their normal jobs. So it's not just difficult on the applicants; but there is a really good reason for it.
Some people say "So, just hire bad people and fire them later". Google isn't like a normal company. Stuff is done so differently that you can't afford to spend months bringing them up to speed, then find out that they can't hack it.
As has been mentioned before, comparing the images of Tianamen from the google.cn and google.com sites is unhelpful. In China, Tiananmen is a major tourist attraction. In the rest of the world, it's the epitome of ruthless suppression of democratic protest.
All you are doing is highlighting the difference between two cultures, and explaining *why* censorship is done.
Where sensible Health & Safety laws fine employers who don't do enough to make sure people don't work over 48 hours a week - and that includes on-call time. In our company, we all do 40 hour weeks. If you do more than eight hours of on call, you *must* get a day off the following week. In our French office, there is someone who is responsible for kicking people out by 18:00, in case they work too long by accident, or get carried away.
Don't listen to big business who say it harms the economy, to have people working like slaves. It does more damage to people and family to let them work those hours. It's amazing that it's a given that there are US laws to stop you poisoning yourself with cocaine, but not against killing yourself with work stress.
Given Google don't make any money from Picasa, the Linux client is a loss-leader. So, it makes sense to get the first Linux version in the easiest way possible. And that is Wine. That's what it was written for. In the free software world, there is always someone who will say "I want that for free!", and "Now that I have it for free, I want it better". If you do that in a restaurant, they'll sprinkle crumbled turd on your food. On the internet, all they can do is ignore you.
Check out the code contributions - there are lots of bugs found & fixed by the Google guys that are working on this. It's not like they are saying "Go run on wine, we don't care", it's "Go run on wine, and we've given you the most help we can".
A tool like Picasa, which was written from the ground up for Windows, is not a candidate for a "Linux Port". It would need a "Linux re-write". Maybe a future version could be built using tools to help with platform independance...but Google have much bigger things to worry about.
I want a disk equivalent of top - something that'll tell me what processes are kicking the shit out of the disks, and by how much.
If Linux could do that - it's more a VM thing than a filesystem - I'd stick with ext3 for years to come.
Who needs a filesystem in a database when you have a database that lives on your filesystem (updatedb). Get that updating in realtime, with more things (like permissions, access times etc.) and a lot of the work is done.
Be careful. Don't use these for anything important. In more than one application, I've seen the C3 cause crashes and wierd behaviour on Linux.
Via never release errata, so Linux guys can't work around them *when* they happen.
My experience was on an embedded linux box with a Via motherboard and C3. Wierd crashes very infrequently. Things like SSH sessions crashing when you hit "tab". Login again, same thing happens. Reboot, problem disappeares. SNMPd crashing in the same place (ucd_build_snmp_packet or some such) until a reboot happened. GDB traces of the core dumps showed the instruction pointer jump halfway through multi-byte instructions.
We replaced the chip with an earlier model, no problems. Celeron, no problems. Different motherboard, same problems. YMMV.
When contacted Via said there were no bugs in the C3 chips, that it must be a software problem.
Slashdot Mirror
What about Shingle recording ? There are prototype drives already built that can near 100TB on a single device. It has limitations, of course, like 2GB+ block sizes, but it'll get interesting. Give it 3-5 years (as usual, for mythical tech that might ship someday).
So, this is bullshit. You can keep secrets as long as the people involved think secrecy is warranted.
Google have an astonishing track record of not leaking projects to the press. They've worked on some incredible stuff, and the vast majority don't get leaked at all, or get leaked accidentally. Huge numbers of internal/infrastructure projects never get told about outside the company. Sure, some projects are pre-announced because by working with outside companies they assume there will be leaks (ChromeOS, Android).
Internally people get told "Please don't leak unannounced projects. A leak could cause your co-workers to have to launch an unfinished or unpolished project ahead of time, reducing the impact of months or years of their time".
The problem with Apple is that they work with a lot of outside agents, all of whom can leak without thinking of the personal consequences to friends, just financial/legal ones (which can be avoided). Their own engineers have a pretty good track record of keeping quiet about 'important' things.
"That's nearly twice what Google spend on a datacenter" - yeah. They'll likely stock it with Xserves, rather than decent priced hardware!
Yeah, we had problems in Google with these too; we have large networks of machines that used to use multiple different NTP servers (for resilience). Turns out not all NTP servers implemented leap seconds the same way, and many cluster based applications get upset when they aren't synchronized to within 100ms.
Now, we run a dry-run of a fake leap-second with all software a few weeks before the leap-second failover. It's the only way to be 100% sure that applications changed since the last leap second won't have problems. Though, most unittest frameworks now have the ability to implement second skewing, since the suffering caused by the 2005 leap second.
The main problem is that the POSIX description of how to do a leap second is retarded; you basically go from 00:00:00 to 00:00:59, some apps also get upset when they see the same time twice.
John
Um...not as such. There are many parts of Google Apps that are better off running on a cluster than a 'server farm'.
Especially searching. If you need to search through your 100GB of indexed documents, you want to be able to transparently break up that search query over multiple machines.
Obviously, a cluster is just a fabric running on top of a server farm. But having a fabric that spots slow/loaded machines and repairs or unloads them, and gives a rich API to things like GFS/Bigtable/Chubby/Mapreduce etc. means a big difference to Apps developers.
Upgrades are planned to have no impact on availability. Check the article; they have no 'unplanned' outages. I don't think Google have ever suffered a user-visible data loss; when they talk about downtime, it means user connections can't get to the backends for some reason. If your network connection dies, Google Docs retries till you get back in coverage. Hopefully that'll happen sometime.
If you plan to take out a service with an upgrade, you are doing it wrong. Seriously.
Google certainly don't use the stock linux kernel.
They have a team of kernel programmers that rework the VM slightly, every time another team comes across a problem; usually the index guys. That said, I think the 2.4 kernel needed an order of magnitude more patches than the 2.6 one.
Given that most cluster machines run only a few processes at once, OS overhead isn't that significant.
John
Sun have a corporate culture of not suing people. But they also countersue, and in this case, HARD.
John
Google don't release stats on the size of their clusters, obviously. But this is what they were doing four years ago;
http://labs.google.com/papers/gfs.html
http://labs.google.com/papers/bigtable.html
Since then, they've double their engineering staff every year, and rolled out a lot of new purpose-build datacenters. I've seen public guesstimates as high as 400,000 machines. With SATA, it's not unreasonable to imagine 5 or 10 disks in every machine. That's a lot of storage. Of course, having one 'SAN' makes no sense; you need it distributed, so latency to each user is as low as possible.
Bigtable in particular is interesting; a database designed to handle many petabytes of data in a low-latency environment.
Sigh. I really really wish we could be completely candid about all this stuff. Maybe over time...
John
Thankfully, archeologists will have loads to work on pre-20thC stuff, so it's not quite the end for Archeology. However, in 10000 years, after we no longer use digital computers, we'll need very advanced archeologists to recover data from defunct machines, buried hard drives etc.
Not all civilisations last forever. Even global ones.
John
2.4 is *NOT* more stable than 2.6.
The 2.6 VM is an order of magnitude better, as far as I can see, based on some stats I've gathered on a few thousand hardworking 2.4 machines and a few thousand 2.6 machines. Only reason to go 2.4 is...less stuff to change.
john
Most countries don't have a fair use clause in their national copyright laws. Belgium is one of them.
John
Google and Apple have wildly different management styles.
Google is run for the engineers. The engineering managers are interviewed by engineers. If they don't know as much about engineering as the guys they manage, they don't get hired. So, the end result is engineers running at 100% efficiency, giving everything they have to every crazy project they come up with. The public then choose which products they like - its not like you have to splash out 1000 bucks to try a new google product out. And most hires come from friends of engineers. Much more important than a HR dept. that knows the difference between someone with a unix certification and the ability to create.
Apple is run by media people. Really good media people & industrial designers who are in those great social networks across the industry. Engineers produce stuff that the media people want. They may only be running at 70% efficiency, but they are told exactly what to do, and because the managers are usually right, the engineers are happy to go alone with that that.
I'd a chat with some Google engineers that used to work for Microsoft. One guy was proud of the fact that not one line of code or one bugfix he'd put in over seven years ever made it into a product that shipped. What do you think his morale & efficiency was running at
John
Everytime you buy something made in China, you help the chinese government. You'd hold google to a higher standard ? It's not like they are selling them censorship equipment or anything, just giving people the option of uncensored search, or censored search, and letting them know the censored stuff is censored...which is more than anyone else does.
They give a lot more than lipservice to open source. 77 million to Mozilla last year, through advertising. GFS and MapReduce is their core business. You want them to give that away ?
John
I'm pretty sure he was on a rocket. I saw it on telly.
John
Wow. That's a bit mad. Isn't that dangerous ?
In Ireland, householders go crazy if they move into a new house to find that power lines haven't been buried yet. Of course, it's likely that all the talk of EM from power lines causing cancer is complete bullshit, but its certainly unsightly, and doesn't cost that much to bury them, if you have telecoms cables that need to be buried anyway.
It just needs a little planning and cooperation; something local governments force on Irish utility companies.
John
Duh. If you are a hiker, in LA, you are really lost. QED.
john
A guy I know in Sunnyvale, CA bought a bottling machine for his meadery.
The day he got it, he spent all night pricking around with the interface to it, you know, where you set speeds, cork or screwcap, how much to put in each bottle etc.
And, then just for the hell of it, he realised you could reprogram it. He added a LCARS type interface, for the main screen. Funny as hell. It really confused the engineer who turned up the next day to show him how to turn it on...
john
I'd rather risk losing someone good than be lumped with someone weak who got through a light the recruitment process.
I don't know what the ratio to hire-nohire is, but some engineers do a hundred 45 min interviews a year, on top of their normal jobs. So it's not just difficult on the applicants; but there is a really good reason for it.
Some people say "So, just hire bad people and fire them later". Google isn't like a normal company. Stuff is done so differently that you can't afford to spend months bringing them up to speed, then find out that they can't hack it.
John
As has been mentioned before, comparing the images of Tianamen from the google.cn and google.com sites is unhelpful. In China, Tiananmen is a major tourist attraction. In the rest of the world, it's the epitome of ruthless suppression of democratic protest.
All you are doing is highlighting the difference between two cultures, and explaining *why* censorship is done.
John
Where sensible Health & Safety laws fine employers who don't do enough to make sure people don't work over 48 hours a week - and that includes on-call time. In our company, we all do 40 hour weeks. If you do more than eight hours of on call, you *must* get a day off the following week. In our French office, there is someone who is responsible for kicking people out by 18:00, in case they work too long by accident, or get carried away.
Don't listen to big business who say it harms the economy, to have people working like slaves. It does more damage to people and family to let them work those hours. It's amazing that it's a given that there are US laws to stop you poisoning yourself with cocaine, but not against killing yourself with work stress.
john
Given Google don't make any money from Picasa, the Linux client is a loss-leader. So, it makes sense to get the first Linux version in the easiest way possible. And that is Wine. That's what it was written for. In the free software world, there is always someone who will say "I want that for free!", and "Now that I have it for free, I want it better". If you do that in a restaurant, they'll sprinkle crumbled turd on your food. On the internet, all they can do is ignore you.
Check out the code contributions - there are lots of bugs found & fixed by the Google guys that are working on this. It's not like they are saying "Go run on wine, we don't care", it's "Go run on wine, and we've given you the most help we can".
A tool like Picasa, which was written from the ground up for Windows, is not a candidate for a "Linux Port". It would need a "Linux re-write". Maybe a future version could be built using tools to help with platform independance...but Google have much bigger things to worry about.
John
Eh, it's CNET that were using "private" information for their own ends. Google just noticed that it was there.
John
I want a disk equivalent of top - something that'll tell me what processes are kicking the shit out of the disks, and by how much.
If Linux could do that - it's more a VM thing than a filesystem - I'd stick with ext3 for years to come.
Who needs a filesystem in a database when you have a database that lives on your filesystem (updatedb). Get that updating in realtime, with more things (like permissions, access times etc.) and a lot of the work is done.
john
Be careful. Don't use these for anything important. In more than one application, I've seen the C3 cause crashes and wierd behaviour on Linux.
Via never release errata, so Linux guys can't work around them *when* they happen.
My experience was on an embedded linux box with a Via motherboard and C3. Wierd crashes very infrequently. Things like SSH sessions crashing when you hit "tab". Login again, same thing happens. Reboot, problem disappeares. SNMPd crashing in the same place (ucd_build_snmp_packet or some such) until a reboot happened. GDB traces of the core dumps showed the instruction pointer jump halfway through multi-byte instructions.
We replaced the chip with an earlier model, no problems. Celeron, no problems. Different motherboard, same problems. YMMV.
When contacted Via said there were no bugs in the C3 chips, that it must be a software problem.
John