Slashdot Mirror
Facebook VP Slams Intel's, AMD's Chip Performance Claims
narramissic writes "In an interview on stage at GigaOm's Structure conference in San Francisco on Thursday, Jonathan Heiliger, Facebook's VP of technical operations, told Om Malik that the latest generations of server processors from Intel and AMD don't deliver the performance gains that 'they're touting in the press.' 'And we're, literally in real time right now, trying to figure out why that is,' Heiliger said. He also had some harsh words for server makers: 'You guys don't get it,' Heiliger said. 'To build servers for companies like Facebook, and Amazon, and other people who are operating fairly homogeneous applications, the servers have to be cheap, and they have to be super power-efficient.' Heiliger added that Google has done a great job designing and building its own servers for this kind of use."
You guys don't get it
Is it possible to take out a massive life insurance policy on Jonathan Heiliger?
To build servers for companies like Facebook, and Amazon, and other people who are operating fairly homogeneous applications, the servers have to be cheap, and they have to be super power-efficient.
I assure you, despite your misconception that the world revolves around you everyone has those requirements. From the people who build supercomputers right down to the netbook I am typing on while watching Gurren Lagann.
Can we get like a panel of hardware engineers to have a discussion with this guy and can I get some popcorn?
My work here is dung.
To build servers for companies like Facebook, and Amazon, and other people who are operating fairly homogeneous applications, the servers have to be cheap, and they have to be super power-efficient.
Hm, lets see... perhaps because Facebook and Amazon are niche markets? The average server isn't going to even need all the computing horsepower and the power efficiency is simply a drop in the bucket for most companies electrical bills. The average server is going to be much more I/O intensive than CPU intensive unless you do cluster computing or render a lot of stuff. The average server such as a web server or a file server doesn't use that much CPU and usually you are running 1-3 servers, not the hundreds that Facebook or Amazon would run.
And really, why is a VP complaining about this stuff? That he can't either afford custom solutions or spend the money buying more servers?
Taxation is legalized theft, no more, no less.
I have heard from some reliable sources that Facebook and Twitter's backend applications are poorly written.
Are Intel and AMD's claims overblown, sure what hardware manufacter doesn't cherry pick performance claims.
But I don't care what sort of hardware you through at crap code you are always going to get crap performance.
Well, I suppose that if he does not like the offerings from Intel and AMD, they could always go with...
Uh..
Oh.
1) Facebook & Amazon need cheap, power efficient systems
2) Intel and AMD aren't measuring up with processors to power these systems
3) However, Google has systems appropriate for this use (presumably using Intel or AMD processors)
If that's his argument, then it would seem that the real conclusion is that Facebook can't build systems as good as Google's, even though they are using the same processor technology.
It's because your shitty website doesn't have a single line of compiled code. PHP only goes so far.
I agree I think this was writing his own resignation with this crap. The guy is basically telling everyone that he is incapable of finding an acceptable solution for his company and blaming intel and amd because he has committed a great deal of money on something that he didn't plan well enough to know exactly what the long term costs vs performance was. In the very article he says to not be cheap, but in many more words than necessary, probably to try to disguise what he is saying like most politicians, that they were not only too cheap, but made bad decisions on what to be cheap with. Its as if he's already in a public office, hes telling everyone he screwed up, why he screwed up, and trying to make it look like hes teaching everyone lesson to make his mistake to be less of a disaster.
Its the next logical solution... Those T5440 servers with 256 processing threads are MONSTERS in terms of handling simultaneous connections which make them very good web servers, database servers, and file servers, all of which means they are very good for a company who's product is a website.
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
Assuming that a solution was properly engineered, this should not have been a surprise.
Cheap. power efficient, performance. Pick two.
"To those who are overly cautious, everything is impossible. "
This is becoming an annual event for Heiliger, who also complained about server vendors at GigaOm's Structure 08 conference last year. Facebook used to buy a lot of cloud-optimized gear from Rackable/SGI, but no longer appears on the list of their largest customers. Makes you wonder if they're not going to follow Google's lead and build their own servers.
NEWSFLASH! Customer are tightwads.
Performance/Reliability/Price.
Pick any two, Heiliger.
Looks like that to me; he scoped for cheap and cheerful and was bit on the ass when he realised that sometimes you get what you pay for. Like what's the point in having quad-core server CPU without the high-bandwidth buses of server-grade hardware.
In the concurrent DNS/Kaminsky thread, I saw a reference that facebook's DNS TTL is low. A quick investigation reveals that they have a 30 second TTL and are using DNS round-robin for their load balancing.
He's nothing but a blame-shifting cretin.
POKE 36879,8
I think we read different articles. He's not saying he didn't plan well enough, he's saying that Intel and AMD promise that Gen Y processor is 35% faster than Gen X processor, and he's not seeing anywhere near 35% in real world performance. The 35% is a made up number but it doesn't matter what the number is that they claim. He's probably correct. Manufacturers pull this stuff all the time. Look at the recent articles on battery life claims on notebook's. AMD came out and called BS on the whole thing and basically said if you guys don't stop lying through your teeth, the FTC is going to regulate us. From the perspective you are taking, that would mean we have to call AMD incompetent for not understanding how batteries work and not properly selecting them.
Manufacturers ALWAYS overstate claims in computer related products. CRT actual inches vs viewable inches (thank you lcd's for finally being honest... about inches anyway.. brightness and contrast however....) Computer speaker wattage being 1/2 or 1/4 of what's claimed. Power supply efficiency or wattage not measuring up to claims... you name it. He's calling out what he see's to be bogus claims based on a real world experiences in a demanding environment, the type of environment where one is always looking for better performance. I think we should get some more information before declaring him to be the problem as I'm sure he has a whole team of people that are working on this issue.
What I'd like to see from him is some numbers. On this Intel (or AMD) rig, we get so many operations per hour/minute/whatever. On this new Intel (or AMD) rig which they claim is 20% faster than the previous rig, we're only seeing this number of operations per hour which amounts to only a 7% gain, thus we're seeing 13% less than they are claiming. Again, numbers made up for examples sake. I'd also be very interested in what a typical rig of theirs looks like... X Processor, Y Ram, what type of storage system is it connected to, etc. I think such numbers are vital to understanding the issues at hand. We all know that vendors will run the benchmarks that makes their stuff look the best, and that is often not reflective of real world performance. If I was Intel/AMD I'd be chiming in right about now and opening a dialog with Facebook and looking to see what the issues are. Facebook is a big customer with huge name recognition and you want to be able to use them as an example of your solution delivering the promised performance for your marketing. I'm going to assume (I know I know) that they are already working with the server vendor to try and see what's going on here.
They collect a large amount of data on people and mine that for marketing information to turn around and target those same users.
It's the same model as google.
I'm bemused that he implies the problems with his servers are due to Intel and AMD no delivering with their chips, yet at the same time he admires google for how good a job they do in building out their machines.
he must be aware that google uses Intel and AMD chips.
his reasoning just doesn't square.
In a minute there is time For decisions and revisions which a minute will reverse. -T.S. Eliot
How can you be blamed for finding an acceptable solution when there simply isn't one available? He is a software developer, not a hardware one. Not everybody can just go out and design their own servers like Google does. He's saying he's been tripped up by the fact that the server manufacturers aren't delivering on their promises; hardly something he should be blamed for. Your attempts to read more into his comment about "not being cheap" and compare it to the false words of a politician seems like a pretty big stretch.
If you read the entire article, he not only doesn't say that his decisions have led to disasters, but instead says that his infrastructure development decisions have led to very smooth transitions, even when Facebook rolls out big, new features like the customized home page URLs. He is only voicing his disappointment in saying that the servers aren't living up to the hype, and that he is still looking for a better solution.
I will say that his comment to not be cheap seems to be in direct conflict with the rest of his argument, since his criticism over AMD and Intel revolves around the fact that they need to be cheaper. Seems a bit counter-intuitive.
"Give a man fire, and he'll be warm for a day; set a man on fire, and he'll be warm for the rest of his life
Every major server vendor has jumped on the bandwagon of 'look how efficient we are, and 'cheap'. Three years ago, by and large the tier ones wouldn't bother designing systems without forcing even the cheap design to have parts included to facilitate purchase of redundant add-ons (i.e. power distribution cards designed for dual power supplies regardless of one being bought or not). They would always put a high end storage controller on the planar. They would always make their 'entry' platform be burdened with expensive components to make it easier to option it up.
Now, we have tons of 'internet scale', or 'cloud', or whatever buzzword you feel like. They tend to stress energy efficiency, low cost components, with sales and management strategies targeted at thousands of servers (i.e. IBM iDataplex, HP SL6000). Basically, precisely what he prescribes, though probably not as 'cheap' as he wants. The incentive he gives is that the vendors should have zero margin, which is not particularly compelling for companies to work toward. Google's situation works because they brought it in-house and thus have fewer middle-men. Honestly, from all the rumours I hear, it's the logical thing to do when your server consumption is larger than some respectable computer companies' entire production. If he thinks the volume of servers is high enough to pull a google, by all means do it. Otherwise, be prepared for people not jump at the chance to give their designs to him at zero margin.
Of course, if he is calling them out on performance per-watt by avoiding non-x86 solutions, including ARM, that might be a fair criticism. However, I think company forays into 'exotic' architectures have not panned out in the market recently. Sun's niagra, despite all the worthy praise, couldn't attract a mass-market required to subsidize it for those who benefited most from it. Last year, IBM seemed to be saying Cell architecture would light the world on fire, but have been a lot quieter about it now. The message their buisness leaders have probably taken in is that while these things have their target market, that market isn't worth the expense of developing products that are refused by the larger market and focus instead on leveraging commonly accepted building blocks to do as best they can for that niche, even if it means skipping the 'perfect' solution. Sure, IBM still sells plenty of POWER, but I haven't heard that be *particularly* praised on the performance/watt category like I hear a lot for Niagra, Cell, and ARM. And if not for POWER's legacy, it probably would be still born in the market today. The PA-RISC->Itanium decision for HP probably sank their HP-UX product line faster than banking on legacy of PA-RISC installs, and it seems IBM won't make that mistake, but at the same time I don't hear much about *new* POWER customers.
XML is like violence. If it doesn't solve the problem, use more.
You can better identify your bottlenecks by benchmarking. Facebook's scalability is likely not as cpu-bound as predicted, thus the dude's angst on discovering that CPU upgrades weren't a silver bullet.
In your case, you haven't looked past the RAID configuration for the root-cause of your performance issues. Without benchmarking you don't really know if it was an issue with: the filesystem, the block size, stripe size, or a caching tunable.
Systems architecture isn't as easy as PC builders would have you believe.
POKE 36879,8
Not necessarily, no.
It's all about how CPU limited the workload is.
You might be running a program that's CPU limited on one processor, then upgrade the processor and discover that it's suddenly discover that instead of being CPU-bound, now you're memory-bound. Or I/O bound. Or whatever.
Point is, just because you've hit the wall in terms of CPU doesn't mean you'll get a 50% improvement with a 50% increase in CPU ... you'll only get that if all the rest of the server's systems have 50% overhead to spare. And in most cases they don't. One of them will hit the performance wall before you return to being CPU-bound with the shiny new processor.
There are exceptions to this -- renderfarms, for instance, or some distributed HPC stuff -- where you really can reasonably expect to get 50% more performance out of 50% more CPU, but they're exceptions not the rule.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
Since when do we listen to manufacturer's claims? You take the new hardware, stress test it with your custom software, record results, plan servers accordingly. How hard is it really to commission a server design that meets your needs and then QA some prototypes?
Good-bye
Google's core business is intelligence.
Facebooks core business is stupidity.
Boffoonery - downloadable Comedy Benefit for Bletchley Park
I think we read different articles. He's not saying he didn't plan well enough, he's saying that Intel and AMD promise that Gen Y processor is 35% faster than Gen X processor, and he's not seeing anywhere near 35% in real world performance.
If the application was purely CPU bound, and Y wasn't giving me 35% more than X, I'd complain.
However, if it's a complex system like almost everything else, why would they expect their application to get 35% faster when there's probably 6 or 8 critical subsystems that could all be bottlenecks as well?
More data, damnit!
Uhhh, correct me if I'm wrong. I've been looking at after market bolt on parts for my car. The headers claim increase fuel mileage, the spark plugs, the air filter, the tires, as does a turbocharger. The glass pack mufflers, and the resonator. Oh yeah, the aerodynamic rims, the hood, and spoiler. Don't forget the carbon fiber body panels. Taken all together, those increased MPG's add up to about 150 MPG. You're saying I may not see that much improvement on my 1968 Chevy Malibu? It's just hype? Man - you just saved me about $5,000!!!
"Windows is like the faint smell of piss in a subway: it's there, and there's nothing you can do about it." - Charlie Br
...I'm supposed to care about the comments of the guy who wrote Facebook ?
Hah, hah, hah, hah, hah !At least google needed to actually engineer their solution, but Facebook, come on ! The next time I need to write a PHP script for displaying photos and text, I'll hire my 13 year old daughter.
This isn't just about the CPU, it's about overall system performance.
Despite improvements in CPU performance, memory and IO performance is lagging behind.
A modern SATA drive delivers about 90MB/sec ( peak sequential read ).
Some RAID controllers can do about 600-800MB/sec ( peak sequential read ).
An average AM2 ( K10 core 65nm ) gets about 34,849MB/sec L1, 12,169MB/sec L2, 6371MB/sec L3, 2,741MB/sec DDR2-800 5-5-5-12.
Obviously Opterons scale a lot better since they each have an onboard memory controller and additional HT links which greatly increases bandwidth as you add more CPUs. However adding more cores on the same die which have to share a single memory controller can cause starvation.
Another major issue is software parallelization, writing parallel code is still a difficult problem. If your software doesn't parallelize well it doesn't matter if you have 8, 16 or even 32cores on a single die.
If you had an equal number of CPU cores and memory controllers you could achieve much better performance, however your relatively very slow storage subsystems would still be a major bottleneck.
I once did a large project in which I took a large, slow site in PHP (it was pretty complecated, it was a CRM with a lot of custom business logic) and rewrote all the core functionality from PHP to C / C++, and made it a "module" of PHP. The rewriting was mostly simple translation -- litterally removing all dollar signs, adding some types, and attempting to compile, and just fixing the compile errors until it would build. Then going back through it with a fine-tooth comb to track down all the memory leaks.
The speed increase from doing that is pretty surprising. Simple loops that do a bit of math or something speed up by 100 times, and a loop that creates and destroys an object within the loop will be 100,000 times faster. This is without actually trying to write fast C/C++ code, and not create and delete the same thing over and over in a loop -- just pure dumb translation of the code.
At that point, the web site guys can keep tweaking and changing the web page in PHP just like before; but they load that module in the php.ini and then they have a basic library of stuff, like login_user() or get_user_balance() and etc, that are really fast and do all the heavy lifting.
I would be surprised if Facebook has not already done this. How to do it is well documented in several books, and there are lots of PHP modules written in C/C++ to look at for examples.
I suspect that Facebook's VP is right that AMD and Intel exaggerate their claims, but is also generally true that most computer programs are more IO bound that you expect. This is not a reason to avoid something like I describe above; once you have the more complete control of programming in C, IO issues may be easier to find and address.
He also mentions that the servers offered by Dell and others aren't very power efficient or practicle for him, and he mentions Google designing their own servers. Nothing google did was really rocket science, from what we know, and Facebook probably doesn't have to go as far as they did to get a reasonable benefit. It's not that hard to set up motherboards to run without a case, booting off the network with no harddrive attached.
No, he just found that RAID controllers suck. Which they do, universally, all the time. The only ones that actually perform decently are the ones in external SAN boxes, and inside they are typically servers with software RAID...
Finally! A year of moderation! Ready for 2019?
One of the fun toys Intel has to play with is a complete system simulator, which simulates every single component in a computer for early testing. This lets them vary parameters that aren't feasible yet while they're working on their design goals. A few years ago they did a test; what happens to the system performance if you make the CPU infinitely fast? They adjusted the simulator so that every CPU operation took zero simulated time and ran their benchmark suite. It ran twice as fast (in simulated time) as it was before.
A CPU-bound workload can quickly become a RAM-speed bound or a disk-speed bound workload if you make the CPU faster but don't upgrade anything else.
I am TheRaven on Soylent News