Slashdot Mirror
Open Compute Project Comes Under Fire
judgecorp writes: The Open Compute Project, the Facebook-backed effort to create low-cost open source hardware for data centers has come under fire for a slack testing regime. The criticism was first aired at The Register where an anonymous test engineer described the project's testing as a "complete and total joke." The founding director of the project, Cole Crawford has penned an open letter in reply. The issue seems to be that the testing for standard highly-reliable hardware used by telcos and the like is very thorough and expensive. Some want the OCP to use more rigorous testing to replicate that level of reliability. Crawford argues that web-scale data centers are designed to cope with hardware failures, and "Tier 1" reliability would be a waste of effort.
Probably Cisco trolling against a movement that's going to put them out of business.
Sooner the better, I say.
Some people just have to get a burr up their ass [arse] about everything.
Wait, Register is still up? Do they still say 'boffin' every paragraph? I couldn't bear to click through.
I'll agree to a point that "web-scale data centers are designed to cope with hardware failures" bit, but when you are standing there with an internal customer shitting a brick because their product or project that is supposed to be mission critical is on COTS hardware and has no redundancies built in you tend to wonder why something of this nature was done. Specially with networking gear, I've seen a lot of companies use cheap hardware, throw a lot of it at the problem, and when something large scale happens (usually once a year with this stuff) everyone starts asking questions and you shrugging isn't a good enough answer even though you didn't buy it, didn't test it, and were forced to use it, maintain it, and take responsibility for it.
Web-scale? Way to be tone-deaf there Mr. Crawford.
Or maybe the ridicule heaped on users of that particular term is something indulged only by the neckbeard wannabes that haunt Slashdot. In which case, carry on!
Maw! Fire up the karma burner!
Webscale.
You don't need expensive hardware to run datacenters. You need cheap commodity hardware with smart software on top. Just ask Google or Facebook.
But testing well is really, really hard. And expensive, especially for data center scenarios. If you haven't put it in an oven and observed the effects, it's not tested for telco data centers.
"web-scale data centers are designed to cope with hardware failures". So.... it's OK if you use my motherboard design and they randomly fail, because you should just make up for that in software or hardware redundancy? Um, no.
Just because I can hook a shark from a boat, I do no offer to wrestle it in the water.
FTF Crawford.
I don't know if it's a good idea or not(probably depends on who you are, and I'm sure that there will be some people who chose incorrectly); but is it really a surprise that OCP would be doing their testing on the cheap 'n cheerful side of things?
It was my understanding that their premise, from the beginning, was that existing hardware vendors were excessively focused on adding costly, thermally demanding, and often proprietary, features at the hardware level that were unnecessary if you were willing to compensate for their absence in your software design.
There is obviously some level of reliability below which no compensation at the software level is possible(if you can't run the algorithm for detecting errors because it keeps glitching out, it's probably not going to work); but the impression they always conveyed was that many of the more sophisticated reliability mechanisms are really features aimed at people who are substantially less able to cope with failure; and are therefore willing to pay substantially more for hardware that can invisibly paper over a variety of moderately serious failures and allow the software on top to run without incident; rather than buying lots of cheap hardware that has a risk of going down in a screaming heap.
So long as nobody gets any stupid optimistic ideas, I don't really see the issue. Sure, if Facebook were about sending men to mars, they should seriously consider having three CPUs running in lockstep and voting on all operations and so on; but this project is about delivering as many ad impressions per dollar as possible; no reason to get worked up over the occasional glitch.
it would be alot better than water
I'm gonna side with OCP on this one. It is far more economical to deal with reliability via redundancy than it is via expensive parts. This is why we use RAID rather than speccing our drives to last 10 years minimum. All the big players in the datacenter market have put thousands of hours each into designing systems tolerant of missing parts.
The downside is that writing custom stacks tolerant of missing pieces is fucking hard and a huge up-front investment for a company. Most off-the-shelf software does not have that level of redundancy and fault tolerance baked in already. This means that for many small to medium sized deployments it's cheaper to buy a really expensive fault tolerant rack of servers and run your off-the-shelf software on it than it is to buy into OCP with inexpensive hardware that's more open to failure, because your software is NOT open to failure.
Different strokes for different folks and all. Use the right tool for the job. And OCP was made by companies with massive data farms to fit their needs... and their needs are probably not your needs.
"I will trust Google to 'do no evil' until the founders no longer run it." Hello Alphabet.
Yep. This thread is full of people pooh-poohing this idea and meanwhile it's the strategy used by the most successful corporations on the internet. Welcome to Slashdot!
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Sounds like Hooli XYZ! Where's Nelson Big Head Bighetti?
Pick two...
It all boils down to what you want, but of the three things we all say we want, you get only two...
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
...Crawford argues that web-scale data centers are designed to cope with hardware failures...
By that logic, the telco data centers are not designed to cope with hardware failures?
.
Of course, I really don't care if facebook has downtime due to hardware reliability issues. facebook is more a waste of time than anything else.
It's called Open Compute Project for a reason. The willing may pursue different goals with Open Telecom Project. Or read Jim Gray's technical report number 85.7 for the Tandem Computers.
Note that their datacenter disciplines are not actually proven to be the best, but boy do they think so. You ask their datacenter guys and you'd think it's because of *them* that the business plan works.
Notably, the mindset centers around *ZERO* warranty and no testing at all. This encourages some nasty behavior in the vendors. They may be able to tolerate it, but they are paying a price in terms of how much they have to oversize and replace.
There's a middle ground between ludicrous resiliency inside a single system and 'lose hundreds of servers / racks a day and you would never notice' (except for the replacement parts, spare capacity, power and cooling sucked down by zombie servers before being addressed, and so on and so forth).
it doesn't matter how many redundant servers you have, if they are all going to fail in the same way
I suspect open compute project welcomes additional testing resources for the benefit of everyone... as long as it doesn't involve an oppressive amount of process that simply serve to slow down progress.
But... Web scale IS different, so I can't blame the main sponsors for not prioritizing what isn't as important to them. Once you accept that ALL hardware fails, and that you can either pay more for more reliable hardware, or you can develop better software architecture to handle failures, you look at things differently. Spend your money once on good software engineering, instead of over and over on every server.
Who would believe any "testing" certification these guys came up with anyway? Cheap shit suppliers come and go. Suckers that insist on buying absolute shit will be with us always.
For *some* datacentre tasks you can use cheap, commodity hardware. For others, you need expensive, certified, bullet-proof hardware.
This sig left unintentionally blank.
I think the point is that so far it is only used by "the most successful corporations on the internet". In fact, you can probably count the number of organisations in the entire world that qualify on the fingers of one hand, though it will take a few more fingers to count how much money they have invested to reach this point.
Unfortunately, as lovely and friendly as all the Software Defined X advances seem with their mantra of openness, almost no-one is actually building a "web-scale data centre" with a 24/7 staff dedicated to just swapping out broken hardware and effectively unlimited resources to devote to designing hardware architectures and building control software that can cope with frequent failures without losing significant amounts of real money. For normal organisations, even those with heavy IT requirements and 12 figure market caps, running your critical infrastructure on hardware that does have a serious level of testing and consequent robustness may still be advantageous.
(Full disclosure: I sometimes work for clients in the networking industry, though whether an industry shift towards things like OCP would benefit or harm them would be open to debate so I think I'm still reasonably neutral here.)
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
While I was working at Amazon we were told to expect hardware failures and to build our software around it. I have a couple of friends doing hardware testing for AWS and all of their hardware is of extremely low quality and has major visable issues such as bowing, flimsy connectors, and little to no hardware redundancy in the hardware itself(no dual power supplies or hot swappable anything). This really isn't a surprise at all, its just where the industry is going.
Unfortunately, as lovely and friendly as all the Software Defined X advances seem with their mantra of openness, almost no-one is actually building a "web-scale data centre" with a 24/7 staff dedicated to just swapping out broken hardware and effectively unlimited resources to devote to designing hardware architectures and building control software that can cope with frequent failures without losing significant amounts of real money.
I think that's because most customers don't want that, partly because they don't understand how they would use it yet — but also because there is the fundamental problem of paying a middleman. If you are depending on someone to build the cloud for you, you're going to have to accept that they're going to want to get paid for their trouble. And nobody likes to write checks, they like to cash 'em.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
MongoDB is Web-scale.
Isn't this the point of the cloud: don't buy/build/maintain your own, rent from us and save because we do it cheaper and better than you ever could on your own?
I think by the time you reach a scale where you have 24/7/365.24 staffing adequate to handle the failures as they happen, you can take advantage of the higher failure rate / lower cost equipment. You don't need to be Google scale to do this.
Test engineer says... big companies need to hire more test engineers.
Are we surprised?
So many chances to go metric and add other improvements, all deliberately missed. Now we have a "standard" that sits right between two telco standards, with no obvious indication why it would be better than either: It's just more of the same. Thus the thing is an elaborate shtick to be speshul and troll the manufacturers into getting to do facebook's bidding.
The problem is when managers want to replicate this with cheap commodity developers and cheap commodity IT support on top of unreliable hardware infrastructure instead of the expensive, and rare, high-end personnel and internal resources that Google and Facebook have.
Since most companies won't be able to hire the top 1% of those people, might it be more worthwhile to buy more reliable and expensive hardware?
You need cheap commodity hardware with smart software on top. Just ask Google or Facebook.
The software used by the rest of us (e.g. MySQL) isn't that smart, and it's very expensive to get software that is that smart --- requires hundreds of thousands of ops engineer developer man hours, potentially to build that software system.
There are open source products that can be that smart, with enough deployment work. Developing smart custom applications is a bear.
It may very well be cheaper in many cases for smaller scale applications to spend the extra money on some more reliable hardware instead of massive $$$ on extra development.
I guess you could say then definitively now that OpenCompute is not for everyone.... it's especially not for IaaS hosting providers, if the components are more prone to failures that the service provider will be held responsible for.
instead of the expensive, and rare, high-end personnel and internal resources that Google and Facebook have.
Then they are destined to fail, if they are unwilling to invest in suitably skilled personnel AND high enough quality development for the chosen architecture to implement their intended plan.
might it be more worthwhile to buy more reliable and expensive hardware?
Paying up to keep the more qualified personnel on staff can have other benefits. I think the competition for good people is much less than you imply.... if you are willing to pay up. Many times the top 1% of the technical talent does not wind up with significantly more pay than the next 30% down.
Developers in the top 70% can still build highly-resilient applications, also, and if you pay more than the typical market rate for them, you can likely pick many of them up.
It's those "C" level folks that are so hard to avoid, and the fact is, No interview screening procedures the average person will come up with are likely to reliably distinguish and eliminate those.
Well, I have a few issues with the cloud hype, starting with the scarcity of evidence to support claims about cloud services being cheaper and/or more secure and/or more reliable than doing things yourself. Every major cloud provider has had serious downtime, and there is only so much you can attribute to being more visible at greater scale or to users not configuring HA tools properly. Far too many on-line services also run into significant security/privacy problems. And cost-wise going with the cloud rather than your own systems tends to be favourable at certain levels (other things being equal) but it can be outrageously expensive in other cases.
These myths aren't really the point here anyway. The point in this case is that no matter how fast your recovery time may be, whatever was happening on your hardware at the time it failed is lost, and in some cases you simply can't make that transparent to your users. Not everything in the world of programming is a distributed map-reduce where losing a hardware node means you just redistribute the 0.0001% of the job it was doing to another and no-one notices. Not everything in the world of networking can tolerate a multi-second failover process without an observable blip in connectivity. As for redundant/HA storage, the CAP theorem called and asked to speak with you about your database, but I think you were on with physics at the time so I just took a message.
It's not just about whether the wastage due to more frequent failures works out cheaper economically than paying a premium for better hardware. It's also about how much downtime you (or your customers) are willing to tolerate and what proportion of overall system time is spent just recovering from failures. If you've ever had the joy of watching the (N+1)-th drive fail in your RAID with N-way redundancy while it's still rebuilding from replacing the earlier failures, you'll know what I mean.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
I've never had an N+1 drive fail in a RAID setup. What I have had happen is the power supply to the whole array fail... then we can talk about redundant power supplies, but, really, the data needs to be mirrored offsite at a place where a serious (fire / flood / riot / meteor strike / whatever) event doesn't take down all copies of the data / service. This was sort of the founding principle of ARPANET, anyway.
Economics varies, people negotiate bad contracts all the time that lead to higher costs of whatever approach they have taken. Not surprising that something with the hype of "the cloud" can get people to sign bad deals. Also not surprising that some "bulletproof" hardware is excessively premiumed compared to the advantages it conveys.
In a "rich infrastructure" 100 cheap cars beats a single tank. In a desert with bad to non-existent roads and a costly supply chain, you'll want the tank. Assuming these "cloud" data centers have sufficient infrastructure and scale, they should be able to do it better and cheaper. Of course, it's always possible to mismanage anything, and this goes double for security concerns.
If you want / need control and you can't afford to point to a sub-contractor not living up to their contract terms when something goes wrong, then do it in-house. If in-house is a single site, or the multiple sites you do have can't afford around the clock technical maintenance presence, then, yeah, go for the "good stuff" and let the expensive machines help you in your (ultimately futile) pursuit of perfection. If your organization is numbered in the 10,000s or larger, and top management takes IT seriously, they should seriously be employing fault tolerant methodologies - whether you use cheap crap for equipment or not.
The N+1 failure days will happen, and multi-second fail-overs response times sound perfectly acceptable to me, unless you are in high-speed trading, in which case - a pox on you and your servers and may you lose Billions in your next equipment snafu. But, those days when you have the unacceptable failure (Fukushima Daiichi?) are the days when you step back and improve the design and methods. Generally speaking, there are bigger gains to be had with redundancy and distribution than there are with "more bulletproof" hardware slotted back into the same system design that just bit you.
I don't think I'd ever go to the cloud because it's cheaper or more secure or more reliable. The main benefit that I see is flexibility.
If your loads are stable and known in advance, it's likely cheaper to buy hardware and staff people to take care of it. On the other hand if loads spike wildly from one day to the next the cloud makes perfect sense. Need a thousand cores of compute power right this second? Amazon/Google/Rackspace/HP would be happy to rent it to you.
I worked on a telecom switch that ran processing on cards that had two CPUs in lockstep. If the output of the two ever differed the card was taken out of service and its last transaction was rolled back. Memory contents were stored in at least three places at any given time. The dataplane was inductively coupled to avoid the possibility of DC current damaging things.
We replaced it with commodity hardware and smarter software. It wasn't *quite* as reliable, but it was a whole lot cheaper and the speeds ramped up much faster.
If you'd been watching the attack maps, you'd know that:
(1) It's China
(2) It's likely at the government level
If you'd been watching current events, you know that:
(3) China's economy has been crashing, going on three weeks now
(4) They're really unhappy about people taking money out of, and shorting, Chinese stocks, adding to the crash
(5) They've lost $3.25T in market cap since June 12th
(6) That's just over 20% of their Gross National Product
So it's likely they are attacking our financial markets over that.
See also:
"Key things to know about China's market meltdown"
http://www.cnn.com/2015/07/08/...
Ever notice just exactly who pushes renting everything? That would be people who own stuff. Cloud computing is like privatization in government. It will never get past the fact that somebody wants to make a profit, and so it will never be as cheap as everyone says.
Look--if somebody buys cheap hardware, I can too. Control and monitoring stuff is getting better all the time. I can get that too. I don't need 'web scale' for everything and, unlike Google and Facebook, it damned well does matter if my data is in a consistent state everywhere all the time. They simply have different business needs for their own operations. It's not good or bad, it just is.
Cloud computing isn't bad either. It's an excellent choice in some situations and an expensive and poor choice in others. It is a tool, and unless your business is very small, it had best not be your only tool.
"has come under fire for a slack testing regime"
The correct word is "regimen", not "regime".
The arguments for less pedigreed hardware are
The application is ok with an occasional server failure.
A fancy compute server from a tier 1 vendor comes from the same ODM as an OCP server.
A compute server will be obsolete in 3 years, but a telco platform is expected to last for 10-20.
Both design and manufacturing make reliability. The OCP designs may actually get more thought and testing than their tier 1 cousins.
(For example, the OCP power plan with distributed backup appears an improvement over the telco 48volt centralized battery plant.)
Some possible problems with this plan are:
Part of the testing is for safety, this still seems necessary.
Replacing h/w every 3 years isn't green.
It seems to me that, if you populate a data center with junk, then failures might be more than occasional.
Another problem is that some failures might just be flakies which is another thing for the application to deal with to prevent bad results.
In anything new there will be other issues we don't know about yet.
The OCP hardware needs to be good enough to avoid these problems, but no better.
Even with zero testing in the design and manufacturing phases, it should be obvious in the deployment if the equipment meets this criteria.
At best, this test engineer is saying that waiting till deployment is wastefull (or dangerous?).
If there is a problem with OCP, it may be that writing general purpose applications to run on flakey hardware may be a harder problem than just building stable hardware. Which says that market forces may force the OCP ODM's to make pretty good stuff.
Note that their datacenter disciplines are not actually proven to be the best, but boy do they think so.
They are proven to be the best for their specific type of operations. I'm quite sure that their SOPs won't work for the banking or healthcare industry for example.
If Facebook goes down, a bunch of 30 year olds are going to complain (teens use other social media these days, and grandparents won't care and try again later). If the Sutter Health (norcal hospital chain) network/DC goes down, people's health will be affected.
Different operations and requirements, require different budgets and ways of working. For hyperscalers as FB and Google, RAID makes sense. Where RAID in this case is Redundant Amount of Inexpensive Devices.
I'm not a complete idiot... Some parts are missing.