Slashdot Mirror

← Back to Stories (view on slashdot.org)

'Cultlike' Devotion: Apple Once Refused To Join Open Compute Project, So Their Entire Networking Team Quit (businessinsider.com)

Posted by BeauHD on from the cult-like-following dept.

mattydread23 writes: Great story about the Open Compute Project from Business Insider's Julie Bort here, including this fun tidbit: "'OCP has a cultlike following,' one person with knowledge of the situation told Business Insider. 'The whole industry, internet companies, vendors, and enterprises are monitoring OCP.' OCP aims to do for computer hardware what the Linux operating system did for software: make it 'open source' so anyone can take the designs for free and modify them, with contract manufacturers standing by to build them. In its six years, OCP has grown into a global entity, with board members from Facebook, Goldman Sachs, Intel, and Microsoft. In fact, there's a well-known story among OCP insiders that demonstrates this cultlike phenom. It involves Apple's networking team. This team was responsible for building a network at Apple that was so reliable, it never goes down. Not rarely -- never. Building a 100% reliable network to meet Apple's exacting standards was no easy task. So, instead of going it alone under Apple's secrecy, the Apple networking team wanted to participate in the revolution, contributing and receiving help. But when the Apple team asked to join OCP, Apple said 'no.' 'The whole team quit the same week,' this person told us."

43 of 239 comments (clear)

  1. odd--- by Anonymous Coward · · Score: 5, Funny

    the cult-like Apple doesn't like competing cults?

    1. Re:odd--- by Austerity+Empowers · · Score: 5, Interesting

      the cult-like Apple doesn't like competing cults?

      More like engineers are move devoted to their technology than to whomever happens to employ them at any given time. Particularly large, overbearing corporations that saddle them with a lot of rules and marketing idiots.

      This is more a story about how technical people work than it is about Apple.

    2. Re:odd--- by haruchai · · Score: 4, Interesting

      Don't forget to mention.
      Who work in an area where they can easily find a job elsewhere.
      I bet these guys had jobs ligned up before the "bravely" quit Apple.
      Rarely a whole department will quit at the same time. If the job really sucks you will see a migration where people quit over the course of months. Because normally before you quit you need an other job.
      Having Apple not join open compute sounds more liike the first intent to look elsewhere. Then the reason to quit.
      I expect they all just got picked up by some companies they found out they were all quitting at the same time so they used that as the reason for their exit interview.

      What they did was found a new company, SnapRoute - http://www.snaproute.com/our-s...

      --
      Pain is merely failure leaving the body
    3. Re:odd--- by fuzzyfuzzyfungus · · Score: 2

      It's also a story about technical people who have options. If Apple's standards for their network were so exacting and impressive, it is pretty unlikely that they had anyone just clinging to the job because they didn't have much hope of finding another one.

      If you are already considered good enough with the existing tech that unemployment isn't a serious concern; and your current employer is specifically denying you the opportunity to be part of the cool new tech, why would that inspire you to stay with them?

      You can get real hotshots, if the project is interesting and/or the money is good(or the stock options are risky but have the possibility of being really, really, good); and you can usually find people to work with a given system, no matter how legacy, weird, or unpleasant, if the money is good enough; and you can also get people who are unambitious and pretty easy to keep happy; but getting all of those simultaneously is much, much, less likely, if possible at all.

      I don't doubt that Apple was able to hire a new networking team; they can certainly afford it; but telling people "No, it is going to be your job to maintain this legacy system and we aren't going to touch the cool new thing" is not exactly a motivational speech.

  2. Never Down by speedplane · · Score: 5, Insightful

    This team was responsible for building a network at Apple that was so reliable it would never down. Not rarely — never.

    Leave it to business insider to make ludicrous claims about network availability. If Apple's network had 99.99% uptime, and it would cost ten billion dollars to add another 9 to it, I'm pretty sure they'd rather pocket that money than spend it on more redundant switches/routers.

    --
    Fast Federal Court and I.T.C. updates
    1. Re:Never Down by cheater512 · · Score: 5, Insightful

      Full redundancy still has outages, even significant ones.

      In my experience the more layers of redundancies, the more edge cases you need to catch.

    2. Re:Never Down by AK+Marc · · Score: 5, Insightful

      Full redundancy has piles of single points of failure. I've seen a BGP flap take out a Fortune 100 company with N+1 redundancy that cost billions. Redundancy increases complexity, and there's always a point where the "redundancy enabling" technology becomes a single point of failure.

      --
      Learn to love Alaska
    3. Re:Never Down by David_Hart · · Score: 2

      Basic networking. What's your redundancy? HSRP? What happens when someone spoofs your VIP/virtual MAC? Everything is down. I've seen large offices taken down becuase they used 192.168.1.1 as an important device, and someone plugged in a home router under their desk as an AP, causing a conflict that took down a "redundant" network.

      Someone can always take it down. So, go for 802.1x on every port to combat that. Now, if you radius server has an issue, nobody can work. Brilliant. Redundancy and security reduce stability. Go back to networking 101. Even redundant SUPs in a chassis-based system have a single linked management. One wrong command in one of the SUPs and you can take down everything. Redundancy rarely survives user error, and makes it harder to bring it back up.

      Um, you do realize that there are networking technologies to protect the network from practically every scenario that you mentioned? Even your port security example falls flat on it's face because you would have redundant radius servers, etc.

      And no, redundancy doesn't make things harder as long as it's implemented properly (i.e. you have a well documented primary path that's always used unless there is a problem in which case the network switches to a well defined backup path). As for redundant Sups, most large companies that need up-time have redundant hardware/chassis rather than relying on line cards in a single chassis. Removes the human error and hardware failure. Because you are right that the biggest problem is human error.

      The key to any network implementation, same as any other IT service, is that you have experienced network engineers to architect the network, well defined standards, and good support engineers to run it and follow the standards.

    4. Re:Never Down by Gussington · · Score: 5, Insightful

      Um, you do realize that there are networking technologies to protect the network from practically every scenario that you mentioned?

      There are no free lunches. You either have a simple network that could fail and is easy to understand and fix, or a complex network that could also fail but is a nightmare to understand and fix. The other big issue with the latter network, is the size and complexity makes upgrades and patching difficult and expensive, and if people leave it's difficult to bring them up to speed on how it works. The results is that it costs more to maintain than you would lose with a few hours downtime each year. So there is no such thing as a network that never goes down, you either have one that is cost effective or you don't.

    5. Re:Never Down by mwvdlee · · Score: 4, Funny

      Marketing is 100% BS.

      --
      Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
    6. Re:Never Down by AK+Marc · · Score: 4, Insightful

      And no, redundancy doesn't make things harder as long as it's implemented properly

      "properly" by your definition is prohibitively expensive. Almost nobody does it. Realistic redundancy leaves lots of gaps and holes. And in many cases, active/standby is dangerous. HSRP, STP, and many other protocols are active/standby with errors in the standby allowing massive networking failures. And, of course, the protocol to manage that redundancy is a single point of failure. You could abandon HSRP to avoid that single point of failure, and instead have multiple gateways and every endpoint running a dynamic routing protocol but that just moves the single point of failure to whatever routing protocol you pick, and isn't generally done for a variety of very good reasons.

      Nope, the simplest network is often more reliable than the rube goldberg redundant networks I've seen experts like yourself put together. KISS is one of the first rules, and the more you know, the more it matters. KISS. Anything else is expense for the sake of complexity.

      --
      Learn to love Alaska
    7. Re:Never Down by Anonymous Coward · · Score: 3, Interesting

      I worked for a company that had one of their data centres divided into 5 zones, each zone had its own UPS and generator. The was a transfer switch between each zone so that if one zone's backup went down it would be supplied by the adjacent zone which had enough capacity for two zones. After about 5 years there was a major power outage with the local utility. The whole place went down. The investigation found that:

      a. For the last 5 years management had decided not to do annual maintenance since we were "too busy";
      b. The transfer switches were battery powered since they're supposed to work during a power outage;
      c. The batteries are supposed to be replaced during annual maintenance;
      d. The batteries have a useful life of 3-4 years;
      e. If one zone goes down they all do; and
      f. One zone didn't stay up due to a lack of annual maintenance.

      I nearly sent them an energizer bunny but decided it might be a CLM.

  3. Re:Probably a little more to it than that by Narcocide · · Score: 2

    Well, if true, some of them are probably reading this right now. Anybody care to elaborate guys? Also, did you end up managing to make any meaningful contributions to the OCP after that?

  4. Opportunists, not Cultists by slacktide · · Score: 5, Interesting

    They did not quit because they had some sort of cultlike devotion. They quit because they recognized a business opportunity to "get in on the ground floor" and form a startup. "Instead, they founded a startup called SnapRoute, led by Jason Forrester, the former team leader. While Forrester declined to talk to us for this article, SnapRoute's website hints at the story. " Lord knows I've been tempted to leave my big ol' company to pursue similar ventures... Can never convince enough principles to join me. The lure of that pension plan (yes, still have one...) is too strong.

    1. Re:Opportunists, not Cultists by ooloorie · · Score: 2

      Can never convince enough principles to join me.

      Perhaps if didn't call them opportunists, they'd be more interested in joining you?

      The lure of that pension plan (yes, still have one...) is too strong.

      Which is why many people are starting to realize that 401k's are actually a better choice.

  5. Re:Reasonable by DogDude · · Score: 5, Insightful

    We use Chinese made hardware because it works better than US made stuff (Cisco). If the Chinese gov't wants to spy on our business, they can have at it. It's worth it to us to have reliable equipment.

    --
    I don't respond to AC's.
  6. Been there, done that, got cancelled by Anonymous Coward · · Score: 5, Interesting

    I was part of Apple's licensing software team twenty years ago---1996.

    At the time, the Common Hardware Reference Platform (CHRP) was the big item, essentially the same as OCP. Standardize the API at a hardware-abstraction layer, and let everyone build compatible machines. The manufacturer would write the HAL (BIOS) and a variety of operating systems can run on the hardware. (At the time, it was MacOS, OS/2, Novell Netware, and a couple others I've forgotten.)

    My question at the time was "How does Apple make any money when the platform becomes a commodity, and millions of units come into the market on barges from Chinese manufacturers?" Naturally, Apple would cease to sell computers. This was the OS-licensing situation in spades.

    Steve Jobs cancelled Apple's participation in CHRP as well as all OS licensing, knowing that Apple makes most of its money selling the computer. IBM got out of the PC business when Dell and Gateway built PC-compatibles cheaper. Now they're starving as Acer and everyone else builds the platform.

    Why would this be any different today than it was two decades ago?

    1. Re:Been there, done that, got cancelled by Anonymous Coward · · Score: 2, Informative

      Here's a link: https://en.wikipedia.org/wiki/Common_Hardware_Reference_Platform

    2. Re:Been there, done that, got cancelled by ZenShadow · · Score: 2

      This might have been interesting if OCP was in anything even approaching the same space as CHRP used to be.

      It isn't.

      --
      -- sigs cause cancer.
    3. Re:Been there, done that, got cancelled by ooloorie · · Score: 4, Informative

      IBM got out of the PC business when Dell and Gateway built PC-compatibles cheaper. Now they're starving

      No, they are not "starving". They got out of the PC business when it became a low margin business, a good decision.

      Why would this be any different today than it was two decades ago?

      Apple isn't selling server hardware anymore (they already failed in that market once).

      As for Apple's desktops, they should get out of that market entirely because they won't be able to make the margins they are accustomed to in the future.

    4. Re:Been there, done that, got cancelled by ItsJustAPseudonym · · Score: 2

      As for Apple's desktops, they should get out of that market entirely because they won't be able to make the margins they are accustomed to in the future.

      Never underestimate the allure of a white case with rounded corners. To some.

    5. Re:Been there, done that, got cancelled by fuzzyfuzzyfungus · · Score: 2

      This is why most of the people involved with OCP are either companies that buy enormous amounts of server capacity; or suppliers who fear that they'll be discarded entirely if they don't participate.

      CHRP cut directly against Apple's business of selling computers. OCP is gunning for servers and switches. Apple sells neither; but buys a lot of both given how much 'cloud' they are serving up these days.

      Clearly they decided that it wasn't in their interests to participate(whether because they'd rather do it in house; or just because their margins allow them to sit back and adopt anything interesting once it matures); but OCP doesn't directly cannibalize Apple's business in the way CHRP did.

  7. My network has 100% uptime. 2-0 team is undefeated by raymorris · · Score: 3, Insightful

    100% uptime means the network wasn't down in time period you're talking about. My network has 100% uptime this week.
    Maybe last year I had crappy up time, but this year my network doesn't go down (hasn't gone down).

    I enjoy the first few weeks of football season because my team is always undefeated, at least until the end of the first game.

    Actually 100% uptime even over a long period isn't THAT difficult - heteregenous reduncancy pretty much does the trick. That's heterogenous, not homogenous. In other words, you have redundancy for everything, but not by having two of the exact same things. You have a pair of connections (or sets of connections) to the outside world - a metro ethernet connection from one provider, and a direct MPLS connection from another. A Cisco router in the metroE and a Juniper on the MPLS.

      It's extremely unlikely that both providers will go down at the same time. It's extremely unlikely that both the Cisco (or pair of Ciscos) and the pair of Junipers will crap out simultaneously.

  8. Re:My network has 100% uptime. 2-0 team is undefea by plover · · Score: 5, Funny

      It's extremely unlikely that both providers will go down at the same time. It's extremely unlikely that both the Cisco (or pair of Ciscos) and the pair of Junipers will crap out simultaneously.

    ...says the guy who has obviously never run a Juniper. :-)

    --
    John
  9. Apple makes stupid hardware decisions by Anonymous Coward · · Score: 3, Insightful

    If I were an apple hardware engineer I'd quit too. Clearly a company that's selling 3-4 year old technology as a new "top spec" computer doesn't value their hardware wing.

    Don't get me started on the fact that their only laptop with a network adapter is 4+ years old...

    1. Re:Apple makes stupid hardware decisions by swalve · · Score: 5, Funny

      That's just courageous. Someone will invent gigabit wireless.

  10. Re:My network has 100% uptime. 2-0 team is undefea by Anonymous Coward · · Score: 5, Funny

    My network has 100% uptime

    so does mi

  11. Re: After ripping BSD they deserve it by BlackSabbath · · Score: 3, Insightful

    > After ripping BSD

    *facepalm*

  12. Re:Reasonable by Killall+-9+Bash · · Score: 2

    It's more about the firmware.

    Do I buy a Chinese brand, which I'm 99% sure has Chinese government back doors, or do I buy Cisco, which I'm 100% sure has a complete NSA spyware suite installed?

    --
    "Prediction: within 10 years, Windows will be a Linux distribution." Me, 7-6-2016
  13. Re:Reasonable by 93+Escort+Wagon · · Score: 5, Insightful

    Cisco buys from the Chinese. All the best stuff is made in China

    All the worst stuff too.

    --
    #DeleteChrome
  14. Re:Probably a little more to it than that by Gumbercules!! · · Score: 5, Informative

    In TFA it says they formed their own OCP inspired startup, SnapRoute.

  15. YOU HAVE TWENTY SECONDS TO COMPLY by Anonymous Coward · · Score: 3, Funny

    OCP should just stick to robots that shoot people. It's what they do best.

  16. Come on... by somenickname · · Score: 5, Interesting

    I have friends that work for Apple, Google, Oracle, whatever. And I have friends that have quit en-masse from those companies. They almost always quit because they went from a cool startup to a tiny cog in a gigantic machine. These gigantic internet companies consume smaller companies and spit out all the parts they don't like. In many cases, that's most parts.

    This is not an Apple problem, it's an industry and maybe even a societal problem. I don't even think it's possible to get a good job, get an A+ rating for every performance review ever, and expect to stay at that job for 5+ years. After 10 years, you are too expensive to keep around.

    It's a race to the bottom. Throw enough cheap shit at the wall and you'll eventually meet your short term profit goals but, damn, that's a lot of shit to clean off the walls. In fact, you may not be able to clean it all off.

    Greetings, Humans. The machine churns. I'd like to introduce you to the grinding wheel...

    1. Re:Come on... by Required+Snark · · Score: 3, Insightful
      You want to know why software never really gets better? It's because "old timers" are deliberately flushed from the system, so there is no institutional memory. Over time the same mistakes get made over and over again because no one remembers what happened the last time.

      This is not the same as hardware, because in real engineering work there are people who have a long time track record. (The phrase Software Engineering is an oxymoron.) Unfortunately the same short sighted behavior is starting to invade some engineering disciplines, so they will end up producing crap as well.

      --
      Why is Snark Required?
    2. Re:Come on... by LinuxIsGarbage · · Score: 2

      I agree on software footprint / resources.

      The other thing I wonder about is continual refactoring of UI.
      -CLI
      -Desktop GUI
      -Touch GUI

      Those are three distinct usage modes that the UI has to be refactored for an app to go between them. I'm not talking about that.

      But every time I update apps on my phone, usually they change the UI for no apparent reason other than trying to mimic the latest Android release "lets make them flat ugly colours this time!"

      Chrome changes UI at random too "Lets make the 'hamburger menu' three dots now instead of three lines!"

      Firefox: Self explanatory.

      Apps, programs, and websites keep feeling the need to redesign their UI, usually without any real improvement to user experience.

  17. Re: After ripping BSD they deserve it by Anonymous Coward · · Score: 5, Informative

    http://opensource.apple.com

    Also, the one of the best things to ever happen to Linux and the open source OSes was Apple taking ownership of CUPS and making it usable.

  18. They lost me at Goldman Sachs by plopez · · Score: 4, Funny

    If Goldman Sachs is involved I am very suspicious. They are never to be trusted.

    --
    putting the 'B' in LGBTQ+
  19. Re: After ripping BSD they deserve it by narcc · · Score: 2

    Credit where credit is due.

    It still sucks though. Just not as much.

    --
    Required reading for internet skeptics
  20. Re:My network has 100% uptime. 2-0 team is undefea by I'm+New+Around+Here · · Score: 2, Funny

    My network has 100% uptime

    so does mi

    Ahh, Slashdot humor. It never fails to serve.

    --
    If you think I voted for Trump because of this post, you're wrong. I voted for Dr. Jill Stein of the Green Party. Again.
  21. Re:After ripping BSD they deserve it by pmontra · · Score: 5, Informative

    The BSD license explicitly allows ripping, it's the whole point of it. If you publish source using BSD or MIT or similar licenses you should expect and like to be ripped off. Let's say you're very altruistic or you have some plan for profiting by it. If you don't like that, go for GPL.

  22. They violated Apple's first commandment by Opportunist · · Score: 4, Funny

    Thou shalt not have any cults next to me!

    --
    We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
  23. Re:You can't protect against everything. by Anonymous Coward · · Score: 5, Informative

    A true Carrington Event level disaster will fry most IC parts, and has a high chance of setting back the entire human civilization by a century or so

    Not even close. Geomagnetic storms, including the original Carrignton event, involve very slow changing magnet fields (10s of minutes to hours) over very large areas. The source of any damage (on Earth at least) comes from induced voltages, which depends on the rate change of the magnetic field and the area of the circuit of interest. You can work out the numbers, and find that the effect it will have on something small like a cell phone would be less than walking past a fridge magnet. Even a house sized circuit would struggle to produce significant effects, as a change of a couple microtesla of field strength over 10 minutes (on the fast side) would only induce more than a microvolt of voltage if you had multiple loops of unpaired wire around your house. The area within any paired wire is much, much smaller.

    The only place such events can cause issues, due to the very small the rate change of the magnetic field, is by having lots of area. This is where power systems and old fashion communication systems are involved, because they can involve networks over very larger areas and can involve return paths through ground which is susceptible to ground currents in large systems. Modern communication based on fiber or twisted pair conductors would see no direct effect, and issues would just come down to what the power systems do. Even with the power systems, it comes down to having DC breakers installed in the right place, something already demonstrated to protect equipment just fine in past storms (a bigger storm wouldn't change that).

    So no, such an even has nothing to do with destroying ICs or sending human civilization into some pre-electronic age. The only long term concern is what would happen to large power systems where corners have been cut and a potential mess for satellites, which is certainly capable of causing massive economic damage without becoming a prepper fantasy. Otherwise, there would just be a short term power and satellite communication interruption.

  24. Re:After ripping BSD they deserve it by DuckDodgers · · Score: 3, Insightful

    You're not being fair and you know it. Most of the public, through no fault of their own, is not educated in the value of software freedom. So they take walled gardens and digital rights management as a given. Now consider the difference between an iPhone and a Macbook versus a Samsung Galaxy S-something and a high end Dell laptop.

    First, Apple does have an edge in aesthetics in the judgment of most people. If that didn't matter, we Linux enthusiasts would be merrily running FVWM and Blackbox.
    The iPhone is likely to get software updates and security updates from Apple much longer than the Android device. Software updates for the Macbook might only be for four or five years, while Windows 13 will probably run on the Dell. The new Apple operating systems are cheap, too.
    And Apple support might charge through the nose, but it's fast and efficient. If you have to call Dell support, it's probably less painful to just light yourself on fire and be done with it.

    Android and Windows own most of their respective consumer markets because the great majority of smart phone and laptop shoppers can't budget the iPhone and a $1000 machine. But for people who can afford high end devices, Apple is not a waste of money only pursued by fashion victims and phonies.