Slashdot Mirror


'A Lot of Hoped-for Automation Was Counterproductive', Remembers Elon Musk (bloomberg.com)

Thursday Elon Musk gave a surprisingly candid interview about Tesla's massive push to increase production of Model 3 sedans to 5,000 a week. An anonymous reader quotes Musk's remarks to Bloomberg: I spent almost the entire time in the factory the final week, and yeah, it was essentially three months with a tiny break of like one day that I wasn't there. I was wearing the same clothes for five days. Yeah, it was really intense. And everybody else was really intense, too... I think we had to prove that we could make 5,000 cars in a week -- 5,000 Model 3s and at the same time make 2,000 S and X's, so essentially show that we could make 7,000 cars. We had to prove ourselves. The number of people who thought we would actually make it is very tiny, like vanishingly small. There was suddenly the credibility of the company, my credibility, you know, the credibility of the whole team. It was like, "Can you actually do this or not?"

There were a lot of issues that we had to address in order to do it. You know, we had to create the new general assembly line in basically less than a month -- to create it and get to an excess of a 1,000-cars-a-week rate in like four weeks... A lot of the hoped-for automation was counterproductive. It's not like we knew it would be bad, because why would we buy a ticket to hell...? A whole bunch of the robots are turned off, and it was reverted to a manual station because the robots kept faulting out. When the robot faults out -- like the vision system can't figure out how to put the object in -- then you've got to reset the system. You've got to manually seat the components. It stops the whole production line while you sort out why the robot faults out.

When the interviewer asks why that happens, Musk replies, "Because we were huge idiots and didn't know what we were doing. That's why."

37 of 208 comments (clear)

  1. best way to do it by phantomfive · · Score: 4, Insightful

    For a large automation project like this, it's better to start with something you know works. That is, a human assembly line (or a mix like all factories have these days). Then instead of redesigning the whole thing from scratch, replace one humn component at a time. Then you have minimal risk. (That may notbe the best way to do it, but it is a way that works)

    --
    "First they came for the slanderers and i said nothing."
    1. Re:best way to do it by Anonymous Coward · · Score: 5, Insightful

      Nothing particularly wrong with automation. I deal with it all the time, but my production is predictable, has accumulation to deal for errors and gives me a lot of balance. I find automation that starts to deal with vision systems where things completely fall apart. I've yet to find a "good" vision system, they're all pretty primitive, even for quality control, but if you allow for errors, you can develop things to mitigate the inherit problems with it.

      I don't know what Tesla's automation has or what particular situation they have, but I'm sure they have good people trying to think of new ways to automate things. But I do run into a lot of designers trying to eliminate any type of accumulation system (Because it takes up a lot of space) and then utterly fail because of low tolerances in the system. Human error translates into computer systems multiplied by a huge factor.

    2. Re:best way to do it by Mr+D+from+63 · · Score: 2, Insightful

      For a large automation project like this, it's better to start with something you know works.....

      But I've been told that because Musk could land a rocket booster, he was absolutely going to be able revolutionize the car assembly line. It perfect logic, right?

      The problem for Musk isn't that he's had to change his approach, the problem is that he now requires more assembly lines and labor than he planned, therefore production costs will remain higher than planned.

    3. Re:best way to do it by SWPadnos · · Score: 5, Interesting

      From TFS;

      That seems to me a lack of fault tolerance in the overall design approach. A major oversight if so. An entire line should not shut down due to one fault.

      Well, yes and no. Assembly lines are tuned so that you produce the correct quantity of each item in the same amount of time it takes to complete the "full product". It's called Takt time. There are just over 10,000 minutes in a week, so the production goal of 5,000 cars per week requires a new Model 3 to roll off the line every 2 minutes.

      Let's say they're building the battery packs as part of the assembly line, and it takes 60 minutes to build each pack. Just for grins, let's assume that it takes 10 minutes to install that battery pack. This implies that you need 6x as many battery assembly stations (or sub-assembly lines) to make batteries as you need battery installation stations.

      If you have a robot installing batteries, and that robot fails, then you want to stop making batteries before you fill up all the available space with battery packs you can't use (because you can't install them). Similarly, you no longer have chassis with batteries to move further down the line, so anything after the battery installation has to stop, at least for that line.

      There is slack in the process - it might be 59:30 to build the packs or 9:18 to install them, and there would likely be some parts storage as well, so the line could whether an employee going to the bathroom or something. It couldn't whether a continuous failure or a permanent change in process time (ie, it may take a human 13 minutes to install the pack and inspect their work).

      Most assembly lines work this way. The more slack you have, the less efficient your line is (because you have all this extra time ...). The less slack you have, the less you can tolerate a fault.

      I wonder how much $$$ was lost on that automation equipment now collecting dust.

      I'd bet most of the lost money is whatever was spent on programming the units, and the lost opportunity cost for both the hardware and its configuration. The robots themselves are likely still usable, but they'll need to refine how they're used in order to recoup that investment.

      --
      - The Sigless Wonder
    4. Re: best way to do it by UnknowingFool · · Score: 2

      The problem is that automation works really well when the processes are established or the variance to an established process is minor. For example, automating the creation of Lego blocks for Lego is probably pretty easy. Automating a new sky blue variant of an existing block is even easier. A competitor automating a brand new line of toys that compete with Lego blocks is vastly more difficult.

      --
      Well, there's spam egg sausage and spam, that's not got much spam in it.
    5. Re: best way to do it by UnknowingFool · · Score: 3

      Also most automation is designed to handle a small number of faults because if a well tuned process produces only a small number. For large number of faults, it is hard to deal with automation or not. For example the famous Lucille Ball skit where she and her friend are working on a chocolate candy assembly line. At first she and her friend can deal with the mounting issues but as they start piling she resorts to eating the chocolate.

      --
      Well, there's spam egg sausage and spam, that's not got much spam in it.
    6. Re:best way to do it by ElizabethGreene · · Score: 4, Interesting

      This slack time is a serious consideration as you get toward the tail of a production line. A mile-long production line is a hugely complex dependency chain. Each of the hundreds of individual steps might have 99% uptime, but since each depends on the next the real uptime is 50%.

      Case in point: my father worked in the packaging/shipping area of an auto glass manufacturer. When the line had a problem he could go an entire shift or even two (with overtime!) with nothing to do. When things were running right, he could barely keep up.

      The former was much more common than the latter.

  2. Re:I am God's gift to you rotten bastards... apk by gravewax · · Score: 5, Insightful

    I don't like censorship, but seriously isn't it about time Slashdot took some measures to actively block this spamming cunt!

  3. Re:Elon Musk is like the facebook generation by Calydor · · Score: 4, Interesting

    Would you prefer a hundred 'erm' and 'uhh' while he figures out how to phrase the next few words so it's both accurate and relatively unlikely to be taken out of context?

    I have no idea why American interviews (and subtitles, I've noticed) don't do a bit of cleanup before posting but absolutely HAVE to be completely verbatim.

    --
    -=This sig has nothing to do with my comment. Move along now=-
  4. So Musk Admits... by Anonymous Coward · · Score: 3, Insightful

    I think only Musk and his fanboys believed that they were smarter than the countless production engineers in 120 year old trillion dollar auto industry.

    1. Re:So Musk Admits... by DeBaas · · Score: 5, Insightful

      Well I would choose the 'let's try something ambitious and if it doesn't work just admit and stop doing it' over the trillion dollar corporate method of 'very small, committee stamped and approved, steps towards innovation' any day.
      But admittedly, I am a fanboy...

      --
      ---
    2. Re: So Musk Admits... by Qbertino · · Score: 4, Insightful

      They are smarter. Also smart enough to admit when they've gone down a wrong path and need to retract.

      --
      We suffer more in our imagination than in reality. - Seneca
    3. Re:So Musk Admits... by Anonymous Coward · · Score: 3, Insightful

      I think only Musk and his fanboys believed that they were smarter than the countless production engineers in 120 year old trillion dollar auto industry.

      His willingness to question industry doctrines is the reason why he is rich and successful.

    4. Re:So Musk Admits... by MrL0G1C · · Score: 4, Insightful

      I'm not a fanboy but I'm still impressed at how well Musk gets this stuff working. They met their target, no-one thought they would, I was pretty sceptical myself.

      --
      Waterfox - a Firefox fork with legacy extension support, security updates and better privacy by default.
    5. Re:So Musk Admits... by bgarcia · · Score: 5, Interesting

      Would that be the same 120 year old trillion dollar auto industry which said it was impossible to design and develop an electric car that people would actually buy?

      --
      I'm a leaf on the wind. Watch how I soar.
    6. Re: So Musk Admits... by jiriw · · Score: 4, Interesting

      Then he still has a 100 years to make up for it, doesn't he?

      I jest. No-one will deny Musk isn't stubbornly trying to partly redo what the big car manufacturers already can do while sleeping. It's the other parts those big boys, also stubbornly, refuse to do that he tries to make a viable business from. Most EVs from other manufacturers are either 'show productions' with limited numbers or have known horrible flaws baked in (especially in battery degradation) from the start. I'd thank Musk for his tries to do it right (not saying he does already, but he's damned well trying) and I admire him for doing it in grand style.

      If I had a drivers license* and enough money to spare for a car, I'd buy a Tesla. Like I bought a Ryzen the moment they were released. I like to put my money in the camp that tries and manages to make at least a decent product over those that sit on their butts and make money while sleeping.

      (*By the way, I don't have a drivers license because I never had the need to - my job is a decent bike ride away - the exercise keeps me healthy, and we have proper public transport where I live.)

    7. Re: So Musk Admits... by Anonymous Coward · · Score: 2, Informative

      happening to cash out of the dot-com bubble at the right time is why he is rich

    8. Re:So Musk Admits... by Anonymous Coward · · Score: 2, Insightful

      Would that be the same 120 year old trillion dollar auto industry which said it was impossible to design and develop an electric car that people would actually buy?

      To be fair, please acknowledge the context: mainly battery, motor, and controller technology available at the time. We now have LiON, very efficient power switching transistors to control the motor, neodymium permanent magnets and switched reluctance motors. An electric car of 20 years ago would have far less power and range, still be expensive, and only a few would buy it. https://en.wikipedia.org/wiki/General_Motors_EV1

      They had to tear those EV1s out of their owner's hands. GM at no point wanted to make an EV. They still don't. Customers absolutely wanted them and probably would have paid a higher price to get them. GM fights tooth and nail at every point to not do it and they are one of the auto makers that's better with EVs. The Volt was only released as a condition of GM getting a bailout in 2008. The big auto makers don't want to make a real EV. They could, but they won't. And they won't change their minds until after its all over. Such is the nature of old bureaucracies. Seems weird that some on /. want to come done on the side of old bureaucracies in a technology fight.

    9. Re:So Musk Admits... by raftpeople · · Score: 4, Insightful

      He thinks he is going to out-Toyota Toyota. If you think Toyota's approach is 'very small, committee stamped and approved, steps towards innovation' then I would encourage you to read up on the history and workings of Toyota's systems, it's pretty impressive. They were smart, from top down, no Dilbert Pointy Haired Boss stuff but rather thoughtful systems designed and tuned over the years.

    10. Re: So Musk Admits... by bgarcia · · Score: 2

      I agree. But that's implied in my statement. People wouldn't buy the cars either because they would cost too much, or wouldn't be able to go far enough on a charge, or couldn't be recharged quickly enough for long trips. My point is that Tesla is in the process of proving that the rest of the auto industry was wrong. If Elon had listened to them, Tesla wouldn't exist.

      --
      I'm a leaf on the wind. Watch how I soar.
    11. Re:So Musk Admits... by bgarcia · · Score: 2

      To be fair, please acknowledge the context: mainly battery, motor, and controller technology available at the time. We now have LiON, very efficient power switching transistors to control the motor, neodymium permanent magnets and switched reluctance motors. An electric car of 20 years ago would have far less power and range, still be expensive, and only a few would buy it

      To be fair? If you're trying to be fair, why are you talking about 20 years ago? Tesla Motors was founded 15 years ago. They produced their first car 10 years ago. Yet it's only been within the past 3 years or so that "old auto" is finally starting to talk seriously about electric car development as something more than "compliance cars".

      --
      I'm a leaf on the wind. Watch how I soar.
    12. Re:So Musk Admits... by jimbo · · Score: 2

      Not only that but he's brutally honest about mistakes and takes responsibility. How often do we see that in any industry?! He's an engineer and communicates openly and honestly like so many engineers I've worked with, as opposed to the career management people/CEOs that was typically full of hot air.

    13. Re:So Musk Admits... by bobby · · Score: 2

      I wasn't attacking you. You've misunderstood my wording- I'm not a wordsmith. Because of the horrible climate here on /. I rarely contribute. People don't seem too resilient or forgiving here. Misunderstandings are replied to with attacks and defensiveness. It's certainly not you- the whole place stinks, so nobody can tell who farted.

      I just meant that things have changed since that statement was made. It seems obvious, to me, that those kinds of statements make the originator look a bit foolish, which is what I think you were trying to say?

    14. Re:So Musk Admits... by Waccoon · · Score: 2

      Outside of the luxury market, that's still largely the case. Even the "mainstream" Model 3 is selling for almost twice its projected price, and Tesla still wants to make more expensive luxury versions so they don't have to sell at a loss.

      Middle-class electric cars are still not a profitable market. Tesla can survive on the wow factor among the BMW/Mercedes crowd, but they won't magically overtake a mature, trillion-dollar industry with 120 years of experience.

  5. Re:Know thyself by jiriw · · Score: 5, Insightful

    You are mixing up long-term success (visionary) with short-term success (being a huge idiot sometimes). But you probably did that on purpose and are just trolling. Well, I like to feed sometimes ;) I have karma to burn.

    If you can't be a huge fucking idiot sometimes you will not accomplish anything in life. It's when we naively make our greatest mistakes, we grow the most as a human being. The point is, learning and not making the same mistake again.

    Do you know anyone that can operate an automatic assembly-line from birth, like it's in their DNA, other than its own digestive tract? No? Indeed, didn't think so. Eating and shitting all over the place comes naturally, as we see often enough here in the comments. The rest we have to learn. Sometimes we can learn from others, but if we want to do something innovative, we have to learn the hard way. That means being huge idiots until you know how to do it right.

    If you think you can do better than Musk, prove it to the world, or forever hold your peace.

  6. Re:Elon Musk is like the facebook generation by JaredOfEuropa · · Score: 5, Informative

    I prefer filler words such as "like" as well as the occasional "erm" from people who actually think about what they have to say, over people who talk and talk and talk with practiced ease but without actually saying anything. Too many interviews are just lips making noise with years of media training behind it, and an interviewer unable to break through that barrier. That's why these interviews with Musk are unusual and refreshing: he actually has something to say, speaks his mind, and isn't as prone to evade questions.

    --
    If construction was anything like programming, an incorrectly fitted lock would bring down the entire building...
  7. Re:Automation does not start in production phase. by Corbets · · Score: 5, Insightful

    Ironically, that character flaw is also what has allowed him to succeed in breaking into markets the world thought he could never enter (upsetting established auto manufacturers with a tech electric startup, reusable rockets stealing market share from the Lockheeds of the world).

    So maybe we ought to write it as “flaw” with the quotes instead.

  8. Re:Know thyself by MrL0G1C · · Score: 3

    Like the fairy-tale where one man runs a company that sends rockets in to space, is racing ahead with a luxury electric car company, produces leading battery storage for vehicles and properties and has some innovative solar products to boot.

    Yeah, what an idiot, an intelligent person would have done far better. I'm not a Musk fan boy, but it's pretty obvious the man is no idiot.

    It's just a shame he can't keep up with google WRT autonomous driving.

    --
    Waterfox - a Firefox fork with legacy extension support, security updates and better privacy by default.
  9. Re:Other manufacturers by AHuxley · · Score: 2

    That would depend on the price range and brand of vehicle? What other brands are making a product range at that quality and price?

    --
    Domestic spying is now "Benign Information Gathering"
  10. Lessons GM learned in the 80s by sphealey · · Score: 2

    = = = A lot of the hoped-for automation was counterproductive. It's not like we knew it would be bad, because why would we buy a ticket to hell...? A whole bunch of the robots are turned off, and it was reverted to a manual station because the robots kept faulting out = = =

    How could Musk have possibly known that would happen? Just because he was operating in a factory he bought from GM, which went through the exact same process in the 1980s and the failures of excess automation in the assembly process were well documented in both the trade press and business press? No One Could Have Anticipated(tm).

    Sometimes the people who have been doing something for 120 years are hidebound. And sometimes they really do know what they are doing.

    [IMHO the stock market has done Telsa a real disservice by bidding up the stock price beyond reasonable levels. At the time Ford needed the cash it received from selling Land Rover and Jaguar, but now it really needs a new luxury division. Tesla would make an excellent division of Ford - but at the stock price that can't happen]

    1. Re:Lessons GM learned in the 80s by Anonymous Coward · · Score: 2, Insightful

      There would be no progress if you just replicated what other companies had done in the past.

      New products made with new processes require you to not listen to the experienced people at some point. It's extremely difficult to tell when you need to listen and when you need to ignore. Tesla made a mistake here, but the mistake was definitely not that they were ignoring some of what GM had done previously, it's that they made a bad choice on which things to ignore and which not to ignore. They had to ignore some of it. GM produced and tanked the EV1. If Musk listened to all of GM's experience, there would be no Tesla at all.

      Also GM's problems with automation 30 years ago might be a good thing to revisit, as technology has changed massively since then. A $5 Raspberry Pi is far more powerful than any computer they had back then. Robotics and automation has massively changed. Thinking it can be done better now is not radical.

      I think it's a good sign here is that the mistake has been recognized and they owned it. It's the companies who think they can't make mistakes and shift the blame externally that cause massive problems in a vain attempt to be right.

  11. Re:Other manufacturers by Big+Nemo+'60 · · Score: 3, Informative

    Many years ago (keep this in mind - things may have changed) I worked at an European company, designing and building production plants for the automotive industry, especially for the body shops.

    A typical production line for European mid-sized cars had a theoretical capacity of fifty (50) car bodies per hour - that would be at 100% efficiency, actual yeld was lower but not much. A typical production facility for non-premium class cars had two production lines running in parallel on two eight-hour shifts per day (third shift for clean-up and maintenance), five days a week. 50*2*8*2*5=8000 car bodies per week at 100% efficiency (as I said, actual yeld was something less than that). Of course, most manufacturers had (still have) more than one facility running.

    Premium / luxury production lines are usually smaller than that, of course. Think Ferrari.

    I hope this is of interest.

    --
    In the long run we are all dead. - John Maynard Keynes (1883 - 1946)
  12. Re:Know thyself by magzteel · · Score: 5, Insightful

    You are mixing up long-term success (visionary) with short-term success (being a huge idiot sometimes). But you probably did that on purpose and are just trolling. Well, I like to feed sometimes ;) I have karma to burn.

    If you can't be a huge fucking idiot sometimes you will not accomplish anything in life. It's when we naively make our greatest mistakes, we grow the most as a human being. The point is, learning and not making the same mistake again.

    Do you know anyone that can operate an automatic assembly-line from birth, like it's in their DNA, other than its own digestive tract? No? Indeed, didn't think so. Eating and shitting all over the place comes naturally, as we see often enough here in the comments. The rest we have to learn. Sometimes we can learn from others, but if we want to do something innovative, we have to learn the hard way. That means being huge idiots until you know how to do it right.

    If you think you can do better than Musk, prove it to the world, or forever hold your peace.

    Well said.
    It's so easy to never fail. Just never try anything and spend all your time criticizing others who do.

  13. Re:Automation does not start in production phase. by theskipper · · Score: 2

    Re: character flaw, calling the diver in the Thailand rescue a pedophile this morning lends credence to the fact that he doesn't listen to his large investors, let alone those related to development of the company's product.

    https://twitter.com/MidwestHed...
    https://twitter.com/TeslaChart...

    A while back Baillie clearly told him to shut up with the unprofessional tweeting and focus on the work. And it seems they're not alone. The other large funds are clearly spooked by the balance sheet, product defects and CEO behavior and have been dumping their common positions (i.e. the least-senior in bankruptcy). Fidelity primarily but even TRowe, the most steadfast, is showing signs they fear the worst: https://twitter.com/WintonCapP...).

    Of course if we have truly have entered an era where you can say anything without repercussion from other stakeholders, and can make your own reality, then none of this matters and the cult mentality will continue. What that says about the future of corporate governance and execution in this country is left as an exercise to the reader.

  14. "Because we were huge idiots..." by gweihir · · Score: 2

    And that is why Musk is successful. He may only be a mediocre engineer, but he is not only able to learn, he is able to be brutally honest with himself and that puts him far ahead of the crowd.

    --
    Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
  15. Hubris by Elfich47 · · Score: 2

    Musk decided to ignore the comments and advice of experienced automotive engineers and industrial designers. There are easily twenty other car manufacturers out there that learned the lessons that Tesla is relearning now. The lessons that Tesla is learning now are the ones all the other car makers have as their daily mantra:

    1. Have an assembly line with the lowest downtime and most consistent production rate available.
    2. Research improvements to the line. Develop a replacement station/process for particular portions of the assembly. Test and debug the new process so it is as reliable and as fast as the previous process. Once the new process is refined and complete - Then change the assembly line.
    3. Retrain the workers once in the new process. Go forward in the new manufacturing process.
    4. Understand the "Pull method" of manufacturing. Toyota implemented it in the 70's and 80's and forced everyone else to get on the band wagon.
    Yes, this sounds boring. But the big boys out there all know that the things that kill profitability is: downtime, reliability, rework, supply chain foul ups, and warranty work. So unless the new process improves all of these things, it won't be changed.

    --
    Architectural plans are like computer source code with a couple of differences: You only compile once.
  16. Re:No Need For The Song, Then? by SunTzuWarmaster · · Score: 2

    Come with me
    And you'll be
    In a world of Tesla Automation

    Take a look
    And you'll see
    Tesla Automation

    We'll begin
    With a spin
    Traveling in
    The world of Musks' creation

    What we'll see
    Will defy
    Explanation

    If you want to view paradise
    Simply look around and view it
    Anything you want to, do it
    Want to change the world?
    There's nothing to it

    There is no
    Life I know
    To compare with pure automation

    Living there
    You'll be free
    If you truly wish to be
    If you want to see Martian lands

    Close your eyes and you will see one
    Want to be a dreamer, be one
    Anytime you please and please save me one