'A Lot of Hoped-for Automation Was Counterproductive', Remembers Elon Musk (bloomberg.com)
Thursday Elon Musk gave a surprisingly candid interview about Tesla's massive push to increase production of Model 3 sedans to 5,000 a week. An anonymous reader quotes Musk's remarks to Bloomberg:
I spent almost the entire time in the factory the final week, and yeah, it was essentially three months with a tiny break of like one day that I wasn't there. I was wearing the same clothes for five days. Yeah, it was really intense. And everybody else was really intense, too... I think we had to prove that we could make 5,000 cars in a week -- 5,000 Model 3s and at the same time make 2,000 S and X's, so essentially show that we could make 7,000 cars. We had to prove ourselves. The number of people who thought we would actually make it is very tiny, like vanishingly small. There was suddenly the credibility of the company, my credibility, you know, the credibility of the whole team. It was like, "Can you actually do this or not?"
There were a lot of issues that we had to address in order to do it. You know, we had to create the new general assembly line in basically less than a month -- to create it and get to an excess of a 1,000-cars-a-week rate in like four weeks... A lot of the hoped-for automation was counterproductive. It's not like we knew it would be bad, because why would we buy a ticket to hell...? A whole bunch of the robots are turned off, and it was reverted to a manual station because the robots kept faulting out. When the robot faults out -- like the vision system can't figure out how to put the object in -- then you've got to reset the system. You've got to manually seat the components. It stops the whole production line while you sort out why the robot faults out.
When the interviewer asks why that happens, Musk replies, "Because we were huge idiots and didn't know what we were doing. That's why."
There were a lot of issues that we had to address in order to do it. You know, we had to create the new general assembly line in basically less than a month -- to create it and get to an excess of a 1,000-cars-a-week rate in like four weeks... A lot of the hoped-for automation was counterproductive. It's not like we knew it would be bad, because why would we buy a ticket to hell...? A whole bunch of the robots are turned off, and it was reverted to a manual station because the robots kept faulting out. When the robot faults out -- like the vision system can't figure out how to put the object in -- then you've got to reset the system. You've got to manually seat the components. It stops the whole production line while you sort out why the robot faults out.
When the interviewer asks why that happens, Musk replies, "Because we were huge idiots and didn't know what we were doing. That's why."
For a large automation project like this, it's better to start with something you know works. That is, a human assembly line (or a mix like all factories have these days). Then instead of redesigning the whole thing from scratch, replace one humn component at a time. Then you have minimal risk. (That may notbe the best way to do it, but it is a way that works)
"First they came for the slanderers and i said nothing."
I don't like censorship, but seriously isn't it about time Slashdot took some measures to actively block this spamming cunt!
My eyes see the white space in the one line shown and just pass on to the next. Best censorship system available, even better than APK's porn-hating hosts list.
-=This sig has nothing to do with my comment. Move along now=-
Would you prefer a hundred 'erm' and 'uhh' while he figures out how to phrase the next few words so it's both accurate and relatively unlikely to be taken out of context?
I have no idea why American interviews (and subtitles, I've noticed) don't do a bit of cleanup before posting but absolutely HAVE to be completely verbatim.
-=This sig has nothing to do with my comment. Move along now=-
I think only Musk and his fanboys believed that they were smarter than the countless production engineers in 120 year old trillion dollar auto industry.
You are mixing up long-term success (visionary) with short-term success (being a huge idiot sometimes). But you probably did that on purpose and are just trolling. Well, I like to feed sometimes ;) I have karma to burn.
If you can't be a huge fucking idiot sometimes you will not accomplish anything in life. It's when we naively make our greatest mistakes, we grow the most as a human being. The point is, learning and not making the same mistake again.
Do you know anyone that can operate an automatic assembly-line from birth, like it's in their DNA, other than its own digestive tract? No? Indeed, didn't think so. Eating and shitting all over the place comes naturally, as we see often enough here in the comments. The rest we have to learn. Sometimes we can learn from others, but if we want to do something innovative, we have to learn the hard way. That means being huge idiots until you know how to do it right.
If you think you can do better than Musk, prove it to the world, or forever hold your peace.
At this point it does not even matter anymore whether you are right or not. Your behavior is a huge red flag. Nothing you produce will ever touch my systems for that reason alone.
I guess I am hardly alone in this.
I prefer filler words such as "like" as well as the occasional "erm" from people who actually think about what they have to say, over people who talk and talk and talk with practiced ease but without actually saying anything. Too many interviews are just lips making noise with years of media training behind it, and an interviewer unable to break through that barrier. That's why these interviews with Musk are unusual and refreshing: he actually has something to say, speaks his mind, and isn't as prone to evade questions.
If construction was anything like programming, an incorrectly fitted lock would bring down the entire building...
The most popular EVs sell around 20k units *per year*. Once they've worked through all the hype-driven backorders I see no reason to think Teslas will sell an order of magnitude more, especially if they never manage to get the price down to """only""" $35k.
It starts in the design phase. When you design your "Object"(automobile for example) without certain constraints an automation unit can easily work within - then it needs manual labour because "HI" can adapt easily - or try your luck with CV Systems. However computer 3D-Vision is much more complex and error prone in contrast to for example state of the art 2D-Vision systems where you can really high speed place & sort and do things.
I think it's good that Elon Musk is true about that fact - we didnt know what we did - however I would guess he was told that the design wasn't ready for automation, and most likely that person got demoted to janitor or was fired.
From all information I have on Elon Musk, my picture of him tells me that has the same problem as Trump. He is impervious to counciling from people that actually know better and are more "earthbound".
This character flaw is actually very dangerous for Tesla, because we actually had a near miss of a collapsing Tesla company. I personally would not like to see that because Tesla is a strong driver that has shaken up the automobile combustion gallore.
Bill Clinton, an orator? That was Monica Lewinsky's job.
Come with me, and you'll be... in a world of Tesla Automation!
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
How many vehicles can other auto manufacturers produce in one week? I would have through their volume is more than 13,009 (7,000 + 5,000)
If construction was anything like programming, an incorrectly fitted lock would bring down the entire building...
But it was ... just thinking how few old traditional buildings made with soft materials survived strong central american (where I live) continuous earthquakes. This is why seismic codes are becoming more and more strict, because of the bad experiences and fallen buildings
The real difference is that humans have several thousands of years making buildings and just some decades working with software. And Tesla took a very risky path working many new things that could carry them where they are today. I expect this learning process will help them very much in the future.
Like the fairy-tale where one man runs a company that sends rockets in to space, is racing ahead with a luxury electric car company, produces leading battery storage for vehicles and properties and has some innovative solar products to boot.
Yeah, what an idiot, an intelligent person would have done far better. I'm not a Musk fan boy, but it's pretty obvious the man is no idiot.
It's just a shame he can't keep up with google WRT autonomous driving.
Waterfox - a Firefox fork with legacy extension support, security updates and better privacy by default.
Truth. I once met someone who said they like opening spam and reading it. I wanted to punch them because they are the problem.
There are mistakes and then there are mistakes that shouldn't have happened. Tesla's visionary approach does not preclude them from hiring people who know what they are doing. It's not-invented-here syndrome.
How could Musk have possibly known that would happen? Just because he was operating in a factory he bought from GM, which went through the exact same process in the 1980s and the failures of excess automation in the assembly process were well documented in both the trade press and business press? No One Could Have Anticipated(tm).
Sometimes the people who have been doing something for 120 years are hidebound. And sometimes they really do know what they are doing.
[IMHO the stock market has done Telsa a real disservice by bidding up the stock price beyond reasonable levels. At the time Ford needed the cash it received from selling Land Rover and Jaguar, but now it really needs a new luxury division. Tesla would make an excellent division of Ford - but at the stock price that can't happen]
Take a look at reality
You are mixing up long-term success (visionary) with short-term success (being a huge idiot sometimes). But you probably did that on purpose and are just trolling. Well, I like to feed sometimes ;) I have karma to burn.
If you can't be a huge fucking idiot sometimes you will not accomplish anything in life. It's when we naively make our greatest mistakes, we grow the most as a human being. The point is, learning and not making the same mistake again.
Do you know anyone that can operate an automatic assembly-line from birth, like it's in their DNA, other than its own digestive tract? No? Indeed, didn't think so. Eating and shitting all over the place comes naturally, as we see often enough here in the comments. The rest we have to learn. Sometimes we can learn from others, but if we want to do something innovative, we have to learn the hard way. That means being huge idiots until you know how to do it right.
If you think you can do better than Musk, prove it to the world, or forever hold your peace.
Well said.
It's so easy to never fail. Just never try anything and spend all your time criticizing others who do.
A lot of the hoped-for automation was counterproductive. It's not like we knew it would be bad, because why would we buy a ticket to hell? We don't actually want to go for hell. We just didn't realize it was a ticket to hell. We thought it would be good, but it was not good.
I read that I could think of countless situations on company projects where you could replace "automation" in the above sentence with a third party library, or some super complex internally developed framework that was supposed to cure all ills.
I've read through some of the other comments here about production systems and various backseat driving as to what he should have done. So I'll offer my own - they tried to automate everything all at once on paper before trying anything. They should have probably tried to automate much smaller sections of the process with something they could check out in reality then add that to an assembly line.. like for software, how it really is production to do test fragments of something you are trying to build and run a lot of things through them to see if everything works as you predicted.
No matter how you slice it though enormous systems are just always going to be complex beasts with lots of failure along the way. I''m not really a Musk fan to the depth some are, but one thing I do admire about Musk is that he actually can deliver because he doesn't give up, and puts in the tremendous personal effort it takes to resolve issues. The unlikely successes of SPaceX made me pretty sure he could fix Tesla too, and after reading this I feel pretty sure he has.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
It may seem ironic but I'll give credit to Elon Musk for saying "We were huge idiots" (emphasis mine) where he could have said 'THEY (my employees) were huge idiots'. He's part of the team, he's taking responsibility for failures as quickly as he takes responsibility for successes; this is a trait that in my perception is all to uncommon these last few decades.
As a future M3 owner, it concerns me that Elon would admit that he was a "huge idiot" to rely on automation at a time when his company is selling a product that promises to use similar tech for self-driving, If Tesla can't even get their robot's vision system to recognize parts and where to put them in a controlled environment, what are their chances of getting their cars to recognize objects out on on the road and act accordingly?
5,000 a WEEK, not a MONTH. Let's try that again:
2 min and a shaved second between cars. That's running flat out 24/7. No coffee breaks, no shift change, no stop-the-line-for-an-oopsie. (Easy to see why he needed more than one line.)
A target of 5,000 cars a WEEK this early in the company's history? And they HIT it? I'm FLOORED!
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
This is what happens when people try to do something new. It is called Research and Development, and it happens on -every- project to some extent. Whether it is planned that way or not...
What is different is telling the truth, which has become quite rare in recent decades. Partly because the "Internet" seems to require lies.
If you punish those who say "I was wrong" ot "I don't know", then you will end up buying from the liars. Good luck with that. 8-}
And that is why Musk is successful. He may only be a mediocre engineer, but he is not only able to learn, he is able to be brutally honest with himself and that puts him far ahead of the crowd.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
What have you achieved, exactly?
You know, other than trolling on slashdot.
"Oh my God. This is terrible. This is the end of my Presidency. I'm fucked."; ~ Donald J. Trump
As a future M3 owner, it concerns me that Elon would admit that he was a "huge idiot" to rely on automation
He didn't say they were huge idiots in relying on automation. They still use automation in a number of places in production.
He was saying they were "huge idiots" generally in designing the whole manufacturing process, which they obviously had to re-work quite a bit. Automation is only one small part of that, simple logistics in moving things around another... all had to be re-jiggered it turns out.
Remember it's not like Tesla built the automation systems they scrapped, he was talking about robots going into failure modes they had to have a slow reset from, Tesla's own automation and software in cars seems to be lots more stable.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
What you have just described is non-trivial and would involve designing each assembly station twice: Once for automation and once for manual. And then you have to be able to have both of those stations fit in the same space. And then you have to decide how much extra man power you need on standby in case an automated station breaks and you want to get the manual station running. And then you have to figure when to repair the automated section of the line without endangering someone.
Architectural plans are like computer source code with a couple of differences: You only compile once.
Designed a system that can grow plants without light?
Designed a nearly-universal LED blend that works on over 95% of plants through all stages of life (the purple glow of which you can see in the video mentioned above?)
Developed Aquarium lighting units which can light an entire 55 gallon saltwater reef tank with less than 50 watts of power consumed hourly, and produce consistent growth?
Created prototype lighting systems for Scripps Institute of Oceanography for figuring out *THE* limits of photosynthesis?
What have *YOU* achieved, sonny-boy, besides bloated code that doesn't help humanity in most any way?
*yawn*
Still waiting on Serviscope_minor to wake up to fucking reality and realize that Jessica Price isn't going to fuck him.
Slashdot's moderation system is working very well. He hasn't been censored, but he has been modded down into oblivion.
The best of both worlds. We should not ask for anything more.
That that is is that that that that is not is not.
Musk decided to ignore the comments and advice of experienced automotive engineers and industrial designers. There are easily twenty other car manufacturers out there that learned the lessons that Tesla is relearning now. The lessons that Tesla is learning now are the ones all the other car makers have as their daily mantra:
1. Have an assembly line with the lowest downtime and most consistent production rate available.
2. Research improvements to the line. Develop a replacement station/process for particular portions of the assembly. Test and debug the new process so it is as reliable and as fast as the previous process. Once the new process is refined and complete - Then change the assembly line.
3. Retrain the workers once in the new process. Go forward in the new manufacturing process.
4. Understand the "Pull method" of manufacturing. Toyota implemented it in the 70's and 80's and forced everyone else to get on the band wagon.
Yes, this sounds boring. But the big boys out there all know that the things that kill profitability is: downtime, reliability, rework, supply chain foul ups, and warranty work. So unless the new process improves all of these things, it won't be changed.
Architectural plans are like computer source code with a couple of differences: You only compile once.
As robots become smarter and more versatile, will we start so see shorter, slower production lines in which machines do more things at each step?
Having multiple shorter lines would make logistics much easier. Failures would be localized. And production could be distributed nearer to markets.
The idea of a machine (the production line) pushing out something as complex as a car every couple of minutes boggles the mind. The number of things that could go wrong is huge. It is not surprising that it sometimes stops. It is amazing that it ever goes.
I was shocked to hear the other day that 4chan is not what it once was. Apparently they all moved over to reddit.
If information wants to be free, why does my internet connection cost so much?
...Also probably dishonest or amoral, but that is not strictly necessary.
Musk seems to be a bit better than the average, hell these days the gates foundation is actually pretty good. He, like a few others are not the typical fortune 500 CEO. They made their wealth on that new fangled internet thing long before the typical wall street troll got their teeth stuck in.
I for one welcome our new billionaire overlords. They seem a lot better than the last lot.
If information wants to be free, why does my internet connection cost so much?
"Oh, so you took a few COTS LEDs in different hues and glued them together so people can grow marijuana indoors?"
Guessing you ignore the Scripps Institute part. Okay, you have fun over there in the ball pit since you've shown you're not even up to the reading and thinking level of a 10 year old.
Still waiting on Serviscope_minor to wake up to fucking reality and realize that Jessica Price isn't going to fuck him.