Slashdot Mirror
How Elon Musk Approaches IT At Tesla
onehitwonder writes "In short, they build it themselves. When Tesla Motors needed to improve the back-end software that runs its business, CEO Elon Musk decided not to upgrade the company's SAP system. Instead, he told his CIO, Jay Vijayan, to have the IT organization build a new back-end system, according to The Wall Street Journal. The company's team of 25 software engineers developed the new system in about four months, and it provided the company with speed and agility at a time when it was experiencing costly delivery delays on its all-electric Model S."
S - end
A - nother
P - ayment
Don't bother clicking through - nothing but the same summary.
Did you see where it only took 4 months? I haven't seen an SAP **upgrade** that went that fast, much less a deployment.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
How many SAP installs come in at or below budget? How many are actually completed at all, let alone on-time?
By Rachael King
Reporter
Half Moon Bay, Calif. — Leave it to Elon Musk to buck conventional wisdom. When Tesla Motors Inc.TSLA +7.29%, the Silicon Valley-based automaker he founded, needed to improve the backend software that runs its business, he decided not to upgrade the company’s software from SAP AGSAP.XE 0.00%. Instead, he told CIO Jay Vijayan to build it himself.
“Initially, I was very skeptical,” said Mr. Vijayan, Thursday, at Constellation Research Inc.’s Connected Enterprise Conference in Half Moon Bay, Calif. But, in the end, “Elon was right,” he said, adding that the new system gives his company the speed and agility it needs. His team built it in just four months.
Guus Schoonewille/AFP/Getty Images
A view of a Tesla car on an assembly line
Last year, Tesla was facing delivery delays of the all-electric Model S which it introduced on June 22, 2012. At the same time, Mr. Vijayan’s team of about 25 software engineers was working hard to build a system that could support ramped up production. The improved information technology systems are important for managing high volume production of the Model S, according to company filings. The system went live in July 2012.
Backend software, known as enterprise resource planning software, can make or break a company. SAP has become the world’s largest business software company by building incredibly complex software that can manage customers, suppliers, and the entire lifecycle of a product. SAP says that it is a leading provider of technology for the automobile industry, with nine out of the top 10 companies running SAP applications.
The software is widely used by other large companies as well. Hewlett-Packard Co., for example, uses SAP software to manage the operations needed to sell its printers, servers and PCs. H-P CIO Ramon Baez, also attending the conference, told CIO Journal that it operates at too large a scale to build its own custom enterprise resource planning software.
“You can shoot yourself in the foot if you don’t know what you’re doing,” said Mr. Vijayan. “You need the right team,” he said.
Yet, Mr. Vijayan was in a tough spot. It can take more than a year and millions of dollars to roll out SAP software because of all the integration required. For example, NTT Data is currently undergoing a two-year, $20 million enterprise resource planning consolidation. Tesla didn’t have the time needed to undertake such a project. By creating a custom software project, he was able to get it up and running quickly, partially because it didn’t need integration of disparate applications. Because Mr. Musk made a clear decision, it also helped Mr. Vijayan get immediate cooperation from business leaders.
Yet, there will likely be challenges ahead as Tesla grows. Building and running a lightweight enterprise resource planning system can be done when a company is relatively small but the problem is making it scale, Ben Haines, CIO at Box Inc. told CIO Journal.
“I’m super confident that it’s going to be able to scale very well,” said Mr. Vijayan. “It’s now one of the best systems we have.”
Maintaining it another. One of the hardest things to do is keep up with tax and regulatory changes in your software. You have to be aware of a change before you can implement it.
the major advances in civilization are processes which all but wreck the societies in which they occur - A.N. White
I disagree.
Some of the most successful IT shops I've ever worked in have been 'build' vs. 'buy' shops. They get tremendous cost advantage from having internally-developed tools that exactly meet the needs of their business.
Done right, it works very, very well.
There's usually very good reasons it goes wrong. If they tried re-writing SAP, tehy would have as well. I would guess they had very clear specifications for what they needed, and implemented only that, ideally in a concise maintainable well. Bad specs and scope creep kill most projects like this, yet people rarely seem to learn from past mistakes. They're far more likely skilled than lucky.
The problem is, with solutions like SAP, it's also over budget and under spec.
Atleast with homebrew you have a change to ever reach spec and you don't have to spend the same budget again every next year.
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
I disagree. [...] Done right, it works very, very well.
Yes, the same can be said for any risk-taking behavior. "I haven't worn my seat belt for years and nothing bad has ever happened to me."
#fuckbeta #iamslashdot #dicemustdie
I bet it uses a beowulf cluster architecture. Just make sure some idiot from accounting doesn't spill a bowl of hot grits on the server. Even with a journaling filesystem like ext4, or the superior ReFS, hot grits can petrify most sys admin types.
I wish these Tesla folks the best of luck on this homebrew. In the end though the ROI and TCO will surely be much worse than a suitable off the shelf package from an established vendor like Microsoft or IBM. It may not matter, though, because Tesla will likely flameout and become the next Solyndra
Their system is probably custom-tailored to their business processes. Not only would it not be appropriate for many other businesses and thus have a very small market, but it could also expose some of Tesla's secrets on how they operate, which would then give their competitors (enemies really, because they have no actual competition at the moment) an advantage over them.
They could theoretically invest more money into its development to make it appropriate for mass market consumption. But entering the business software market may not be something they want to do at this very moment. And after pouring additional dollars into the project to make it more generic, it wouldn't be exactly a good use of resources to just open source it either. Nor might it be possible if it makes use of someone else's licensed proprietary technology.
"If a nation expects to be ignorant and free in a state of civilization, it expects what never was and never will be."
Risk vs. reward. What have you gained from not wearing seatbelts other than perhaps a few less wrinkles on your clothes?
Any insufficiently advanced magic is indistinguishable from technology.
We just went through this exact same exercise at the company I work for. When our antiquated, poorly-designed MRP system started causing too many headaches, we carefully counted the cost of moving to something like Salesforce.com or SAP, but ultimately ended up writing out own system from scratch. After running the two systems in parallel for 6 months to ensure the new system had data integrity we were comfortable with, we cut it off. Having just closed our first month "live" on the new system, I must say it's a real breath of fresh air for both the IS staff and the rest of the company. Gone are the days waiting for the slow moving MRP company to update their slow system to add features or fix bugs for us. Our hands were tied with the old system, and now the door is open to all sorts of possibilities. People are constantly saying "wow, we can do xyz now!" or "this wasn't even possible before with the old system". The daily complaints about the old system have been replaced with daily feature requests and positive comments.
The small team that built it remains on staff, and if something doesn't work, we fix it. If someone requests a feature that makes sense to the overall design goal, we sketch it out, schedule it, and implement it. We carefully weigh everyone's feature requests and implement only those that make sense for the overall "greater good" of the system. That way we keep feature creep down while building something that helps people get their job done faster. In the time between bug fixes and feature request, we are constantly polishing the system to make it more efficient. We now have a DEV environment where we can test out new ideas and features and release them to production only when they are ready.
Since we built ours with FOSS technologies, it's been a joy to integrate with our other trading partners (suppliers, our web store shopping cart, etc). The money we are saving on not having to pay for licensing costs (recurring yearly, never ending), we have invested in hardware and infrastructure to run the new system.
Perhaps the biggest benefit is when the system is released, the company will have a team on staff that knows the system inside and out, because they built it.
I would highly recommend any company considering buying an out-of-the-box ERP system to consider having an in-house team build them a custom solution. With the right group of developers, it can be a really rewarding experience for everyone!
*Crickets*
Did you see where it only took 4 months? I haven't seen an SAP **upgrade** that went that fast, much less a deployment.
Of course, the reason for that isn't the (complete) ineptitude of people at SAP, or the superstar statusing of the engineers at Tesla.
Its easy to build a one-off solution that works for what a company needs on day one, do it quickly and be successful. Its vastly harder to build a one-off solution that still works for what the company needs done ten years down the line. And damn near impossible to build a one-off solution that just magically has equivalent success and value to other companies just by open sourcing it.
SAP upgrades can easily take that long, but SAP can easily run organizations an order of magnitude bigger, and two orders of magnitude more complicated than Tesla.
IMO, the key thing people should get from this is the importance of making sure you buy what you actually need. If Tesla could replace their SAP system in four months with 25 engineers, odds are pretty high they had overpurchased when they went with SAP to begin with.
Everyone knows of a company that is implementing SAP. Can anyone name a company that has completed their implementation of SAP?
When every organization did the same thing, had in-house staff to support it, and didn't have to bother with consultants. It can be a problem to keep track of all the different legal changes in the various locations though.
And hope, in 24 years, they've still got some geezers on the payroll who remember that old "Java" language from the early 21st century running on a crufty emulator no one can remember how to reinstall and can figure out all the places the engineers had said 4 bytes were plenty to hold a timestamp.
Maybe they exist, but have you ever seen a company that actually could deploy or upgrade SAP faster than building something in house?
I can play the same game...
Many, if not most, IT initiatives with homebrew tech succeed. It's hurts when it doesn't pays off, but almost always it is under budget and satisfies the unique requirements of the business. If the CEO got unlucky, too bad for him, but his CIO shouldn't be sitting in the big chair if he didn't at least recommend doing it in house.
Unfortunately statements like these are easy to reword the exact opposite and not sound crazy. If you look at Gartner's statistics, they have a pretty even split between successful and failed implementations. From my experience, the outcome is completely dependent on the personnel. If you get people that are intelligent and want it to succeed in leadership roles (and they get along with each other), you'll have a lot better chance than having people who are coming for a huge paycheck and flying home on a Thursday afternoon.
Since Tesla probably has a pretty intelligent staff, I'm not surprised that they would succeed.
Many, if not most, IT initiatives with homebrew tech fails.
I've seen $15k home-brewed storage solutions outpace $50k vended ones as well as $2500 servers outperform $20k ones. Those home-brews rely on talented in-house labor however, so if you can't keep the good employees around, you had better go the $50k route. Oh, don't forget the $4500/year maintenance contract and 3-5 day tier 1 support callbacks from people with such thick accents the interpreter needs an interpreter. That's not fail though because it's a purchased product amiright?
Join the Slashcott! Feb 10 thru Feb 17!
Yep a guy who built his first tech company at 24 got lucky
what does a founder of paypal know about large scale software projects compared to arm chair CIO on slashdot
The question is, does the company even know what it's going to be doing 10 years down the line? Does SAP make it any easier to change as you company evolves over the next 10 years? SAP isn't something you can just install and forget about it. It's just a set of tools. It's like a database server, web server, development environment, or operating system. Similar to SharePoint, except bigger. It doesn't do anything on it's own. You have to do a lot of work to make it work for you.
Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
Given that they only spent a few months on it and don't have experience building broadly applicable SAP systems, we can be pretty certain you are correct in this statement:
> Their system is probably custom-tailored to their business processes. Not only would it not be appropriate for many other businesses ...
It's probably still true if we change a few words:
> Their system is probably custom-tailored to their current business processes. Not only would it not be appropriate for many other businesses, including Tesla a year or two from now, ...
Generally, you should build within your core competency, and buy generic systems for generic tasks.
Tesla should design their own cars, especially electrical subsystems of the cars, but buy trashcans, spreadsheets, and SAP.
Their SAP needs aren't that different from the next company down the road.
In a gray area, where you need something customized to your needs, but it's mostly the same as what other companies use, FOSS fits the bill. You get the 95% of common functionality free, then build the 5% that's unique to your needs.
Where would you find an SAP system that cheap?
There is no way one could be rolled out in 4 months.
I have never seen one that did not go over budget and past schedule, nor did what they wanted on launch day. I have not seen that many, but enough to guess that is the common outcome.
SAP can easily run organizations an order of magnitude bigger
I lost it at easily.
There are two types of people in the world: Those who crave closure
It is easy! The users and their business models are simply all wrong LOL
It doesn't mean much now, it's built for the future.
In my experience, the biggest problem is when the CIO is not allowed to refuse requests. Once that is cleared (and the CIO is competent) then projects get finished on time and on budget.
In this case, it sounds like Elon had a lot of confidence in Jay's ability as CIO.
Tesla does a tremendous amount of the work in-house. This includes things like the class A metal stamping, battery packs and a slew of custom parts and electronics. Most auto makers warehouse pre-stamped body panels and parts. Tesla warehouses raw rolled steel and aluminum. They make the parts as needed. They have one of the most automated factories in the world so it's unlikely that an outside supplier would be able to do it cheaper.
While they do have a lot of things they get from other vendors, it's a fairly small list in comparison to most transportation manufacturers. In addition they have a relatively small number of products they make (including parts for other auto makers). Because of this they simply don't need SAP. It's a size and scope that you could do in-house. GM or Ford could never scrap their logistics suite and have a replacement in 4 months.
Risk vs. reward. What have you gained from not wearing seatbelts other than perhaps a few less wrinkles on your clothes
I have learned that most people are utter shit at estimating risk. Especially people who think they're smart and are good at it, but don't actually do the math. We spend trillions to prevent terrorism, but next to nothing to prevent drunk driving. It's because people think that risks they have control over are far less than those they don't, so drunk driving is "Well, I'll be driving, and I'm a good driver, so the risk must be low", and terrorism is "I'll be strapped into the plane and not in control... so it must be much, much worse."
The same kind of thinking applies to rolling your own software, instead of buying it. People are not objective about risk. They flat out suck at it. As for me... what I've learned is to wear my goddamned seat belt, because I read the statistics and know that there's about a 1 in 5 risk of getting into a car accident every year, and the seat belt means a 90% reduction in probable injury -- Without it, I'm just hamburger through the window.
Which is like most companies when they decide to cook their own complex software... they usually wind up paying more, but because they never analyze their own decisions, they, like you, think it's actually less.
#fuckbeta #iamslashdot #dicemustdie
"exactly meets the needs of the business" is important for some things, a huge waste of time and money for others. Those "some things" where it matters are generally the core competency of the business - what sets them apart from competitors. Google search needs a database that exactly meets their needs for searching a huge database. MySQL won't meet their need. For 99% of businesses, building a custom database engine would be stupid - MySQL, MS SQL, or Oracle would meet not only their current needs, but also their future needs.
Future needs can be a huge hidden expense for custom work; you've saddled the company with a requirement to build 2.0, 3,0, etc. on down the road if your business is built on something custom. So you should ask yourself "does the next company down the block have a similar need as we do?" If so, you and the company down the block should probably be sharing the development cost by both buying the same off-the-shelf software. You don't custom design your own trash cans, and most software is the same - yours should be about the same as the other guy uses.
If off-the-shelf software provides 95% of what you need but you need 5% custom, that's where FOSS is a perfect fit.
You get 95% of it done, tested and ready to go, for free, and you just develop the 5% that you need special.
Maybe they exist, but have you ever seen a company that actually could deploy or upgrade SAP faster than building something in house?
Depends how complete their understanding of their immediate, short term and mid-term requirements are.
I'd say yes, for companies that actually had a good handle on their requirements. I also think most companies are bad about calculating the real costs of doing a complex system like that DIY. Its easy to enumerate the up front costs, but much harder to enumerate the costs over time of support, enhancements, and hardest of all, operational inefficiencies because the system couldn't do "X" and there wasn't time, resources, understanding, etc, to implement it.
On the other hand, the statistics about SAP rollouts would tend to indicate a very high degree of risk inherent in attempting to use that system.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
The trick is to tell the CIO of the company that is your customer that the price and deadline time doubles every single time they make changes. The trick is to get an accurate scope and then a project manager that will tell the customer NO! in a way that they understand, I.E. excessively high prices and they sign a paper that the deadline is ok to ignore. Suddenly that feature that they dreamed up in a meeting is not so important anymore. If it's really important then they will be OK with letting the system solidify and then start a new project to add it.
Software is 100% identical to Construction. you can not add a new Solarium in the middle of it after construction is 80% complete, et most software companies dont have the balls to tell the customer, "It will cost you an additional $22,000,000 at this point to add that and we have to throw away everything and start over.
The ones that do are the ones that are successful.
Do not look at laser with remaining good eye.
>> SAP can easily run organizations an order of magnitude bigger
..."
> I lost it at easily.
I misread it as "SAP can easily ruin
Also FatPhil on SoylentNews, id 863
I think all that stuff is just as much a problem with SAP. Lots of costs because it did not do X on launch day.
Not saying you're wrong, just saying it is not unique to building in house.
Sure they built it in 4 months...
But likely spent the last 9 years figuring out why SAP was bad. Hence they knew what they wanted (by now)... Hire some good s/w developers and voila... you'll have a better system from the get-go. That's business systems 101: it's all about domain knowledge. Sure they built it in 4 months, but I see it took them 8.6 years to create it... by understanding why the SAP solution sucked and the experience on what worked and what didn't.
If they started from scratch with no SAP experience.... well I'm sure we'd see a different story. The same story as Oracle, MS, HP, IBM, and SAP (i.e. their in-house systems suck big time).
Now some new MBA graduate will disagree: now new systems can be built in 4 months, muck did it... then again...
It's one of those great mysteries.
I've yet to meet anyone that didn't despise SAP. It's a terrible user experience, it's a terrible admin experience, and it costs a metric-fucktonne.
Trying to implement SAP has literally bankrupted companies!
And yet, it persists. It must provide some value to someone, somewhere..
What? ERP systems have been around since the 70s... SAP released R/2 in '79. If you're talking about R/3 (when they introduced server-client architecture), it was released in 1992.
"Trolls they were, but filled with the evil will of their master: a fell race..." -- J.R.R. Tolkien on Olog-hai
This sentence from the summary is just great:
“When Tesla Motors needed to improve the back-end software that runs its business, CEO Elon Musk decided not to upgrade the company's SAP system.”
Someone should make a poster from it.
To me one of the 'hidden' returns on a gamble & investment like Tesla has made is a social one. I can't even imagine the pride the IT team at Tesla feels to go from resetting user passwords to bragging to their friends that they built from scratch an operating ERP system. In a podcast Jay Vijayan mentions that the upfront costs were similar in analysis, but the later savings in cost can be 're-invested' into the people and organization. To me it seems a no-brainer to embark on this investment, since the returned value is something that can't be reasonably purchased. (IMHO) podcast here: http://www.metisstrategy.com/interview/jay-vijayan/ -Ryan
Elon Musk is the alpha-dog to rule them all. And not surprisingly, he hires alpha-dogs and superstars in all his businesses. THIS is how they achieved the near-impossible, by engineering and launching their own man-rated launcher and spacecraft all by themselves.
We know that small teams of highly-motivated and highly-skilled people can achieve great things. So it should really come as little surprise that they can pull writing an in-house SAP replacement in such a ridiculously small amount of time.
Guess we're seeing just how well the 'only hire A-players' policy works...
That's the point. There is software that can be used to run any company in any field. It's called a compiler, although it does require a good deal of customization.
I don't know, but it works for me.
I disagree. For our implementation (see my post above), it was two of us and we managed to connect many disparate applications. As long as you understand what you're interfacing with and sufficient API documentation exists, it's not that difficult. We managed to connect to UPS, FedEx and USPS for shipping label generation/package tracking, Authorize.net for payment processing, DBA Manufacturing (the legacy MRP system we were using) for importing the customers, suppliers, and items, and OpenCart for our web store.
It took the two of us about 12 months of solid work to accomplish this. I'm sure Telsa's ERP system involves a lot of supply-chain connections (good ERP systems need this capability), yet they were able to pull it off in 4 months.