Slashdot Mirror
Invented-Here Syndrome
edA-qa writes: Are you afraid to write code? Does the thought linger in your brain that somewhere out there somebody has already done this? Do you find yourself trapped in an analysis cycle where nothing is getting done? Is your product mutating to accommodate third party components? If yes, then perhaps you are suffering from invented-here syndrome.
Most of use are aware of not-invented-here syndrome, but the opposite problem is perhaps equally troublesome. We can get stuck in the mindset that there must be a product, library, or code sample, that already does what we want. Instead of just writing the code we need a lot of effort is spent testing out modules and trying to accommodate our own code. At some point we need to just say, 'stop!', and write the code ourselves.
Most of use are aware of not-invented-here syndrome, but the opposite problem is perhaps equally troublesome. We can get stuck in the mindset that there must be a product, library, or code sample, that already does what we want. Instead of just writing the code we need a lot of effort is spent testing out modules and trying to accommodate our own code. At some point we need to just say, 'stop!', and write the code ourselves.
Isn't a running joke that they've invented/done everything already ?
UPS Sucks
I do not see this topic brought up nearly as much, which worries me. I have worked on quite a few projects where the unwillingness to write functionality internally lead to excessive testing of external options and overuse of generic frameworks which not only increased the dependency/complexity of the project but often required just as many lines of code to use as just writing our own damn module would have.
It feels like this is worst in the Java (enterprise) community, but that could be my imagination. Sometimes I think those programmers need their 3rd party instantiation taken away from them....
And crap, looks like I have been moved over to slashdot-beta so I will probably never see if I get responses....
It comes down to the quality of the solution, versus the added headaches of managing dependencies. It's not a religious issue, but one of experience and engineering. It's also a good idea to make sure that if you're inventing something that someone else might eventually provide, you at least loosely couple it, so it can be swapped out later.
With those thoughts in mind, don't get stuck in analysis paralysis. Use judgment and move on.
At some point we need to just say, 'stop!', and write the code ourselves.
I wonder how much of "invented here" syndrome is related with frustration with curation on the popular curated platforms (iOS, Windows Phone, Windows RT, and game consoles). Cryptographic lockdown applied by the operating system publisher blocks end users from writing their own applications or writing a mod for an existing application. Because people are unwilling to go through the organizational overhead of becoming a licensed developer, they stick with the vanilla version of whatever they can get from the platform's official app store.
"At some point we need to just say, 'stop!', and write the code ourselves."
Yeaaaa, about that...
This should explain that idiotic thought outright.
I'd say part of the cause of "invented-here syndrome" can be "not-good-enough syndrome." I'm often comparing my programming skills to people I see online - people whose skills far outpace my own. So when it comes time to access my programming skills, I'll understate how good I am because I'm simply not as good as those "coding superstars." Of course, when you see the online results of code people have written, you don't see the idiotic mistakes they made, the typos they've had to correct, the hours they spent Googling for an answer to a pesky problem. You just see some elegant, amazing looking code. It can be a daily struggle to balance admiring the programming skills of others without trying to compare myself to them (and thus knocking my own skills).
My sci-fi novel, Ghost Thief, is now available from Amazon.com.
I just read it as InventeD here SystemD as some sort of new Daemon that get's included :)
Sometimes a particular problem has already been solved, but that doesn't mean it's not worth taking the time to code your own solution in order to improve your own abilities and to engage in the kind of thinking necessary to develop algorithms and solve difficult problems. You learn a surprising amount of things when you have to build or implement something yourself as opposed to taking something that already exists.
While there's definitely a business case to be made for using existing solutions, if you're doing something on your own and don't need it immediately, there's no reason why you can't take the time to roll your own code and likely improve your abilities and knowledge in the process.
If you're looking for a pre-existing library, then it wasn't written here, it was written there. This should be called invented there syndrome.
You can lead a horse to water, but you can't make it dissolve.
Take idiots that put "GMT" or "MST" at the end of a time when our system (and most others) can't handle that crap.
The data was created by someone else. They most likely had the ability to export it in the correct way, but were too .... inexperienced .... to do it the right way.
Instead of trying to teach/convince them to do it right, I simply write a script to fix whatever bit of stupidity they created.
excitingthingstodo.blogspot.com
I've ran into this ALLL THE TIME! There is some perception that if it's done elsewhere it's good, and if it's done internally it's crap. This exists in not only management but in coworkers. I try to explain all the time the advantages of in-house software
1. Does exactly what we want, how we want it
2. Does it now
3. We update on our own terms, we are not forced to migrate when external migrates
It is a huge waste of time to have to update tons of things internally to work with some refactoring done externally, and a huge waste of time to jump loops and hoops to try to shove the square peg in the round hole.
I've presented solutions using ~20 lines of python code, but been told "No, we don't want to have to maintain that" (a bogus argument), "Let's instead leverage these 5 million+ line codebase external products, configured in a non-intended way, to accomplish 80% of the goal you accomplished in 20 direct lines".
It's a very similar trend I think to what I see all the time with knowledge. There is no such thing as first-hand knowledge. I can never tell people my discoveries or theories, "Nope, that's not what wiki says." Just because they know me and the source of wiki is some anonymous as far as they are concerned, the "anonymous" source becomes "truth", because it can be verified through some sort of self-fulfilling dogma.
you're missing the Point. The point is: Do not RE-invent the wheel.
Honestly I think my problem is that I tend to write my own code for stuff rather than trying to figure out what someone else has done and fit my thought processes into how they were working it through.
Examples:
I wrote a status management program to keep track of the work I do for yearly reviews. Tracks projects, hours, and even builds a timecard.
I wrote an inventory management program to keep track of our systems. We have an official asset tracking tool but it doesn't accommodate virtual machines, I can't script updates to it like I can my inventory program, the asset tracking system doesn't keep track of build dates or graph out the life timeline of systems, etc. I can customize it to do pretty much anything.
I wrote a budget management program where I can enter my financials (bank statements, credit card statements), plot out the trends, and do forecasts of what my financials look like as far out as I can to set a limit (currently 18 months out).
I wrote a combat management tool for one of the games I play where I just need to enter the data and it spits out my target numbers.
I think this goes back to when I started back in the early 80's. When you needed to do something, you either had to pay for it or learn how to do it yourself. They may not have the features other programs have but they're certainly customized to my needs.
[John]
Shit better not happen!
I will usually spend an hour or so looking for an existing solution if I expect it to take only a couple days to write the code. Even when I expect it to take a week or two I won't spend more than a couple hours looking around. The problem is that even if I find code that does pretty much what I want to do, it will usually take a day or two of screwing around with it to figure out how to make some required part of it work in my overall project. Maybe 95% will be just the way I need it but there seems to always be a gotcha that I don't find until I'm well into the project. Generally I just find that it's faster to do myself rather than try to bend someone else's code to work with mine.
I've seen the end result of this a lot working in systems integration and engineering. The problem is that, yes, most functionality has been written in some library or available through a public API, web service, whatever. Especially with mobile stuff, Apple/Google/Microsoft give the developer huge amounts of pre-built functionality, and encourage its use.
The overall problems with it are, in my mind,
- Developers and systems people not knowing how that huge chunk of functionality they use actually does what it does
- Introducing dependencies on third party applications which may or may not be around later, have spotty support, etc.
- Making applications more complex to deploy and debug -- "is this my bug or the API's bug? Why is a single row database update taking 45 seconds and 100% of a CPU core?"
The reverse problem on the other hand has the potential to be worse. No one should be coding core functionality that has the potential to fail spectacularly or have major security problems. Examples might be writing your own PKI stack instead of relying on the OS/webserver to do it, designing your own file transfer protocols unless you have a _really_ good reason, and many more. So with NIH syndrome, you have to really trust that your developers did everything right. With IH syndrome, you need to install an application, plus the 45 modules it depends on, plus provide it access to public APIs, etc.
I think the "solution", even though there's no right answer for all situations, is to make sure app developers are actually understanding development. It's too easy to write applications by gluing together pieces. With the framework movement, the pieces are much bigger and hide way more from the developer than, say, a library function.
From my side, in systems, we have way too many admins who are scared of scripting. Windows installations are moving towards PowerShell now, and while very useful, PS hides almost everything from the end user. Scripts that used to be 100 lines of loading/parsing/checking code are reduced to a single call to a chain of cmdlets. Very powerful, but the language itself isn't the most intuitive out there and borrows syntax from many languages. This leads to admins finding something on StackOverflow and copying/using it unmodified and unverified, simply because they don't know what it actually does.
Hey, that sounds where I work! At least until they blame me for the fuckups I've had to cope with...
You will always have to write some code of your own. Even if you use a CodeIgniter, AngularJS, and every prewritten function on StackExchange, still, you will have to write some code to configure the frameworks and to pull it all together.
You will always use some of someone else's code, too. Aren't you using Linux or something? You didn't write your own OS, did you? You're using a database, like MySQL or PostgreSQL or something, right? You didn't write your own database system, did you? And are you using a web server like Apache or Nginx?
So the question is not, should I write my own code or use someone else's. The question is where to draw the line.
I'm a web programmer, your typical LAMP developer (well, LAPP --- I use PostgreSQL). Like many PHP programmers, I first concentrated too much on the PHP. PHP is not the best language, as many have said, but I don't think it's quite as bad as people make it out to be. Anyway, I never took up any of the PHP frameworks. They seemed like too much trouble to adapt. (I should point out that I started with an intranet with a dozen or so applications already built.) I would research PHP frameworks from time to time, but always rejected them all, and felt a little self-doubt in doing so: "Do I suffer from Not-Invented-Here Syndrome?"
But PHP, and scripting languages in general, provide the right level of abstraction, I think. It takes care of memory management. It provides a bunch of functions for HTTP. It has its own templating syntax. It's great if you don't overuse it. In other words, in the MVC pattern, PHP does great for the View and, together with Apache, the Controller. But if you write a lot of your Model in PHP, with all this data processing, checking, calculations, etc. --- well, that's what the database is for, I think.
So, instead of eventually adopting a PHP framework, I learned more and more about Apache and PostgreSQL, and I learned that a lot of the things that I was doing in PHP could be done in SQL or in the Apache configuration, with a lot less typing (a lot more reading but a lot less typing). While most people are busy trying figure out how to write the practically all of the MVC in PHP, I realized that Apache was part of the Controller, PostgreSQL was part of the Model, and the browser was part of the View. I use PHP just to help them out, only when needed.
...unless you have to redesign your car to make it fit.
I think the point of the article was that that's sometimes the case.
Part of the problem is the CYA issue.
If you're writing the code, you sound like a laborer ("I have to..."). If it breaks, it's your fault and you're on the hook publicly.
If you present a third-party component in a meeting, you sound like a manager ("I propose that we..."). Once three or four other people in the meeting have concurred, if something breaks it's the third party's fault. A ticket or two are initiated, it's someone else's problem and everybody gets to cast blame somewhere beyond the walls of the company.
Rational behavior, regrettably.
STOP . AMERICA . NOW
I'd say part of the cause of "invented-here syndrome" can be "not-good-enough syndrome." I'm often comparing my programming skills to people I see online - people whose skills far outpace my own. So when it comes time to access my programming skills, I'll understate how good I am because I'm simply not as good as those "coding superstars."
I have a solution to that exact problem. Just download any of the most popular web CMSes built in PHP. Wordpress, Drupal, Joomla, or, if you think you're a tough one, Typo3 or Typo3 Neos. Install it (good luck with Neos, you'll need it) and load up the ERD with MySQL Workbench and stand amazed at the sight of the shittiest class of software architecture ever concieved by lifeforms able to type on a keyboard.
Seriously, if anything showed me that I must be in the upper single digit percentile of software devs, it's looking at systems that have an install base of 0.5 million or more. In the case of Typo3 Neos and TypoScript it might cause your head to explode. You have been warned.
Some of these systems are 10 years in making and emphasise that there are many people around that have no business programming what-so-ever. Right now I'm trying to do something useful with Wordpress taxonomies and categories - it's beyond insane what these people have built, I guarantee you. I actually just now had to take a coffee break of 1.5 hours just to let my frustration ease off. ... I spent that time designing a CMS architecture, if only not to forget how it's actually done right.
Looking at those systems will restore your self-confidence, that's guaranteed. Although it will also seriously make you doubt humanity in general. It's a tough tradeoff, I admit. But a change in perspective.
We suffer more in our imagination than in reality. - Seneca
I always fantasized that some specific mexican icon represented in fact some counterintuitive physical body image... I went to google and, voila, somebody else had already created the diagram. No need to create the illustration material: http://img.desmotivaciones.es/201202/VirgenVagina.jpg
Largely, that is.
My time to write a function is accounted entirely differently than the expense to acquire a commercial package that does the same. The bias is to prefer something that can be purchased over that which is written in-house.
Don't even get me started on free software. Management values an application based on how much they paid for it, either in developer-hours or from the expense budget. Stuff that costs zero must be worth zero. Never mind what functions it actually provides.
Have gnu, will travel.
Thanks for bringing this up and giving it a name, first off; it needed one.
Over the years I have seen this mentality a lot, but it's almost always pushed onto a team from management rather than home-grown in the minds of the engineers. I see it as a systemic issue at software companies that the leaders see their own engineers as the least preferable choice to actually build new things. I think this arises from a few factors:
- open-source software has so successfully marketed itself as better than third-party enterprise software that managers think they are supposed to prefer it over in-house solutions that are informed by the knowledge of their own business. For obviously generic components like JSON parsers, this is true, but leaders often take this mentality to comical lengths.
- resume-building is about keywords, because that's all recruiters understand. Junior engineers realize that if they build their own components, they will not "get credit for it" later. But put Spring and Hibernate on your resume and the recruiters will come. So why not cram as many third-party components in as possible and really build that skills section?
- it can appear easier to manage a team that is struggling under the bloat of a third-party framework rather than creating something better. PMs feel more in control; managers don't have to think as much, and the engineers get their keywords for their resume. Everyone wins!
- a single phrase: "don't re-invent the wheel." I think this one sentence is responsible for the suppression of countless creative ideas. PMs and managers use it like a weapon to keep control; consultants use it to poison the reputations of in-house engineers in the minds of the managers whose money they want. Senior engineers even use it to squash down the ideas of competitors for their position. If we engineers did nothing but thoroughly discredit this phrase, turning it into a shared joke ala "what could go wrong?", the world would be a better place.
"Invented here" is somewhat a illusion.
Working with frameworks and other people's code is a engineering problem (design, integration, etc..)
Developing code is a computer programming problem.
Two different problems, and why Software Engineerings are [desired] unicorns in very large software projects.
MS opened .NET and you can read and step through the code. Code and comments are very good, and you figure out internals of particular class/function.
However, try reading Native C++ libraries that being shipped with Visual Studio. Modern C++ std.libraries are so complex, that average programmer will never figure out their inner-workings.
MS version of LinkeList
http://referencesource.microsoft.com/#System/compmod/system/collections/generic/linkedlist.cs,df5a6c7b6b60da4f
Now, pull out VS and try reading/stepping through List class code.
So much that I hate making system calls.
Here's a blog post I wrote on the topic of "finding a function (etc) that already does what you want": http://www.danielbigham.ca/cgi... I think there are actually some pretty exciting possibilities in this area...
Should be "invented somewhere" or "invented there" syndrome, not "invented here" (for something you are not inventing because you are caught up in analysis).
I'd rather write something that works than add a dependency for a framework/library that will probably be abandoned (either by the creators or users moving on to the next framework), or version 2.0 will break code compatibility with version 1.0. Nothing is sadder than the constant parade of job postings looking for people with skills in loser frameworks/libraries because a company has code stuck using them long after the rest of the world has moved on.
Having a simple modules base that can be combined gives you the best of both worlds. You can quickly test your candidates (as they are simple) and write the code that glues them together yourself.
That linked list code is actually very straightforward. I wouldn't need to step through because it's written as I'd expect; no Visual Studio necessary. (I've also written a lot of linked lists in my time.)
However, it is a lot of code for what is one of the simplest and most common data structures around.
"I found code that already does this" sounds like you're a better problem solver. It's self-congratulatory nonsense. If the result is very complex, you congratulate yourself even more, for being able to do complex things, rather than punching yourself in the head like you should, for making things more difficult than they need to be.
If you found code that really does make your life easier, that's great. It hardly ever happens.
Haskell has a nice search engine for functions: https://www.haskell.org/hoogle...
Given that Haskell's type system is rather expressive, you can give the type of the function and expect to find what you are looking for.
A good example is https://www.haskell.org/hoogle...
It returns Sort and HeapSort as top results.
I see that one a lot. The assumption that because the software vendor did it one way, That's. The. Way. It's. Done.
It's nonsense of course. Our testing system was designed for the most common use case. Maybe 50 dialogs with 5 or 6 controls each. Our system has thousands of dialogs, some of which have as many as 50 controls on a dialog (It's old, legacy, badly designed. I know....). To make that system scale, we had to develop our own abstraction system, an API and a different object mapping system.
In the end, it worked quite nicely. Had we just used it as was designed out of the box, we'd have scrapped it by now, or quadrupled our staff running on that maintenance treadmill. Vendors can be pretty bad at scaling or special cases.
Please do not read this sig. Thank you.
It's strange that software people are always urging each other not to re-invent the wheel, given they inhabit a world where all wheels are square.
Half of me thinks this is indeed one of the largest problems of the software industry, en masse, at all levels. There are SO MANY dependencies that often import tens or even hundreds of megabytes just for the sake of doing something flashy with no real long-term value. These are really business decisions, though, not design decisions. IMHO any developer can and should be able to approach a problem from either perspective (with the exception of obvious large-scale frameworks that are more-or-less ubiquitous and non-debatable)
I work for a large US federal agency. Holy moly, we've got not invented here syndrome.
The problem with black-box programming is that it's a trap. Far more often than anyone cares to admit, the black box implements functionality in an unreliable or inefficient manner. When you're dealing with code that you wrote yourself, you can correct that behaviour of the "grey" box. But with a third-party black box, all you can do is file a bug report and hope that someone can not only replicate the problem, but that they'll give it high enough priority to fix it before you retire or your project is cancelled.
The worst culprit for black box problems are frameworks of all kinds. Some say you're not a "real programmer" until you've written your own framework. I firmly believe that's true, because what is a reusable code base on a large project except a custom framework?
The difference between a custom framework and an off-the-shelf one is that your custom framework is designed and coded with your project in mind, to service the bulk of your project's needs while maintaining enough flexibility to deal with the exceptional cases of your project. A third party black box framework is pretty much never designed that way. It was designed to serve the needs of someone else's conceptual or real project, then tweaked and adapted to serve needs it wasn't originally designed for, and finally unleashed on an unsuspecting world as "the next big thing."
A pox upon frameworks, I say. Design a solid object model, code to it, use it, and get over the fact that you're going to have to write some code.
At least if you wrote the code, you can fix it. Without worrying about whether some upstream integrator will deign to consider your "fix" worthy of integration to the mainstream code. Without having to wait for someone else to replicate, analyze, prioritize, schedule, implement, and test a fix for your problem.
Realistically, any half decent custom framework isn't going to be more than 10% of your total code base anyhow. "Framework" is just a fancy term for what was called for decades "application library."
I do not fail; I succeed at finding out what does not work.
In the last couple of years, this has become the one and only true and accepted way to do web development, either on the front end or on the server side. It's probably the biggest reason (aside from chronic underestimation of all projects) that I'm getting sick of being a web developer.
But then it's totally the point.
Do invent the wheel, if you need it to build a car.
Do not RE-invent the wheel, just because you built a car without wheels and now the usual wheel does not fit. Build your car in a way, it works with usual wheels.
And then there was the guy we called "the Professor" who wrote his own subroutine to do exactly what "strtok" accomplished even though we endlessly tried to explain to him that there was already a native C expression. To each his own, I guess.
Programmers are sensitive to taking a stance that might bring into question their intelligence, so this is a tough idea to convey sometimes - if a thick book and months of ramp-up are involved to get productive, chances are it's not worth it. The fear of someone saying "you're just not smart enough to use it properly" gets in the way of seeing the truth; generally, the more complicated the tool or framework, the more likely it's crap. I have seen great sins of architecture committed in the name of maintainability that wildly achieved the opposite. And those with the highest degrees were the worst offenders.
The only problem I see with it is that retarded if silverlight crap and the fact that the class has so many dependencies.
Silverlight shit is retarded and needs to go away and it wouldn't be MS if they didn't overcomplicate one of the simplest data structures around.