Slashdot Mirror


Invented-Here Syndrome

edA-qa writes: Are you afraid to write code? Does the thought linger in your brain that somewhere out there somebody has already done this? Do you find yourself trapped in an analysis cycle where nothing is getting done? Is your product mutating to accommodate third party components? If yes, then perhaps you are suffering from invented-here syndrome.

Most of use are aware of not-invented-here syndrome, but the opposite problem is perhaps equally troublesome. We can get stuck in the mindset that there must be a product, library, or code sample, that already does what we want. Instead of just writing the code we need a lot of effort is spent testing out modules and trying to accommodate our own code. At some point we need to just say, 'stop!', and write the code ourselves.

14 of 158 comments (clear)

  1. About time... by jythie · · Score: 5, Insightful

    I do not see this topic brought up nearly as much, which worries me. I have worked on quite a few projects where the unwillingness to write functionality internally lead to excessive testing of external options and overuse of generic frameworks which not only increased the dependency/complexity of the project but often required just as many lines of code to use as just writing our own damn module would have.

    It feels like this is worst in the Java (enterprise) community, but that could be my imagination. Sometimes I think those programmers need their 3rd party instantiation taken away from them....

    And crap, looks like I have been moved over to slashdot-beta so I will probably never see if I get responses....

    1. Re:About time... by jythie · · Score: 5, Interesting

      One example that really drove me crazy was a developer who was tasked with putting in a 'retry' mechanic. Rather than simply working with the socket he pulled in some spring library that altered the socket behavior according to an external XML file and was praised by the lead for doing it this way. The code (java + XML) was several times longer than it would have been otherwise, plus a new library had to be included in the build/distribution, but it was still 'proper'. I think that was the breaking point for me working on that project.

    2. Re:About time... by itzly · · Score: 5, Interesting

      What do you do when some complex, external code has a bug ?

    3. Re:About time... by Austerity+Empowers · · Score: 4, Insightful

      It's the penny-wise pound-foolish issue when engineers and developers are forced to be mindful of schedules and business objectives. We in essence become as brain damanged as our managers, and start behaving irrationally.

      It's well known we have absolutely no capacity to estimate schedule accurately, but we do have the "gut feel". If your gut says that it will take a day to implement functionality, assume it's a week and just do it, it's trivial. If it says 2 weeks, it's actually 2 months plus three squirrel years and a llama month divided by e^-jwt, maybe spend a day or two evaluating options. If it feels like 6 months, try very hard to find something OTS, because this may become the project you're working on when you retire (which in todays parlance means: you die in your swivel chair of old age).

      You could of course be entirely wrong. Last week in fact I decided modifying a script to do what I need would take at least a month, had an epiphany in the shower and had it done in 6 hours. Guts have failed even Homer Simpson. But unless someone comes along who has been-there-done-that with a better option, and who demonstrates he' serious by NOT trying to railroad you in a meeting, but in fact just walks by the cube and says "hey, use this", you're usually better off trying it out yourself. At worst you waste some time but learn the problem, and how to best evaluate other solutions that come up.

    4. Re:About time... by blue9steel · · Score: 3, Insightful

      File a bug report.

    5. Re:About time... by itzly · · Score: 3, Insightful

      And then it gets fixed right away ?

    6. Re:About time... by Greyfox · · Score: 4, Insightful
      Or is just complex and unfamiliar. The problem with these frameworks is they work great when they work, but you only ever see them working because they've been published with the most trivial example. When you actually start trying to do things with them, you have to know implementation-level details of the framework in order to make it work for you. By the time you've invested all that time, you may as well have written something less generic that actually does what you want.

      Oh and when I say they work great, I was kind of lying. I have a favorite example. A while back a developer I was working with wrote some Spring/Hibernate code to pull records in from the database and print a billing report. Soon as he handed it off to me, I thought "What happens if I throw 100000 records at this?" Well what happened is that it crashed. So I cut the number in half and it still crashed. Down around 30000 records, it started taking about half an hour and THEN crashing.

      Turns out he was using the framework to pull all the records from a couple of different tables and doing the join in Java. The SQL join I wrote to test the code took a couple of minutes to run on a million records and returned the correct output. On a hundred thousand it was neighborhood of 10 seconds.

      Now the Spring/Hibernate people will be quick to point out that you can edit some XML file or something and make the framework do a join for you, thus solving that problem. And that is true, if you know a fair bit about the framework. And you'd have to know a fair bit about all the other frameworks they used on that project, too. By the time you got done learning all the frameworks they were using to the level of detail where you could actually be that effective with them, you could have written the application they'd built 10 times over.

      Fortunately this story has a happy ending. The team ended up deciding to run the original developer's code against the billing database several times a day so that it would never have so many records to process that it would crash, thus solving the problem once and for all!

      --

      I'm trying to teach myself to set people on fire with my mind... Is it hot in here?

    7. Re:About time... by Lodragandraoidh · · Score: 3, Interesting

      I've told this story elsewhere, but it applies directly to this issue, so I'll recap in short:

      Vender is contracted to create an integrated support application for large sums of money ($millions) over a 6 month period; contractor chooses an obscure commercial java framework to build the system on. The application is delivered and appears to work fine for several months, then starts getting sluggish, then a month later the application locks up - and has to be restarted. This progressively gets worse, and is asymptotic with the growth of the underlying customer base - and soon becomes completely useless - shutting down within minutes of being started with a memory exhaustion error.

      The main problem we found was the equivalent of a memory leak in Java. The code would instantiate objects based upon the framework in the main loop, and they would never go out of scope. Furthermore the code imported hundreds of libraries that were never used - further impacting clarity and understanding of what the thing was doing.

      To make a long story short, since this was already in production and now there was even more pressure to get a solution in place fast (and all the lawyers threats in the world can't replace a knowledgeable developer) - we rebuilt the whole system using perl in a little over 1 week. That solution is still running today - even as we've scaled orders of magnitude since then.

      So - to your point - this stuff really does happen, and wastes godawful amounts of time and money, when a more simpler home grown solution would do just as well, if not better.

      --

      Lodragan Draoidh
      The more you explain it, the more I don't understand it. - Mark Twain
  2. Quality of the solution. by digsbo · · Score: 4, Insightful

    It comes down to the quality of the solution, versus the added headaches of managing dependencies. It's not a religious issue, but one of experience and engineering. It's also a good idea to make sure that if you're inventing something that someone else might eventually provide, you at least loosely couple it, so it can be swapped out later.

    With those thoughts in mind, don't get stuck in analysis paralysis. Use judgment and move on.

    1. Re:Quality of the solution. by BitZtream · · Score: 3, Interesting

      Do you want to keep up with the world around you?

      I was having a discussion with a programmer who is 50 (I'm 38) the other day about the time when you used to be able to write an OS entirely by yourself, and how we both miss that time. It wasn't trivial, but the OS was that small, it could be done. Remember, Linux is just a kernel, not 'an OS', and Linux hasn't been the work of one make for 20 years at this point (Not trying to discount what Linus has done, but he has help :).

      Today, if you want to be able to produce useful software with sufficient features for most purposes, you can't write it all yourself. Well you can, but you'll be the last person to bring your 'whatever' to market/public view and they'll be 20 others that have more features and more shiny than you because they relied on some other libraries. A single person isn't making Windows 3.1 or newer in any realisticly useful time frame. Windows 95 is well beyond the scope of one person.

      Even a good modern text editor isn't going to be written from scratch, you've just not got the time to write all the basic editor features and things like a regexp engine. Yes, you CAN, but not while staying relevant.

      Sure, there are places where you can get by without dependancies or MUST have no dependencies. Embedded work and Cryptography are two things I have experience with where re-implementing the wheel isn't uncommon due to various constraints placed on you. Note: Not reinventing, reimplement

      With that said ... I'm writing an OS by myself. It will never do anything impressive and no one will ever use it, but it will exist and has already taught me why I hate the x86 processor line in about 18,000 different ways :)

      --
      Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
  3. Not-Good-Enough Syndrome by Jason+Levine · · Score: 4, Insightful

    I'd say part of the cause of "invented-here syndrome" can be "not-good-enough syndrome." I'm often comparing my programming skills to people I see online - people whose skills far outpace my own. So when it comes time to access my programming skills, I'll understate how good I am because I'm simply not as good as those "coding superstars." Of course, when you see the online results of code people have written, you don't see the idiotic mistakes they made, the typos they've had to correct, the hours they spent Googling for an answer to a pesky problem. You just see some elegant, amazing looking code. It can be a daily struggle to balance admiring the programming skills of others without trying to compare myself to them (and thus knocking my own skills).

    --
    My sci-fi novel, Ghost Thief, is now available from Amazon.com.
    1. Re:Not-Good-Enough Syndrome by Kjella · · Score: 4, Insightful

      Sounds like you've never seen what passes for production code in the bowels of a major corporation. Look at the questions on experts-exchange or stackoverflow. You can safely assume most of them are for paid work. After that, you shouldn't have self esteem issues anymore.

      --
      Live today, because you never know what tomorrow brings
  4. The Level of Abstraction by Art3x · · Score: 5, Interesting

    You will always have to write some code of your own. Even if you use a CodeIgniter, AngularJS, and every prewritten function on StackExchange, still, you will have to write some code to configure the frameworks and to pull it all together.

    You will always use some of someone else's code, too. Aren't you using Linux or something? You didn't write your own OS, did you? You're using a database, like MySQL or PostgreSQL or something, right? You didn't write your own database system, did you? And are you using a web server like Apache or Nginx?

    So the question is not, should I write my own code or use someone else's. The question is where to draw the line.

    I'm a web programmer, your typical LAMP developer (well, LAPP --- I use PostgreSQL). Like many PHP programmers, I first concentrated too much on the PHP. PHP is not the best language, as many have said, but I don't think it's quite as bad as people make it out to be. Anyway, I never took up any of the PHP frameworks. They seemed like too much trouble to adapt. (I should point out that I started with an intranet with a dozen or so applications already built.) I would research PHP frameworks from time to time, but always rejected them all, and felt a little self-doubt in doing so: "Do I suffer from Not-Invented-Here Syndrome?"

    But PHP, and scripting languages in general, provide the right level of abstraction, I think. It takes care of memory management. It provides a bunch of functions for HTTP. It has its own templating syntax. It's great if you don't overuse it. In other words, in the MVC pattern, PHP does great for the View and, together with Apache, the Controller. But if you write a lot of your Model in PHP, with all this data processing, checking, calculations, etc. --- well, that's what the database is for, I think.

    So, instead of eventually adopting a PHP framework, I learned more and more about Apache and PostgreSQL, and I learned that a lot of the things that I was doing in PHP could be done in SQL or in the Apache configuration, with a lot less typing (a lot more reading but a lot less typing). While most people are busy trying figure out how to write the practically all of the MVC in PHP, I realized that Apache was part of the Controller, PostgreSQL was part of the Model, and the browser was part of the View. I use PHP just to help them out, only when needed.

  5. Sociological problem: CYA by aussersterne · · Score: 5, Insightful

    Part of the problem is the CYA issue.

    If you're writing the code, you sound like a laborer ("I have to..."). If it breaks, it's your fault and you're on the hook publicly.

    If you present a third-party component in a meeting, you sound like a manager ("I propose that we..."). Once three or four other people in the meeting have concurred, if something breaks it's the third party's fault. A ticket or two are initiated, it's someone else's problem and everybody gets to cast blame somewhere beyond the walls of the company.

    Rational behavior, regrettably.

    --
    STOP . AMERICA . NOW