Slashdot Mirror
Why Software Builds Fail
itwbennett writes: A group of researchers from Google, the Hong Kong University of Science and Technology and the University of Nebraska undertook a study of over 26 million builds by 18,000 Google engineers from November 2012 through July 2013 to better understand what causes software builds to fail and, by extension, to improve developer productivity. And, while Google isn't representative of every developer everywhere, there are a few findings that stand out: Build frequency and developer (in)experience don't affect failure rates, most build errors are dependency-related, and C++ generates more build errors than Java (but they're easier to fix).
Half the time when I'm working on any sort of non-trivial program (that is too large to hold in my head all at once) and I need to make a breaking code change (and one that is not easily managed with refactoring tools), I'll make the change where it is obvious to me and then let the compiler tell me where it broke and hence where I need to make my fixes.
I am Slashdot. Are you Slashdot as well?
We've found that builds fail for three reasons: coding errors, dependency issues, command line argument mistakes. If you're a developer you should check these three things when your builds fail and you'll likely find the issue.
How is C++ / Java build error comparison related to the actual study in any way?
But then again.... I'm dumb.
My LaTeX builds rarely fail in MiKTeX. The compiler itself seems to be able to download packages and classes from a common repository (CTAN and its many mirrors).
If GOOD/Complete unit tests for code exist and this change would break it, How freaking tough is it to run the unit test before committing your change to source code control ?
UPS Sucks
If *only* there was a way to tell which dependencies were missing before I start an hour-long build process... perhaps a ./configure script, or a README file...
But more seriously, I'm currently trying to compile various source packages for MinGW and/or MSYS. MinGW is missing various unix-based headers (no surprise), while MSYS is missing ANSI-C headers (including stdint.h). They also have a habit of breaking well into the compilation process, even after ./configure seems to indicate that everything is okay. While trivially fixed, it's something that needs to be fixed in the source repository rather than on a per install basis.
I'm considering forking MinGW/MSYS on that alone.
Complexity
Coder's Stone: The programming language quick ref for iPad
Strongly typed languages catch more errors during compilation.
How does a unit test prevent a build error?
Dependencies just magnify all other problems. If your code depends on nothing then it won't break unless the compiler changes. Unfortunately such programs don't exist because you can never depend on nothing and do anything useful. In reality if you depended on nothing you'd end up writing your own console, your own I/O, pretty much your own CRT. This sounds great until you realize your dependency is now the hardware itself and it's likely your code won't be portable in any useful sense. That's why we have kernels.
The problem with C++ is that dependency management is usually file-level and developers 'rarely' care about any file-level constructs (and nor should they, it's an abstract packaging concept). As a result you try to drag in one enum and end up with 100 #includes and 500 new classes you don't care about. This causes bigger object files to be emitted, vastly slower linkage and lots of dependencies you don't expect. All it takes now is for one of those includes to #define something unexpected and BOOM...the house of cards comes crashing down.
Also, did I mention? The C preprocessor causes a lot of grief when it's abused.
It's hard to run a unit test on a program that doesn't build.
Deng Zhang commit untested code to repository again. Now compiler error, build box idle, that not good. I need you write script recompile code until it work. You have five minutes finish job now before you laid off.
I agree it's usually about dependencies, and also with their C++/Java conclusions. However, I find it hard to accept that there is no correlation between developer experience and build failures.
Maybe it's because even experienced devs often suck completely at proper build structure and dependency checks.... I certainly get that where I work - the "experienced" guys are just as likely to break builds, but some of the "experienced" guys are also fucking useless and counter-productive and are only considered "experienced" because of how many years they've been writing useless code.
I'm the "build guy" at my shop, because few are OCD enough to do it properly.
Once code is checked in and goes through the standard build process, that's where this is expected to occur because in my experience it's the local environment where the developer does the coding that's the root problem. Why? Developers don't refresh their build environment because of the potential for other problems it may create. I had one gig to unfuck some code at a company a couple of years ago and found out that in order to set up a Dev environment in this place could take two weeks or more depending on what team you were on. You had to go through a script, download this, install that, change this.. A nightmare. Updating dependencies on a local desktop created panics amongst the developers who were reluctant to ever change anything they had which "was working" because you could spend days trying to fix what was broken. Naturally any time they migrated code into test or production (there was no build system) things failed there because of dependency related issues. Also depending on who the developer was, they naturally felt that bypassing the Test/QA cycle was a job perk.
I found dozens of dependencies on desktops that were out of date, deprecated or had major vulnerabilities and that went for the production systems as well. It was bad all the way around from a best practices perspective. Daily production crashes were the norm, the VP of Dev had a monitor on his desk so he could "troubleshoot" production problems it was that bad.
Yes there's shops like this that are still out there.
Harrison's Postulate - "For every action there is an equal and opposite criticism"
The designers of Go talk a lot about dependency management (Go bans cyclic or unused dependencies, and in another sense of "dependency" the compiler links in all the runtime/stdlib it needs, so no external libraries need to be installed on the target machine). I found it interesting because I'd never thought of dependencies as an important focus for a language; maybe it's because of this experience at Google.
re-enter mom
incest is best
As soon as you rely on someone else's 3rd party code module, you're screwed. You're in for 10 years of constant nightmarish problems. So yes, decencies are the problem.
from the article : " A build was defined as “a single request from a programmer which executes one or more compiles” and deemed a failure if any compile in the build failed."
this is stupid .
When I code , I often do a first or second compile only to see all the trivial error I have made and correct them fast. I do not expect zero error .
I had an experience which was somewhat opposite (though, in a lot of ways pretty much the same).
At one point, the company went with a big giant universal build system.
Every piece of software, every module, every final build ... was recompiled from scratch on a nightly basis. It took a massive server farm many hours to do this. Even if no changes had been made.
What would happen would be someone would break a component. The build of that component, and every downstream dependency broke. The system had no concept of "this is a beta build, not for everybody" and "this is a release, and stable".
The result was that sometimes you'd have literally dozens of things which were now suddenly broken. It was too stupid of a build system to use the last known good.
So, all of a sudden you get one trivial change in some module about 4 steps removed from your stuff. But, it was all broken, and your stuff couldn't be properly built until someone fixed their stuff, and the build system went through at least one more cycle, often two.
Sometimes, companies get themselves into such a borked state with their build system (or lack thereof) that it makes doing any work impossible.
Some of us started keeping our own local copies, and writing local build scripts, because we couldn't rely on the company wide one to actually work much of the time.
Lost at C:>. Found at C.
Unit testing! The silver bullet for everything.
If your parents had unit testing you never would've been born.
This, this, this. Dependency issues on a build server are so often things like "we added this dependency on a library locally, but didn't update the build server", or even "we checked in a change that makes it work on the dev box, but didn't make a change needed for the server". That plus forgetting to merge in stuff is really the only way I ever see build breaks (where checked-in code fails to compile, as opposed to compiles failing locally).
Maybe that's part of the problem. Too many cooks spoiling the broth? Perhaps I'm naive, but 18k seems a bit much for what they produce.
And I've worked in many of them. I call this effect "build pollution" and a common cause is simply not checking in files to SVN that were added.
The #includes are referencing something on the local HDD that no other developer would receive if they resync the repo. End result: everyone else is now broken and you have to go poke the dev with a stick. The other problem is that far too few places have a presubmit test that can check if your code will build by the auto-builder *BEFORE* you submit your code. This means you have to break the build to see if the build would break which is bad practice and doesn't scale well to very large teams.
The other problem is that many devs don't practice basic HDD cleanliness as you describe. When the instructions for building your code reach a certain size people stop repeating them because they're too complicated. This means bad builds and local tools can be polluting the local build environment in ways that will affect future builds (which usually results in panic, flapping and comedy). Telling people to tear down their machines to clean state won't work if the process to restore to build state is too complex - they'll ignore you and this circus just keeps coming back into town. This also ties in to build reproducibility: if there is no automation to set up the build state (a preconfigured clean HDD image or VM image) then it becomes really hard to know what the build dependencies are. People forget they installed "crapware bag-o-bolts toolkit" and forget to tell everyone else. The IT department won't know. Possibly the build engineer will know because they set up the build machine nodes but then a new dev joins with a fresh OS and BANG. They should probably ask the auto-build build engineer what's on that build box, but most new devs wouldn't know this detail to ask the right person. :)
You need to know what's on your machine at all times, prune out, clean out, keep the build machine squeaky clean and pristine. Everything that goes onto the build box should be audited and maintained in fresh-boot-state ideally. A VM is extremely helpful in doing this but creating Windows VMs with the appropriate packages can be a painful manual clickfest process. In Linux this practice is rendered trivial and it's par for the course. ...and then there's that time I found build files included via network file share from a random guy's desktop. NOPE.
Is this some sort of cry for help? Are you ok? Do you need us to call someone?
The article doesn't mention what build system was used for the C++ code. GNU Autotools? CMake? Plain old makefiles? A build system can help you modularize your build so that a complete recompile does not occur after a syntax or linker error.
TFA seems to be referring to local builds. I HOPE nobody is watching my local builds, because it would be embarrassing at times.
That said, any developer who commits broken code to source control should be flogged. Unit tests help a lot, but it's rare that anybody wants to pay us to write unit tests.
People not checking this:
Installation A + Patch B = Installation B
That is: the result of patching a known fresh installation results in 'exactly' another known fresh installation. I've seen this a lot with badly maintained Linux kernels in hardward development. People patching up development hardware sometimes results in chaff files on the HDD which interact and cause strange "works-for-me" bugs. This usually results in the hilarious "Downgrade from version 2 to version 1 then upgrade to version 3" debacle. Or you could just...y'know...fresh install version 3 (which no-one does because "it takes too long").
Also: Things taking too long causes developers to get clever and shortcut things that are actually important. Try ninja-style machine maintenance out of office hours - e.g. cleaner bots, re-install bots, stuff like that. They'll never know you're cleaning house for them.
Simplicity of dependencies are the only way I survive C++ development. For instance I was just playing with OgreSDK, and it depended on my having a specific version of CMake installed in a specific way, in a specific directory. That's fine, oddly enough I can live with that. But how long before I combine Ogre with something that requires something different about the CMake installation?
So, for instance I like the Crypto++ library because I can cheat and just slam it into my multi-platform codebase without worrying about keeping it separate. And seeing that over 50% of my code needs to run on 4 platforms I have become very sensitive to this kind of crap. I consider a platform specific #ifdef's to be a personal failure (even though they are lightly sprinkled through my code).
I somewhat like boost with its mostly header only stuff but that is not without some consequences. POCO is cool but I find that having a single codebase that is multiplatform is an IDE configuration nightmare.
So what is my favourite solution? I would say that libraries have many virtues but that in the end any fights with them can be horrific. So I am going to go with my present favourite for multiplatform dependency goodness; and that is Cocos2d-x; it isn't perfect but the nice combination of it being a library and part of your code base results in some of the fewest compilation problems that I have ever had.
I would say that its primary cost is when you go to upgrade to a much newer version of the library. Basically you have a fight on your hands. But that fight is once in a blue moon. Whereas I find some other libraries are a non-stop fight, sometimes with no resolution.
How does a unit test prevent a build error?
Because you have to build your code to run the unit test. The two are (should be) intimately linked. And you should always be building against the latest official working source from the upstream repo.
Stick Men
.. Jones, the guy we let go last month. He's pretty much to blame for all build fails for the next few weeks. Then we can start blaming the marketing dept.
If your parents had unit testing you never would've been born.
I would still have been born, just not so buggy.
Dark Reflection
And of course everyone always builds with the same configuration, same compiler, on the same platform.
(We have CI servers in our environment. They break not infrequently. Why? Because someone commits a change that builds fine on Linux, and when MSVC gets ahold of it, it produces a warning that GCC doesn't catch and so the build fails. Or MSVC accepts some piece of code that is not actually legal C++ because it's too loose, so when the Linux buildbots get ahold of it, they complain.)
Then comes the installation and packaging failures. The dynamic libs get out of synch, wrong dll gets packaged in, etc
Then the build is good, it does not crash on every project. But it fails daily validation suite. After clearing that hurdle the build fails certification (same as validation but more detailed comparison with golden results).
So typically our last daily build is about one day old, last validated build is two or may be three days old. Last certified build could be three to five days old.
And this is a great improvement compared to the past.
Biggest advancement that helped us were dirt cheap prices for storage. So we are able to keep multiple older working builds for the developers. Not all of them need the latest build. Second biggest thing was the multicore machines with enough horsepower to launch all the library builds simultaneously. Third was the drop in network bandwidth prices, we are able to consolidate and synch the source code repo with very small (for a developer I mean, not for a video watcher) latencies.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
Why not just say that you should always build against the latest official working source before checkin? It has nothing to do with unit testing.
At any moment you can do:
1) Take the enum definition out of its source file
2) Add an include in the original, including the enum.
3) Include just the enum in your code, do not include the whole mess.
This would be easier if you would tell the compiler
#include
Of course if those symbols weren't in the include an error would occur, what would ensure that those symbols exist AFTER the include.
Your argument is a tautology: If I can detect the problem, then I should be able to detect the problem. Please say something of value instead.
A couple of years back I worked on a contract project where the original developers had clearly not actually tried a clean build from scratch with someone who didn't know the system. The project had python, postgres, java, and ruby (at least) and each of those had dependencies - some of which were specific versions of packages, some of which could just use the latest. None of it was documented, of course, because the packages were all just there in the original developers' system.
It was a nightmare to build and took maybe three weeks to build cleanly - half of this was because I was working on an old slow machine with only 500MB of memory - but then the company bought a new machine and a couple of packages that took an hour or two to build got down to only a few minutes.
Once the build worked I had a shell script that would run on a clean OS install and required only about three prompts to the user. It was at least minimally documented :) Then I got to start on understanding enough of the code to make the required changes (kind of a problem in itself as the requirements were, at best, foggy and at worst contradictory). Just about the time I had the first real code for them, the contract ended.
And of course everyone always builds with the same configuration, same compiler, on the same platform.
Yes, they should, and it should be scripted so that it's trivial so that there's no excuse for not doing it every single time.
Multiple compilers, multiple OSes and multiple binary architectures should all be used and they should all be available to every developer on the network. There should be enough network, storage and CPU capacity to make the builds quick so that there is no excuse for not doing them.
Stick Men
It sounds like like are borrowing Hollywood's playbook. The entertainment industry always wants to keep a buffer of talent available, so they maintain stables of screenwriters whose scripts will never be filmed, an uncountable number of actors waiting to make it big, and then there are all those doomed pilots and straight-to-video movies that keep entire production crews happy and employed.
I don't think you understand the study or c++ or unit testing.
You need to build to run a unit test, and that would have counted as a build error. In fact, with this study unit tests would have increased the build failures.
I really admire how Gentoo manages the build of so many packages from ebuild. I think the authors of this paper should study this project.
From the study:
Developers may build with the IDE’s own compiler during development, but must ensure the code successfully builds with the centralized build system before committing.
They are building before they commit and that is what is being measured.
When I code, I am thinking at a high level about the problem I'm solving. My builds fail because I make some trivial mistake, like using the wrong method name for the language I'm coding in, or forgetting a semicolon, or forgetting an import statement. Yes, the compiler catches it for me. I'd rather do that than switch from my high-level mental state in the problem domain to a low-level state of compiler and syntax trivialities. I'm solving a problem.
Why not just say that you should always build against the latest official working source before checkin? It has nothing to do with unit testing.
It does. You shouldn't be putting in new logic bugs with your deliveries. The code should compile cleanly and you should do due diligence to avoid putting in new bugs.
Bugs are expensive to fix, in terms of debugging time and refactoring of broken code to change additional changes built on top of the broken code.
It's amazing how much real progress you can make if you follow these simple rules.
Stick Men
I don't think you understand the study
I didn't RTFA because I could tell from the summary that it was a lot of nonsense.
or c++
I understand enough about C++ to know to avoid it wherever possible. I freely admit that my brain isn't big enough to fight against C++ with enough perseverance to get it to do anything useful correctly. I'll stick to C and scripting languages thank you very much.
or unit testing.
*cough* Test Driven Development *cough*
To become half way good at C++ you either need to devote you entire life to it over decades (somewhat like a monk) or have an IQ of at least 200. Life's too short and my IQ is miles too low.
Stick Men
It is fair to say there might have been a "unit" being tested and that is why he was conceived.
This excerpt says it all in my opinion: "Almost 65% of all Java build errors were classified as dependency-related, such as cases where the compiler couldnÃ(TM)t find a symbol (the most common one, 43% of all build errors), a package didnÃ(TM)t exist or Google-specific dependency check failed."
"The study found that the median build failure rate for C++ code was 38.4%, while the median for Java was 28.5%."
"Similarly, almost 53% of all C++ build errors were classified as dependency-related. The most common such errors were using an undeclared identifier and missing class variables."
OK, why can't a compiler just Google for the dependency?
Identify everything with a unique ID. When you need to reference it, your system searches all your code (and any other code repositories) for the unique ID.
Why hasn't this been solved?
> run the unit test before committing
Apparently it's impossible for the typical CONservative. Those Republicans refuse to run unit tests. I have fourteen developers working for me, and nine of them are those religious, racist morons. Their kind refuses to either compile or run unit tests before pushing their changes. That is the way of their kind. They are so entitled they expect others to fix the problems they create. Many(most?) of the problems in this industry could be fixed by not allowing religious or racist people into comp sci programs in college. I know nearly 99% of the problems I have at work are created by those Republicans. They just can't work with others.
The only thing I've found so far that works is that I have Jenkins (formerly known as Hudson, a continuous integration server) send them a dozen SMS messages every time they break the build. After receiving a $100+ SMS bill for the first month, many of them stopped fucking us over so often by knowingly pushing broken changes. Of course that didn't stop their kind for imposing on the rest of us to fix their garbage, but it helped.
I like to push my code to the repo to get the build server to run checkstyle and fail. At some point someone is going to take that stupid checkstyle out
With C++ I try to make the type system catch errors, for instance by using units for length different from time (boost units). And I often set up make to not only build binaries but also run unittests. Thus if unittests fail, the build fails. It is a lot better that the build fails for the developer, than later on during the release test, and must worse if the program is shipped with an error.
"Why do your builds fail? You're not using our platform, languages, framework, or other technologies."
I have often wondered why it seems that most C++ is written by people who would appear to be less than half good.
Sent from my ASR33 using ASCII
I know this is "offtopic" but stay with me and I'll bring it around on-topic...
A big question that people are throwing Billions of dollars & millions of internet comments about is "How can we get more women into programming/coding?"
Ok...b/c our industry is by default very complex, it's not unreasonable that to really drill down to an answer to that question might be fairly complex...the answer can be summarized, sure, but to really get at the problem it involves learning a bit.
Here, in this thread, we find out why...and it affects us **all** not just woman coders, or coders...it affects how the whole company works and the perception of value...witness:
Here we have a central thesis:
"There is a balance between no warnings and pedantic warnings, namely the useful ones."
Parent agrees, and describes how using a **proprietary software** (Eclipse) which adds an **extra abstraction layer** to an already ridiculous process...a process which we all know theoretically should be able to be done on a text editor
the fact that coding, the act of developing, software engineering, the 'real work' has such obtuse solutions, solutions to problems based on...
PEDANTIC choices...overkill...the lack of discretion...there are many reasons for this but that's another rant
it's alienating to new people regardless of gender...the only reason many people work jobs as coders is **for the money**
until we address these fundamental issues, the problems that arise only because some compiler programmer was overly pedantic due to lack of empathy skills will destroy any attempt to get non-traditional types into coding
right now, you basically have to be a bit autistic, or be able to think that way on command, in order to code...part of it is genetic, but part of it is deliberate...you have to train your mind to think in a "code" instruction manner...why would a woman do all this given other options?
the solution to pedantic, tone-deaf coding choices is, of course, a fresh perspective that can help get rid of problems from abstractions...
we need women in coding to help make coding more appealing to women
so, to make this on-topic, I think **more women in coding** is a long-term solution to problems in TFA
Thank you Dave Raggett
People just forget to do these things, then wonder why the nightly breaks.
Builds fail due to engineer error.
Java was made to relieve some of the complexes of memory management and the coding that went with it. Less code in an error prone stage of coding that was inherent in C++.
Hi, I'm one of the authors of the paper and an engineer at Google. I wanted to clarify some points that have come up in the comments.
First, we don't believe that failing builds are bad. We wanted to study the typical edit-compile-debug cycle that all developers (at least those writing in compiled languages) use to write code. It's perfectly fine to do something like change the signature of a method, compile, then use the compiler errors to find all places where you need to fix your code. We were interested in what kinds of compile errors people run into, how long it takes them to fix the errors, and how we can help you go from a failed to a successful build more quickly. For example, for one particular class of dependency error, we saw that people were spending too much time fixing it. So we created a tool to automatically fix the error and included the command to run the tool in the error message emitted by the compiler. After that we saw the fix time for that class of error drop significantly.
Second, this work is not related to checking in broken code. The builds we looked at are work-in-progress builds from Google developers working on their projects, so it's code in intermediate states of development, not code that has been checked in. It's possible that broken code may be checked in, but our continuous build system will catch that quickly and force you to fix the problem. So for all intents and purposes, all of the code checked into our depots builds cleanly.
Third, by dependency issues we probably don't mean what you think we mean. Within Google we use a custom build system with a custom build file format. Source code is grouped into build targets, and build targets depend on each other, even across languages. You can assume that code checked into the depot builds successfully, and that generally engineers are editing only code in their project and not in their dependencies. The dependency errors we describe in the paper usually result because someone added a source-code-level dependency without adding a matching dependency in the build file, resulting in a "cannot find symbol" error. For example, in a JUnit test you might write the code:
Assert.assertTrue(foo);
But if you don't add a dependency on JUnit to the build file, then you will get a compile error because the build system doesn't know where to find the Assert class. We would count that as a dependency error.
Finally, at Google there is no distinction between "builds on my machine" and "builds on someone else's machine." Our build system requires that all dependencies be explicitly declared, even environmental dependencies like compiler versions and environment variables, so that a build is reproducible on any machine. This is how we are able to distribute our builds. So it's impossible for code to build on a developer's local machine but not on the continuous build system.
I'm happy to answer further questions if people are interested.
Developers checking in code that won't compile. There case closed.
Multiple compilers, multiple OSes and multiple binary architectures should all be used and they should all be available to every developer on the network.
That's fine for something that's intended to run cross-platform, but not so much for something targeted to a specific operating system - code that's hooking Windows drivers isn't going to fare too well under Linux, for instance. As regards the unit tests, it's trivially easy to have code that runs fine in the unit tests but won't build for production because the test project contains the needed dependencies, but the coder forgot to put those dependencies into the production project. That's not an excuse for broken builds, but it does happen.
Please stand clear of the doors, por favor mantenganse alejado de las puertas
Developers may build with the IDE’s own compiler during development, but must ensure the code successfully builds with the centralized build system before committing.
Which is fine if you have the good fortune to have a mirror of the build system on your own machine, but in a lot of situations that isn't the case. Where I work we of course have the raw build scripts that we're expected to run locally before committing, but the CB system we use (Jenkins) occasionally just doesn't behave the same way because it has to do additional processing that we can't do locally, and sometimes will flag spurious errors that can only be fixed by clearing the workspace. Yay for incremental builds.
Please stand clear of the doors, por favor mantenganse alejado de las puertas
On the flip side, we usually get bitched at if there are no unit tests for new code.
Please stand clear of the doors, por favor mantenganse alejado de las puertas
Its the language you use!
Nicklaus Wirth has spent his life developing software "as simple as possible, but not simpler" (A. Einstein).
I program in Component Pascal (a super set of Oberon-2).
If I can get past the compiler I am 80% finished whereas with C++ if you can compile you are only 50% of the way there.
Strongly typed language.
Garbage collection (yes, Java pattern theirs based on Oberon).
Modules that are units of compilation and units of execution.
Everything is hidden within a module unless you explicitly export it.
"Header files" are automatically generated by the compiler.
Commands, procedures that can be executed in any modules without a main program.
Component Pascal is a lovely language.
Did you read the article? At all? This is about build breakages during development, while you are still writing code, before you are even committing it. This has nothing to do with submitted code whatsoever.
code that's hooking Windows drivers isn't going to fare too well under Linux
Of course it will, so long as you run Windows under Linux. All you have to do is have your test process spin up a Windows VM on a Linux machine and hook the driver there.
Then run the unit test on the subset of the program that does build.
And the benefit of this extra step is...?
Please stand clear of the doors, por favor mantenganse alejado de las puertas
As I understand it, it's a lot easier to automate starting a VM than to automate a dual boot. So the benefit is that developers' machines don't have to dual-boot for a pre-checkin automated smoke test.
But if you do the build that does not mean that the official build will work. Many of my build failures came because I forgot one less frequently used build option (ie, we build for several different boards across two architectures with a set of options, plus a simulator and set of regression tests). As in the code change looks entirely innocuous, no way it would work on one build but break another build, until I check in in and discover there's a conflict with someone else's code that's only in a configuration I forgot to build...
So solution seems simple, just build all the possible combinations on the local machine, every time, before checking in your one line change. Except that it may take well over an hour for those builds to finish and everyone's waiting on me to stamp the release and send it to QA, or it's 7:30 at night and I want to go home. So people make shortcuts. (and really, it's not a disaster if the daily build fails, you just submit the fix and go again)
Excuse is that it takes one to two hours for the all the combinations to build. We do have an automatic test build kicked off whenever any code is checked in though, which means that if there is a problem it gets found and fixed quickly before the real daily build is started at night. So only drawback to a failed test build is the embarrassment of emails being sent to coworkers.
And do you know how hard it is to get all the equipment necessary to do the job? It's not like I can just go grab this stuff off the shelf. Even getting vmware was a huge nightmare for awhile (most devs didn't need it). We can demand that the build servers get faster but it takes real political clout to make it happen (and I don't have any). For a smaller startup the resources may be far too scarce to get decent equipment.
I think turgid might have intended to allude to the fragile base class problem, where anything that derives from or instantiates the class needs to be recompiled whenever a field or a method signature is added to, changed in, or removed from the class. Apparently C++ suffers from it more than other languages.
That's why people should use the official compiler used in the build and use exactly the same makefiles as the build, and if IDEs are used then they should never count as having built before checkin (if the IDE user can't figure out a command line to type "make" then they need a simpler job).
Because for awhile C++ was extremely popular. Which means lots of developers using it. Which means lots of dumb developers using it.
Though everything will always have the below-median-proficiency developer, of course. But it seems that when you look across the board at all developers, regardless of language/system, that the less adequate developers migrate towards the popular languages and systems. It makes sense, as it's good job security, the laid off inept COBOL programer will have a really tough time finding another COBOL job, but an inept Java programmer will have a much easier time finding a new Java job.
But practically speaking, C++ is a strange hybrid. It is simultaneously a high level language and a low level language, and it mixes together several different programming paradigms, and it prefers to add keywords to allow variations on a feature rather than put its foot down and disallow an option. So there's a LOT to learn to become proficient relative to say Java or C.
Unit testing is like XML. If it's not solving the problem then you're not using enough of it.
I still am not seeing the need for "multiple compilers, multiple OSes, and multiple binary architectures" mentioned by the original poster for a product that is intended for a single platform.
Because platform requirements change.
Some changes are foreseen The "timed exclusive" is a common practice in the video game industry. Some developer programs with Sony Computer Entertainment include a financial incentive for a studio to release on PlayStation 4 several months before other platforms. At that point, the studio will want to have all its bug fixes and DLC in place for the big relaunch on Steam 52 weeks later. So the studio will want it building and running on PC and PS4 as soon as possible. Some changes are not Practically, all video game studios need to start on Windows or mobile unless they are staffed by veterans of a well-known studio. But in some cases, a startup studio's debut on PC might become enough of a hit that Sony offers the developer a devkit for PS4 and PS Vita to make a port and/or an exclusive game. Or an app originally intended for BlackBerry might have to be ported to Android and iOS after the popularity of BlackBerry itself fails to meet expectations.At a previous company we had a build that could take 8 to 16 hours or more on a developers PC. Slower/cheaper PCs, inefficient build system (visual studio using third party compiler), and a very very slow compiler. I fixed all this by migrating to gcc+cygwin+make and got builds down to 20-40 minutes. But in the meantime everyone was extremely nervous to change things that might cause a long build for everyone. So there were some big bugs in a major header file that people left alone and instead worked around, because changing it meant every file had to be rebuilt. There were typos that no one fixed (I used to think saclingFactor was a medical term until I realized it was as typo). And no one ever did a clean build except for the build master for the daily builds.
Ha, that would be bad. One company we had multi-hour builds for some large FPGAs, but that's an inherently long process, and it was always done by the hardware devs and never by the build master. They also had a mult-hour set of calculations on matlab to generate needed components for the system. So it would just run overnight when needed and then verified the next day (and those guys always got the best PCs compared to the hand-me-downs used to write software).
That pretty much reflects our situation, except that "one to two hours" can easily be higher even if all your platforms are going at once, depending on how much hardware you want to throw at it. Getting everything set up so that everyone can easily build on all platforms before submitting would provide relatively little benefit (especially when you consider that running the full test suite on all of those platforms pre-commit is basically completely unfeasible) and cost a tremendous amount of money in hardware. Meanwhile, like Darinbob's setup, if you check in something that borks the build, you get an email a couple hours later (ok, sometimes several hours later) telling you to fix it.
Google has a custom build system. The specification is not particularly groundbreaking but is used consistently across the codebase for all languages, which is nice. The main magic is a distributed storage and compilation system. Yes it is incremental, and caches all sorts of things. Dependency analysis is even used to prune tests which don't need to be re-run. As a result I compile & test a lot more than I would working on a similarly large system elsewhere. This is very helpful if you are doing test-driven-development.
http://google-engtools.blogspot.com/2011/06/build-in-cloud-accessing-source-code.html
http://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-system-works.html
http://google-engtools.blogspot.com/2011/09/build-in-cloud-distributing-build-steps.html
http://google-engtools.blogspot.com/2011/10/build-in-cloud-distributing-build.html
Everybody knows that code changes and mistakes are made. Then, every fix is an opportunity to make it better, stronger and reliable. When this happens developers loses compatibility and such. There is no way to keep code building everytime without errors. Building is a continuous process that the maintainers will keep code up for the rest of the software's life. Good luck and thanks for all the fish.
you're pissed that i'm right...that all the stupid Dungeons/Dragons bullshit of the world of coding is both the cause of "failure of software builds" and "lack of women in coding"
it's two symptoms of the same problem
the fact that you think a solution means this:
proves you are just being obtuse for no reason
***OF COURSE*** i'm not advocating shipping inferior products or making a compiler that is not effective for its intended purpose
you're an idiot...solving these problems will only help you...
unless you're scared of the competition
Thank you Dave Raggett
I read the first few paragraphs. They've *got* to be kidding - novice vs. experience is the number of builds in the last three months?
Let's try this: divide the programmers up into "people with > 1 yr, 3 yrs, and 5yr experience programming, and those with 1 yr. *Now* count the number of builds per group, and the failure rate.
mark
Ugh, I hate magic builds. You know, the kind that only build on a single developer's workstation using code that's not pushed to the central git server. Unacceptable. The only solution is to insist that all software be buildable on a vanilla machine, but most cowboy devs don't like edicts too much.
The way to fix the cowboys is to give them a gift. A gift that takes away their goddamn high horse. That gift is an automated build tool that automatically builds and deploys their pushed changes to the dev environment. That way, their laziness can override their egos and they accept the tool, but in order to use the tool, they have to spend the extra few minutes scripting their builds properly and an extra few seconds pushing their code.
The effects at my current client have been awesome. No more, "Oh, Bob has to do a build because he's the only one who can do it." Or "only Frank has the latest changes." Or "Oh, the build script is this word doc that might be up to date, but then, it might not be." Or "Oh, the production code isn't quite committed to git yet. It's on Chuck's laptop."
Yuck.
They don't grade fathers, but if your daughter's a stripper, you fucked up. --Chris Rock
Indeed. Experience if you did "at least 1,000 builds in the previous nine months". That's not experienced, try 10,000 builds in the last 10 years.
Sounds like a weak excuse for a research project.
I used to work at a company that supported pre-standard C++ compilers on Sun, HP, and DEC, and some of the stuff was also supposed to be compilable with VC++ 6. The build was never clean. Never. When they wanted to add AIX to the mix, some of us pushed back and said we should convert to C++98, which at least gave us a chance of a clean build.
"When you have eliminated the unacceptable, whatever is left, however improbable, must be the truthiness" - Holmes
The gap between (amount of time needed to compile and run at least the faster unit tests) and (amount of time between somebody else committing changes that can potentially interact with yours) seems to be closing around here. I don't have a solution yet.
"When you have eliminated the unacceptable, whatever is left, however improbable, must be the truthiness" - Holmes
so you just "dont get it"
no detail...just "i don't see it"
after 3 comments....if you were interested in discussion you'd have engaged with my points by now
you're just trolling now
also, new words to describe "do the best thing"...whether it be called "essential complexity" or "user-centered design" or whateverthefuck...those are just new words...same concept since humans first made fire...
it's about ******why its not happening*******
which is all addressed in my post...which you haven't discussed except to say
"i don't see it"
that's what i'm going to do to any more of your comments on this thread btw...
Thank you Dave Raggett
http://developers.slashdot.org...
Did you mount a military-grade, variable-focus MASER on an unlicensed artificial intelligence?
ok fine...looks like you want an actual discussion
"people really don't know" my ass...you're giving up without a fight
**bad design is fixable**
just look...at...microsoft...virtually all their shitty design elements can be changed and FIXED
again, anything that is designed is able to be improved
again, you're trying to 'straw man' me...nowhere did i indicate that I thought any of these things
damn you...my whole post was started by me explaining how it's hard to give this kind of concrete evidence and that the conversation i quoted was a window into understanding a complex situation
also, men made all the shitty programming languages in existence right now....ALL MADE BY MEN...so at least ***some*** men will at least tolerate shitty programing language design
the reason for women not being "in coding" right now is...complex...its not **one thing** and it spans millenia
you have to stop over simplifying without understanding the context
there is plenty of evidence to suggest women are ****better communicators****
which is ***all coding is***
humans communicating instructions to machines
you need to remember that when you analyze this situation
i'm still pissed at you for being so abstract with your replies
find a way to state your objection clearly in one thesis statement
Thank you Dave Raggett
haha no I totally see where you are coming from.
this is a good way to summarize it
also, you do a good job of explaining 'complexity' in regards to making a large computer program
see, I agree here too, but I'm not trying to fool you by agreeing with you
the 'general thrust' of my call for more women is programming is to create balance...this is a more abstract concept than what you are elucidating
i want to "deal them in" so to speak...you might even think of my comments coming from the perspective of a tech company owner...the person who sets hiring policy...than from one of the coder guys
thanks for the response...as I said I'm not a full-time coder guy but i like to think i get the concepts...i have a 'systems science' background and a dbase mgmt stint...my coding started with FORTRAN type stuff on huge data sets & massive data transfers between systems
Thank you Dave Raggett