Environment Variables - Dev/Test/Production?

Woody asks: "It's common knowledge that the development environment should be the same as the test environment which should mimic the production environment whenever possible. I'm currently working on a project where we develop on Dell, test on IBM, and the production system is up in the air. For embedded systems the differences can be running on a VM versus running on the actual hardware. What I want to know is what kind of problems have Slashdot readers been faced with because of different environments being used for different phase of their projects, and how have they been overcome?"

Comments

Depends on the discipline of the developers

[BigLinuxGuy]

I've worked on several similar projects and one of the items that bit us was when the developers made assumptions based characteristics of the development platform that were somewhat radically different than the production platform. Assumptions around the size of data (not just word size, but buffer sizes for some text retrieved from "standard" libraries) caused a number of problems on several Java projects I've worked on (not to mention the ones in other languages). Data size seems to be one of the more commonly overlooked items in design and implementation, so I'd urge you to make sure that you've done your due diligence to ensure that you don't find those after the fact.

Another area is performance. For reasons I have yet to understand, there seems to be a prevalent myth that performance can be bolted on after the fact (the "make it work, then make it work fast" mindset). The truth of the matter is that performance has to be engineered in from the beginning or you simply spend a lot of time and money rewriting code that should never have been written in the first place. Sadly, educational institutions don't appear to place any emphasis on actual performance or teach the principles of performance tuning as part of their curricula.

Re:Depends on the discipline of the developers

[GoofyBoy]

>there seems to be a prevalent myth that performance can be bolted on after the fact (the "make it work, then make it work fast" mindset).

Its not a just myth; its a damned lie that hurts all of society as a whole.

Re:Depends on the discipline of the developers

[FroMan]

Here I would disagree. Performance should rarely even be a consideration until the product works.

As such, this does not mean to use braindead implementations, but worry about a working product first.

The first setp to a serious project is to work through the design. This means looking at the interfaces that the software will provide. Whether those are UI or API. Those are your targets that your users will work with.

Then you will need to have test cases for the interfaces that you have agreed on. These test cases should validate the accuracy of the system. Accuracy is key, far more than performance.

Finally, work on implementation. The implementation of the project is the most fluid of the rest of the system. You do not want to be changing APIs or UIs after they have been agreed upon unless absolutely necessary.

Performance, while not exactly an after thought should only be worried about once the problem is known to exist. Often more/better hardware can be thrown at the situation, if not now, 6 months down the road. If there is a problem with the algorithm, then you can change the implementation without affecting the interface.

Re:Depends on the discipline of the developers

[dubl-u]

For reasons I have yet to understand, there seems to be a prevalent myth that performance can be bolted on after the fact (the "make it work, then make it work fast" mindset). The truth of the matter is that performance has to be engineered in from the beginning or you simply spend a lot of time and money rewriting code that should never have been written in the first place.

Myth, eh? Personally, I do no "bolting". The steps I happily follow are

• make it work
• make it right
• make it fast

The notion here is that you get something basic working. Then you refactor your design to be the right one for where you ended up: a simple, clean design. Then you do performance testing, discover what the real bottlenecks are, and find ways to get the speed with minimal design distortion here. The you go back to step 1 and add more functionality.

There are a couple of benefits to doing the optimization last. One is that a clean design is relatively easy to optimize. But the big one is that by waiting until you have actual performance data, you get to spend your optimization time on the small number of actual bottlenecks, rather than the very large number of potential bottlenecks. That in turn means that you don't have a lot of premature optimization code of unknown value cluttering up your code base and retarding your progress.

Of course, this is not a license to be completely stupid. Before you start building, you should have a rough plausible architecture in mind. If there are substantial performance risks at the core of a project's architecture, it's worth spending a day or two hacking together an experiment to see the if your basic theories are sound.

As Knuth says, "We should forget about small efficiencies, say about 97% of the time: PrematureOptimization is the root of all evil."

Re:Depends on the discipline of the developers

[SunFan]

Performance should rarely even be a consideration until the product works. ...until the prototype works. The final product has to perform well. Otherwise, people will find trivial excuses to say it sucks and needs to be replaced, even if, on a whole, it is a decent product.

Re:Depends on the discipline of the developers

[FroMan]

Its a bad assumption that you are taking a terribley long time implementing. The only painstakingly slow part of the process should be the design of the interfaces. The rest of the code after that point should be modular enough to replace a poorly performing module, but the interface still exists.

This allows the best use of developer time by producing "cheap" code for most work. Then the "expensive" code can be written only in the cases where it is needed. Certainly there will be a certain amount of rewrite as the expensive code replaces the cheap code, but that only happens where necessary. Also, the second iteration through the development alone will usually help the situation because the problem domain is better known.

Re:Depends on the discipline of the developers

[Godeke]

So, which "performance" are we optimizing for? Memory footprint? Disk access? CPU utilization? Network utilization?

It turns out you rarely know which will bite your face off until you get a representative data set running. If you made the wrong choice, you probably made things *worse* than if you had opted for working code that could be re-factored from "easy to read" to "mostly easy to read and performs where it counts".

When you are working on server based solutions that will be hit hard, all of those could be your bottleneck: or none.

Re:Apropos

[SpaceLifeForm]

That is all good and fine, however, you didn't address the issue of environment variables, which your testing/QA people are typically afraid to change (or conversely, they do change and create a mess).

The best approach is to create an environment variable that defines the development environment, example DEVENV=DEV, DEVENV=TEST, DEVENV=IT, DEVENV=QA, DEVENV=PROD (or not needed for production at all), and then elsewhere (controlled and basically kept hidden from the testers or users), other environment variables are set based upon the value of DEVENV. Examples of these environment variables would be your PATH variables, ORACLE_SID, etc.

Then, the final problem is educating the testers what DEVENV means, and more importantly, why that one has to be correct and that they should not mess with any other environment variables.

If the testers can't understand that, you need smarter testers.

Tell me about the data, not some minor HW changes.

[_LORAX_]

Repeat after me, it's not about the platform unless the production units are seriously constrained in some way that cannot be replicated on development or testing. The one thing that has consistantly hamstringed projects in the past is not being able to replicate a dataset that comes close to replicating production bugs on development and testing. Without having a full production dataset in either development or testing you push something out that works only to find that the data causes some completly unrelated thing to break.

This is of course assuming that the software platform is adequitly compatible not to introduce stupid bugs because of diffrences between servers.

Re:Cost?

[dubl-u]

What if you can't afford a second machine to cover the "duplicate" test environment?

About 95% of the time I hear this, it's false economy. Most hardware is pretty cheap these days, and good developers are very expensive. It takes very little time savings to justify the purchase of new hardware.

In the few cases where it's too expensive to duplicate hardware, then you can fall back on careful profiling and simulation. For example, if you know that your production hardware has X times the CPU and Y times the I/O bandwidth, you can set performance targets on your development environment that are much lower. Or if you can't afford a network of test boxes to develop your distributed app, then things like VMWare or User Mode Linux will let you find some things out.

Of course, every time your tests diverge from your production environment, you add risk. A classic mistake is to develop a multithreaded app on a single-processor box and then deploy it on a multiple-processor box. So as you get cost savings by reducing hardware, it's good to keep in mind the added cost of inadequate testing.

Development environment: India ...

[Pathetic Coward]

test environment: well, none.

Re:To the other extreme with NDEBUG/assert()

[SunFan]

Why wouldn't you have asserts in the production code?

If the code were properly designed to fail gracefully in production, a failed assertion isn't very graceful.

Re:Environment meaning what?

[pete-classic]

Well, maybe they'll read my post and you'll be able to chill out.

-Peter

Re:To the other extreme with NDEBUG/assert()

[stoborrobots]

And a different question: Why would you compile with different options in the test and production builds? That would kind of invalidate the testing.

You shouldn't compile with different options between test and production, but you should do so (for things like -DDEBUG) between dev and test... developers need their extra debugging statements, and asserts, but they interfere with appropriate user interfacing in production.

As I see it, the steps are:
1. Develop
2. Test in production mimiced environment with -DDEBUG (still part of development phase)
3. Hand off to QA phase to test in production mimiced environment without -DDEBUG ("production version")
4. Release "production version" to production.

(obligatory...
5. ???
6. CODE NIRVANA! )

Yes...and yet, no.

[oneiros27]

There's another myth about projects -- the requirements were actually correct.

Odds are, if someone is rushing for you to get a project done on an unrealistic timeline, they haven't done their analysis of the project correctly, either. Having _any_ prototype up there can help drive the requirements analysis, so that you can figure out what needs to be changed.

But yes, then you scrap that entire thing, so you can do it correctly.

If you're making minor modifications to an existing system, then yes, you most likely wouldn't need a whole new prototype, but then again, you'd not be designing from the ground up, either, I would hope. [unless you get one some idiot manager who decides a new language is better, or you have to make some sort of fundamental internal change]

Oh -- and if an outside contractor asks for a couple of weeks of logs of the former systems, get rid of them -- a couple of _months_ will not identify cyclic trends that may be present. [especially when you work for a university, and it's the summer]

But be realistic of your goals for the project -- sometimes you're working to optimize on CPU, optimize on memory, optimize on disk usage, or optimize on the programmer's time. Until you get _everything_ running, you won't know which one will be the bottleneck. [although prior experience can give clues]

Continous integration

[maxmg]

We are developing J2EE applications using a continous integration server (currently anthill open source, but others are available). Ant is used for building, testing and deploying.
Now we have a number of environment-specific settings, for example database connection details, etc.
All environment-specific stuff goes into .properties files which are included conditionally by the ant script (based on a single environment variable or ant parameter). All of those properties files live in a directory conf/<environment name>, where environment name is either a developer's name, or "test", "production", "staging", etc. Each night, new deployment packages for each of the different deployment targets (test, prod, etc.) are built and made available through anthill. Some of those targets are also automatically deployed for the testing team every night, so the latest features are always available to be tested somewhere the next day.
Every successful build is tagged in CVS with an autoincrementing build number. When we have identified a release candidate, it is as simple as instructing anthill to (re)build a deployment bundle for a particular target with a specific build number. That deployment bundle (usually a .ear or .war) is then simply dropped into the production environment - remember that all the environment-specific settings are already included in that particular bundle. The benefit of this is that all environmental settings are maintained in the main source repository, the downside being that different packages exist for the different targets, but in practice that has not proved to cause any problems.
An additional benefit is that each environment's individual settings (including development machines) is always available to all developers for comparison and troubleshooting.
I guess the lesson learned is this:

• Automate your build!
• Extend your build system to include the environmental configuration
• Automatically build separate targets for different environments

hyperthreading

[marcovje]

One of the worst problems I had was related to hyperthreading. Turned out an 3rd party component that was a black box was threadsafe, but not multi-processing safe.

When I finally figured it out (of course it happened only once in days, had to write and tune exercisers for weeks to get it to occur within minutes), an update became available.
(we nailed 4 from the 6 spots in that patch already ourselves by then)