Ask Slashdot: Taming a Wild, One-Man Codebase?

New submitter tavi.g writes "Working for an ISP, along with my main job (networking) I get to create some useful code (Bash and Python) that's running on various internal machines. Among them: glue scripts, Cisco interaction / automatization tools, backup tools, alerting tools, IP-to-Serial OOB stuff, even a couple of web applications (LAMPython and CherryPy). Code has piled up — maybe over 20,000 lines — and I need a way to reliably work on it and deploy it. So far I used headers at the beginning of the scripts, but now I'm migrating the code over to Bazaar with TracBzr, because it seems best for my situation. My question for the Slashdot community is: in the case of single developer (for now), multiple machines, and a small-ish user base, what would be your suggestions for code versioning and deployment, considering that there are no real test environments and most code just goes into production ? This is relevant because lacking a test environment, I got used to immediate feedback from the scripts, since they were in production, and now a versioning system would mean going through proper deployment/rollback in order to get real feedback."

Comments

first thought:

[Tastecicles]

rectify the testbed lack.

'cos there's nothing more likely to cause immediate termination of your employment than a bit of rogue code taking down the bread of the business.

Test it first.

Re:first thought:

[Anonymous Coward]

The scripts are irrelevant if not ran on the real environment, the test environment would have to be a clone of the production environments. Good luck with that with the described environment! He could test each piece of the scripts in testing - which he probably does - but that only gets you so far and tells you that there's no typos.

Re:first thought:

[Anonymous Coward]

Yes, by all means test. Then, deploy your tests into production by mistake, like on Wall Street or something, LOL. Seriously though, testing is good and I unit test right from the start; but there are no silver bullets.

Re:first thought:

[Relayman]

There's also the issue of professional pride: If I were in his situation and I got hit by a bus (see the Six Feet Under pilot), I would want someone else to be able to pick up where I left off. I would also want this replacement to compliment the quality of my work. You need a test environment to do this.

Re:first thought:

[ILongForDarkness]

rectify the testbed lack, like Yoda is it. I agree you need a testbed. Heck run a few vm's on a workstation. If you can't build a vm to test something it shouldn't be deployed IMHO.

Re:first thought:

[Stiletto]

It's not a silver bullet, but lack of a test environment is sure to eventually cause disaster. It's by far the biggest problem mentioned above, even more of a problem than lack of version control.

Re:first thought:

[Anne Thwacks]

And spend a week or two reading http://thedailywtf.com/

Re:first thought:

[jekewa]

How does that meme go? Something like this, I think:

I don't always test my code, but when I do, I test in production.

Re:first thought:

[arth1]

testing is good and I unit test right from the start

Out of curiosity, what tools do you use to unit test bash scripts?

Re:first thought:

[rvw]

It's not a silver bullet, but lack of a test environment is sure to eventually cause disaster. It's by far the biggest problem mentioned above, even more of a problem than lack of version control.

I would start with a versioning system. That's a lot easier to get working. You could get that working in one day. And it doesn't need a test environment. Yes it should, but it's not a requirement. You can use the trunk as the production codebase. The big advantage is that you can rollback easily. You can even code on the server itself, and then update the codebase from there. No, not the wisest thing to do, but it's possible and probably a lot wiser than coding on the server without versioning. And use comments for each version update!

Re:first thought:

[MaxQuordlepleen]

It's unlikely that an inheriting developer is ever going to compliment the quality of your code. If he did, he'd never get the green light to trash all your work and start again from scratch.

Re:first thought:

[luis_a_espinal]

The scripts are irrelevant if not ran on the real environment,

Well, that's an oxymoron. Any program, large or small, is irrelevant if it never runs on the intended target platform. That's no excuse for having a test server, however feeble compared to production it might be.

the test environment would have to be a clone of the production environments.

A clone does not have to be equivalent in terms of hardware or data. A good example is a test db box for testing your SQL scripts. Such a box can have the exact same software, OS and patches, and with equivalent database configuration and schemas, but on lower-cost hardware and with a fraction of the data. As long as a test bench can provide a reasonable, objective measure of comfort of your code, that is all you need. You do not need an absolute guarantee (as there is never one anyways.)

Good luck with that with the described environment!

Yeah, because the task is so hard, he might as well give up, right, right, right? Let's do the paralysis-by-analysis chicken dance, shall we?

He could test each piece of the scripts in testing - which he probably does - but that only gets you so far

Which is better than nothing, and it is always better to carry tests, however little they might be on a test/sacrificial box than on production. It's not rocket science man.

and tells you that there's no typos.

No. It can also tell you that it will not do something bad, like deleting all records in a table, or initiating a shutdown, or filling up the /tmp partition. Better to detect such things on a mickey mouse test box than on the real thing. It might not catch bugs that are triggered by the actual characteristics present in a production environment, but it will most likely catch up bugs (annoying or fatal) that are not dependent on such characteristics.

Ideal? No. Better than nothing? Hell yeah.

Re:first thought:

[turbidostato]

"A clone does not have to be equivalent in terms of hardware or data."

Ahhh, developers...

Do you know what's left to the systems guys when you have finished (hopefully thoroughly enough) testing your code? The hardware and data (and the integration).

So, where do you think the hardest problems systems teams have to affront will come from?

If you thought hardware and data (and integration), you hitted the mark.

But they usually don't have the luxury to use the "hey, it works on my desktop" for a excuse.

Re:first thought:

[Anonymous Coward]

Don't need the Six Feet Under pilot. Hit by a Bus Syndrome has been known for decades. And not just in software. Only the stupid managers let it develop where they work.

Re:first thought:

[MrBigInThePants]

I would agree depending on the size of the domain and the time available.

Dropping tools and working on a test suite that should realistically contain 10,000 tests to get adequate coverage is not going to happen with a 1 person team for example. Especially if the test harness is difficult to work with.

My advice would be:

Good luck and god be with you...

Re:first thought:

[slim]

Googled, found shUnit.

I don't know how good it is, but it exists.

In practice, I wouldn't nowadays use shell to write anything complex enough to need unit testing.

Re:first thought:

[luis_a_espinal]

"A clone does not have to be equivalent in terms of hardware or data."

Ahhh, developers...

I've been a systems and network administrator as well (in addition to being a software engineer and developer.) I know how shit looks and works on both sides of the fence.

Do you know what's left to the systems guys when you have finished (hopefully thoroughly enough) testing your code?

Non sequitur (to the question at hand.) Of course in a real, well-planed environment you will have data and hardware (not just the boxes, but the network, switches, proxies and firewalls) equal or equivalent to production as much as possible, with segregated dev, test, UAT and pre-production environments.

However, the guy is a one-man shop, and he needs to get something, anything up pronto. For his immediate requirements, and with the scarce resources he has, he does not need (nor he is capable of implementing) a full-blown dev/test environment.

Anything is typically better than nothing. Ergo, as I said before: the clone does not have to be equivalent in terms of hardware and data. It is called managing expectations combined with ROI and make shit work with what you have instead of falling into paralysis-by-analysis. That's why we get paid the big bucks (and this is the most important thing devs and sysadmins typically miss.).

The hardware and data (and the integration).

1. Non sequitur (see above.)

2. Divide-and-conquer. Meaning, worry first how to isolate and test your code away from production first and foremost. Later, focus on hardware/data integration testing and troubleshooting. You cannot effectively do the later without the former.

So, where do you think the hardest problems systems teams have to affront will come from?

If you thought hardware and data (and integration), you hitted the mark.

Non sequitor. The guy in question is a dev/system guy, wearing two hats. He needs to separate both hats in order to get shit under control.

But they usually don't have the luxury to use the "hey, it works on my desktop" for a excuse.

Bleh, you can also see the poop on the other side of the fence (I've been on both sides). There is nothing more infuriating that a systems guy telling a developer that the problem is in his code even though the goddam application logs, sys logs, and everything under the heavens points to a misconfigured NIC card (running half-duplex when everything else runs full duplex), telling the dev to go change the code, code that the mother*f* has never seen nor knows what it does.

There is crap on both sides of the fence, each sitting comfortably in its own soap box.

git

[Anonymous Coward]

git

Re:git

[ThorGod]

git

Yes!!! Create git repos of all those various parts on some central git server. Create backups of those repos periodically, like a sane person...

Git really doesn't require a ton of understanding to "just start using git" competently. It's not going to trash whatever you have in place; it's mathematically proven to *not* lose data.

Also, freaking set up a dev server already! (That's like 2 machines, or a private, 3rd party git repo (bitbucket is what I use) and a dedicated test/dev machine).

Re:git

[Anonymous Coward]

Mod this up. Nothing works as well as git, as long as you're not dealing with binary blobs.

Re:git

[Stiletto]

Yes!!! Create git repos of all those various parts on some central git server. Create backups of those repos periodically, like a sane person...

The great thing about git is that there is no need for "central servers" or other such infrastructure. Just back up the current repositories wherever they are located.

Re:git

[Barryke]

Smartgit I can recommend.

Re:git

[Mordok-DestroyerOfWo]

I don't think I could live without git. I used it at home to keep changes to family photos when I (attempt to) enhance them with Photoshop. At work to keep my vim scripts, and bash files consistent between machines. Needless to say, all of my code is also there too. It's dead easy to setup and tutorials abound.

Re:git

[gorzek]

Count me in with the others recommending git. I'm using it in a very diverse environment, where different teams have different platforms, workflows, and deployment strategies. I convinced them all to use git. They are extremely happy with it, and they love the flexibility it offers them over their old (or nonexistent) systems.

Re:git

[Ken_g6]

He's using Bazaar, which is a lot like Git. The main problem I found with Bazaar is that it's S...L...O...W...! Git does things almost instantly.

Re:git

[GoogleShill]

... it's mathematically proven to *not* lose data.

I love git and use it on a daily basis, but you can't mathematically prove that it won't lose data. It is written by humans, and I have encountered bugs in it. You also still have to deal with manual merges, which are error prone. I've also had my local repo get in weird states that are very difficult to get out of. When this happens, I always copy out all my changes because I'm afraid of losing anything.

Re:git

[Anonymous Coward]

Neither does bazaar, mercurial, fossil or veracity. Fossil and veracity also have issue tracking.

You should have a central server, if you have more than one developer. Otherwise you'd need to connect to every developer to push your changes. No need for central server just means they don't need to be connected to that all the time (you have the history on everyone's machine, it gives safety, speed and ease of use), and they can make their own branches and push when ever they are ready.

Code versioning and deployment?

[MetalliQaZ]

I don't understand how code versioning has to be coupled with deployment? You have no test environment, as you said... so just make releases and deploy them manually. Since you are going straight to production, you had better be there in person to roll it back if you screwed up. Right? So, SVN should be all you need...

Re:Code versioning and deployment?

[dgatwood]

Git is a cleaner model in a lot of ways. In particular, the fact that you have a local copy of the entire repository makes it easier to roll back mistakes you make while editing the code. This isn't always important, but if you decide you're going to do a significant rewrite of some piece of code (and in particular, if you are ever remote while you're doing so), it helps a lot.

Re:Code versioning and deployment?

[Skewray]

I don't understand how code versioning has to be coupled with deployment? You have no test environment, as you said... so just make releases and deploy them manually. Since you are going straight to production, you had better be there in person to roll it back if you screwed up. Right? So, SVN should be all you need...

I used to, as a single programmer, use SVN, but I found it nothing but a burden. It left files all over the place, and was really not convenient when no interlocking with another programmer is needed. Now I just make a tarball of everything at obvious breakpoints and store it away.

Re:Code versioning and deployment?

[inKubus]

Here's what I did, pre-git:

Create svn repo, e.g. svn.company.lan/systems
Create structure ./trunk, ./branches, ./tags
Create a directory for each hostname e.g. ./trunk/sql1, ./trunk/web1, ./trunk/web2, etc.
Then you can svn import configuration directories on the host into the repo, e.g. svn import svn.company.lan/trunk/sql1 /etc
Then check out svn co svn.company.lan/trunk/sql1/etc /etc
From that point forward if you make changes locally you can svn ci OR you can make them externally (i.e. in a test environment) then svn up to update your local conf
I keep the same directory structure, so if I have some tomcat conf like /opt/jira/tomcat/conf it will be in svn as svn.company.lan/trunk/web1/opt/jira/tomcat/conf

With some scripts, I automated the process and since then it's been really easy to maintain. I understand that cfengine is quite a bit more complex and can do a lot more, like verifying your configuration and that sort of thing, but for a small shop this is good enough to prevent Oh Shit moments with minimal extra work and almost no maintenance.

Need to make a change? First, check in to make sure repo has latest version. Make your changes, restart your daemons..if it works, check in. If it doesn't work you can keep working or svn revert back to the previous version.

With git, you'd have a similar thing but the repo would be local and you'd have to find a way to back it up, or you could have something like stash running to be a central hub. DO NOT use github to store configs out of habit, because sometimes conf files have private keys and stuff and it is extremely likely that github will be targeted by crackers at some point. Svn is real easy to set up on a random utility server or even a workstation...

Re:Code versioning and deployment?

[amicusNYCL]

Git is a cleaner model in a lot of ways. In particular, the fact that you have a local copy of the entire repository makes it easier to roll back mistakes you make while editing the code.

What does that mean? It's trivial to either check out or export any part of the repository from SVN. What does git bring to the table that I don't already have with SVN?

Re:Code versioning and deployment?

[Anonymous Coward]

spoken like someone who hasn't really used modern scm...

With svn worse is not better, worse is just worse.

Re:Code versioning and deployment?

[arth1]

I do something similar with svn, but the main problem with this is that Subversion doesn't preserve Unix ownerships, permissions, acls or attributes.
A secondary problem is the .svn directories - some directories are parsed automatically by various systems and all files and folders there are acted on. Then a version control system needs to be external to the directory structure.

Re:Code versioning and deployment?

[LordThyGod]

Bazaar does the same, or at least can.

Re:Code versioning and deployment?

[dna_(c)(tm)(r)]

What does git bring to the table that I don't already have with SVN?

A lot. (as does bzr, mercurial or any other distributed versioning system)

• You don't need a central server (but you can have one).
• You don't need to have a network available to check in changes.
• You don't need to have a network available to roll back or switch to another branch. E.g. you could edit /etc/init.d/networking break stuff and roll back...
• It is really fast - it is mostly local stuff.
• ...

Re:Code versioning and deployment?

[amicusNYCL]

Hmm. I prefer the central server, actually, because that's only one place I need to back up. I'll look into git in the future though, there are plenty of problems with the Collabnet Subversion Edge/TortoiseSVN system that we're using that cause us to want to look into alternatives. For example, occasionally there will be a problem during commit where it will lock a file, stopping me from updating or committing further, but when I try to unlock it it just says that nothing is locked. The merge tool sometimes insists on keeping the old bugs in there and tossing our fixes too for whatever reason.

Re:Code versioning and deployment?

[turbidostato]

"but the main problem with this is that Subversion doesn't preserve Unix ownerships, permissions, acls or attributes.
A secondary problem is the .svn directories"

The main problem is that you've never been a C programmer.

Or else you'd have ingrained the notion that any source needs to be configured, made and installed like, *always*, while sometimes one of the stages can be just a 'noop'.

Permissions, ACLs and attributes is a matter of a 'make' stage; getting rid of the .cvs dir is a matter of an install stage.

I.e.: set the proper perms means running a 'make.sh' that sets them properly; getting rid of .cvs/ that of an rsync script that moves data from your sandbox or your build subdir to the real /etc dir.

Of course, nowadays you can do that within a single tool, i.e. puppet, but still the configure, make, make install stages are buried in there.

Re:Code versioning and deployment?

[turbidostato]

"You don't need a central server (but you can have one)."

You *do* need a central server. Systems administration is not a matter of a bazaar but of a cathedral.

"You don't need to have a network available to check in changes."

You *do* need to have a network available or else your changes won't hit neither your staging nor your production environments.

"You don't need to have a network available to roll back or switch to another branch. E.g. you could edit /etc/init.d/networking break stuff and roll back..."

You *do* edit /etc/init.d/networking right on the production servers?

"It is really fast - it is mostly local stuff."

Add a master CD to git and then tell me that again (yes: if it really is a master CD, then it is a source for all practical means and purposes and it does belong within your SCM system).

Re:Code versioning and deployment?

[theshowmecanuck]

So when your workstation crashes it takes out both your working code/directories and the repository? Very convenient. Much simpler to crater that way. I like a different machine to keep code on. One that gets backed up regularly.

Re:Code versioning and deployment?

[arth1]

The main problem is that you've never been a C programmer.

*Raised eyebrow*
If anything, you're the newbie here, talking about "configure, make, make install". Writing a system that breaks in five years because compilers and headers move on while your tool doesn't is as stupid now as it was in the days of xmkmf. Same shit, different wrapping. Sure, macro languages and expert wrappers make things easy to write, but time consuming and incredibly hard to troubleshoot compared to an open system that doesn't depend on compilers and headers not moving on.

Different systems need different installation procedures. Your system needs to be flexible enough to let the installing user write and modify installation routines, and not depend on scripts that may (read: will) be outdated and near impossible to fix without the original and deprecated software base. Sysadmins (who, by the way, like me, often started out as C programmers) don't like to wade through 80k of undocumented abstracted script code to find out that basically, what it does, can be replicated with a six-line makefile and changing an -I.
Abstraction for the sake of abstraction is the surest sign of an incompetent coder.

And I'm sorry, while we do indeed use installations, we do not use it for everything. If nothing else because it may not be feasible. You may want to set a new password, for example, but you may not want to use a script to do that in plain text, nor copy the same salt from a different server.
Or you may not have the leisure of doing it - emergencies happen, and not always due to bad planning. So fix that /etc/resolv.conf pronto, and then check it in so you have a history of what was done and why.
In some cases you may not even have a staging environment or ability to run your special install software or ability to keep the code on a different host. There may not be usable networking on a legacy system. If you want to be able to back out and recover in reasonable time if problems occur, your best bet may very well be a revision control system that works locally. Heck, even rcs is better than a push system that can't be used, or scripts that fail to handle exceptions to interactive commands.

And sometimes you just trust your fingers to do work more than you trust a script. Even if it runs flawlessly in staging, it may still introduce a risk you don't want and may have a hard time backing out of.

Re:Code versioning and deployment?

[Anonymous Coward]

fossil and veracity have locking (if that's what you need), but they do not need a central server either, but of course you can and should have one, if you have more than one developer. Fossil and veracity also have issue tracking and wiki, which are version controlled.

What these new generation version control systems bring is, that you do not need to have a connection to the central server all the time and you can create versions even when you are away. Then when you get the connection, you can push your changes and all the history remains, unlike with SVN, with which you need that connection and then you can't make those versions and, when you get the connection, you can only commit this one big change and try to remember what you have done.

I personally haven't yet decided which to start using, Fossil or veracity. I'm using git, but i want an easy way to track issues also, and i do not need a cenrtal server, because i do this mostly on my own.

Re:Code versioning and deployment?

[dna_(c)(tm)(r)]

The point is that - while in most professional environments you do want some central place to store the history of your code, you are not depending on that.

The central server might be offline, the network might be down, you're on holiday at a tropical island without internet, the server crashed and will never be restored, the company you work for goes bankrupt - but you can still access the history, check out older versions, check in new stuff. And if/when the revered central server is finally available again you can push these changes..

If need be you can even merge stuff between individual devs without the central server being available. I use that sometimes for small projects to sync stuff between laptops/desktop or in the initial stages of a project when experimenting a lot and want to share stuff with somebody else. Instead of copying a zip-file clone a repository and you can easily merge/branch stuff...

Re:Code versioning and deployment?

[dna_(c)(tm)(r)]

Oh boy. Language semantics :)

You apparently need a central server in your particular situation according to your perception. But using Git does not make it necessary to have a central server (nice for tiny projects, experiments, initial stages of a project) however the main point is you can be offline when you do have a central server, you're not depending directly on the central server most operations are local and changes are available locally.

And since you do not need that central server, you can start using small local repositories for tiny projects that would go un-versioned otherwise.

If you look at the original question you see that system administration in this case is very bazaar like ("Taming a wild, one-man code base"), traditionally it was like that too: a sysadmin hacking together glue-scripts, writing terse perl scripts nobody else dares to touch... I agree you have to move away from that. And yet again, a distributed VCS offers more ways to do that, you can have a workflow with a gatekeeper that allows only one person or a few people to release changes to the central repository from which others update or to the one from which test/production builds are done. Try that with SVN.

Check in changes (actually commits) can be done often without involving the server, pulling changes from the central repository and pushing the result of the merge back, need networking. But I suppose you could even replace that with a scenario involving sneakers and a USB stick when sneakernet is the only connection available. Try that with SVN.

Edit /etc/init.d/networking sure, why not? Before they are in production, or offline for maintenance. It was just an example for breaking networking (could be /etc/interfaces or whatever). I was doing this on a test setup with VM guests connected via VNC (so no, not production servers that are online).

The master CD is hardly a typical situation. And without more information, in general it is generated from source files and should be reproducible from that and therefore does not belong in the VCS.

Re:Code versioning and deployment?

[jgrahn]

Hmm. I prefer the central server, actually, because that's only one place I need to back up. I'll look into git in the future though

Huh? There is only one place to back up with Git too! And in fact the usual way to do backups is to 'git clone' that one and 'git fetch' from it regularly.

The only complication is that in the other repositories you must remember to 'git commit' *and* 'git push', or you'll just add versions locally; they don't reach the central repo.

Re:Code versioning and deployment?

[jgrahn]

I don't understand how code versioning has to be coupled with deployment? You have no test environment, as you said... so just make releases and deploy them manually. Since you are going straight to production, you had better be there in person to roll it back if you screwed up. Right? So, SVN should be all you need...

I used to, as a single programmer, use SVN, but I found it nothing but a burden. It left files all over the place, and was really not convenient when no interlocking with another programmer is needed. Now I just make a tarball of everything at obvious breakpoints and store it away.

As a single programmer, I find version control invaluable, and wouldn't tolerate such a working environment. I haven't used SVN so I cannot comment on it, but both RCS and CVS have been universally available since the 1980s, and don't "leave files all over the place". I quickly learned to ignore the RCS/ and CVS/ subdirectories.

A modern version control tool like Git lets the single programmer name and describe her work progress; check what she did yesterday; review changes before commiting them; branch for experimental work; work from multiple computers or directories at will; easily keep backups; ... and many things I forget.

It's too late

[Antipater]

Given the situation you describe, it won't be long before the whole system falls into corruption. Your only hope is to save two lines from every script on a USB stick, then flood the rest.

Re:It's too late

[Anonymous Coward]

and, 7 lines of every script you deem "clean"

Re:It's too late

[psmears]

Your only hope is to save two lines from every script on a USB stick, then flood the rest.

Is that what's known as an ark-hive?

Simple answer

[girlintraining]

My question for the Slashdot community is: in the case of single developer (for now), multiple machines, and a small-ish user base, what would be your suggestions for code versioning and deployment, considering that there are no real test environments and most code just goes into production ?

The simple answer is, "Whatever works best for you." You're the only developer for these projects. Unless your manager is giving you direction on a specific process or requirements, it's your ball game. You know how you work best -- pick your tools accordingly.

Re:Simple answer

[tool462]

Pretty much. This is a very hard thing to answer in general terms. With only one developer, over-engineering the system can be very costly. You'll spend more time maintaining the dev/test/release environment than the actual code itself. But at the same time, some tools and scripts can be absolutely critical to the business and a bug could be disastrous enough that it warrants all the overhead of a more formal dev environment.

What you do is going to depend a lot on the exact details, and may not even be consistent across all the code you support. There's only one of you. Make sure you spend your time on the most critical pieces.

The only thing that I would consider absolutely mandatory is proper revisioning and branching so that you can recover quickly when something goes wrong.

Re:Simple answer

[Barryke]

Even using Git without merging or branching would be worthwhile.

Re:Simple answer

[Sigg3.net]

Which part of asking for advice is so hard to understand?

Yes, he knows all the details, but we may know a method that is more successful in this scenario.

  (Actually, I wouldn't, since IANAP. But I'd _still_

A few things

[jlechem]

1. Buy or get a machine to host SVN for version control. I work on my wife's company website and some basic management tools. SVN has saved my bacon on multiple times where I thought I had lost some code.

2. Get a pre-production server and test your code! Sounds like you're living in the wild west and that shit flies until something goes horribly wrong and you're the guy who gets blamed.

Re:A few things

[jellomizer]

If you can't get the hardware. Try to Virtualize a Test Envionment with like VM Ware or Virtual Box.
At least you have something to play in before it you put it out on the open.

Re:A few things

[Anonymous Coward]

I am in the same situation (sort of) and found SVN to be overkill. Mercurial is a good alternative to SVN in this case because you don't need to set up a separate server just to host the repo. You can use something like BitBucket (which offers free private repos) to backup online if your company policies allow.

Re:A few things

[KingMotley]

Not sure why you think you need a separate server just to host the repo. Just host it on the same machine.

Sure at the office we have a server that hosts the repo, but at my house, I have the repo running on the same machine I develop on. Of course the repo is on a RAID-6, and my local copy I develop on is on a RAID-0, but I didn't need to buy another machine just to host the repo.

Re:A few things

[Anonymous Coward]

For goodness sakes, this isn't the 90's. Pick a good DVCS like Mercurial or Git, not a centralized one like SVN.

Re:A few things

[javawocky]

I am fond on SVN and we use it at my current company - badly. If you have a choice I recommend git for a number of reason. 1) Git is way faster (on the command line) 2) Git has a single folder '.git' to do its bidding at the top of the project tree ( no .svn folder in every folder) 3) You don't need a central server. Each workspace is interested in itself and you tell it about other repositories. Typically you have an Origin for your code which is kind of like a central server but is just another folder. 4) Git commands and errors are way clearer than SVN - IMO You probably want to separate the tool out from the deployment plan. SVN or GIT can work equally as well for deployment, but you need a good plan. When it is well thought out golives are fun - not stressful. Git is cooler, they even wrote a song about it http://www.youtube.com/watch?v=qchPLaiKocI

No real change

[chthon]

You can still change everything in place. Then you can run the script and get feedback. When it works, you commit. When it doesn't, you remove the problem, check and commit.

Or you can make your changes, review them and commit them, then do a run. When you have a problem, you commit again.

It is not because you use a versioning system that you need extra formality. You can still work the way you used to, but now you have an extra safety measure due to the versioning system.

Using trac is a way to better organise your problems. The main thing I can say about using trac effectively is that you always need to have a browser window open on it, and when you have an idea, or notice something, or have problem, then enter it immediately. Afterwards, take your time to look at new and open problems, classify them and process them.

Re:No real change

[cowboy76Spain]

And a few days after you put the changes in production and nothing has burned, make it a tag.

Better yet, make it a tag before putting changes in production (TAGbeta) and a few days later (TAGrelease). Tags are cheap.

Re:No real change

[crontabminusell]

Thanks for saving me the effort of typing that. =) What you wrote, chthon, sounds like the sanest application of version control to the OP's environment. And if another dev gets added to the mix down the road, the version control system will already be in place.

proper deployment/rollback

[turbidostato]

You say that "now a versioning system would mean going through proper deployment/rollback in order to get real feedback."

But then, no, it wouldn't.

Storing your code on a versioning system doesn't mean but that: that you store your code in a versioning system, nothing more, nothing else.

I'm starting to be an old fart so you can believe me when I tell I've already been in your position.

Back then I used CVS and it didn't change my deployment procedures in the slightest -only that I had all those scripts in a single convenient place and I could look in past history when I found a regression or I wanted to look for the way I did something in the past.

The most naive approach is you just got working just the way you are doing now, only that when you are confident on a script/set of scripts you check them in for posterity. You mainly develop in your own desktop and you push your scripts to the servers with an rsync-based script. A bit over this, you use a CM tool (say, puppet) so instead of pushing to the servers you push to the puppetmaster and then run a `puppet agent --test` on the servers: that way configuration becomes code and therefore, repeatibility.

It allows for almost a novel but the basic idea is just the same: SCM is SCM is SCM; nothing more, nothing less.

Re:proper deployment/rollback

[turbidostato]

Oh, by the way, you really should listen to those that tell you *need* some development environment.

Again, I've already been there, so I know you pain: even for the silliest development the developers will have their development environment but for us, systems people, it's expected that everything just fits in place at first try, no second chances. Of course, next heavy refurbish will be near to impossible because while being a good professional allows for more or less "clean" kaizen-style development, anything a bit dangerous is an almost impossibility because of lack of test environments (with luck, next "heavy test window" will be in three/four years when all the servers are decomissioned and new ones come in place) but that's the way it is, take it of leave it.

The good news is that, while not a panacea, virtualization, even at desktop level (you surely need to have a look at vagrant[1]) allows for a lot of testing, impossible in the age or "real-iron only".

[1] http://www.vagrantup.com/

Re:proper deployment/rollback

[SQLGuru]

Another benefit of a versioning system is that you don't have to keep large chunks of commented out code. If it needs to go, delete it. It's in the history if you need to go back to it. This alone will clean up most of the spaghetti that a one-coder shop faces.

Stay with what you have.

[Anonymous Coward]

You are going to put yourself through a heap of misery changing over to something you have to learn all over again. Best stay where you are with Python.

Subversion.

[Anonymous Coward]

Check it into subversion. You can get your build/packaging tools to embed the svn revision into the artifact.

For all the git lovers out there - r564 is so much easier for a human deal with than a large hex string, and most git advantages don't really apply a single developer.

Re:Subversion.

[turbidostato]

"For all the git lovers out there - r564 is so much easier for a human deal with than a large hex string"

Because `git tag r564` if so much of a burden, isn't it?

Rename the files f1, f2, f3, etc.

[Maximum Prophet]

Quick! Rename all the files f1, f2, f3 etc, rename all the variables i1, i2, i3, etc and remove all whitespace.

Keep a translation sheet on you at all times. Suddenly, you're irreplaceable.

(:-) for the humor impaired. This is actually a riff on a joke from WKRP, when an engineer said he was replacing all the color-coded wiring with black wires for job security. (B.t.w. the engineer was played by one of the writers of the show)

Re:Rename the files f1, f2, f3, etc.

[Relayman]

Score:5 Funny (No mod points today, sorry. It looks like I have to comment to get mod points.)

Re:Rename the files f1, f2, f3, etc.

[HornWumpus]

irreplaceable == unpromotable.

Granting it's a one man shop, not likely to be much in the way of upward mobility anyhow.

Re:Rename the files f1, f2, f3, etc.

[Medievalist]

I wasn't familiar with the WKRP schtick, but I actually worked for a DOD subcontractor and saw a guy wire an entire harpoon missile controller using nothing but blue wire. It was for the test environment, of course, not for combat use. Hundreds of individual wires, all pale blue... most of them would be printed circuits in the real controllers.

That is one of the experiences (building out the Internet was another) that convinced me there is no such thing as a non-trivial test environment. You cannot simulate the interactions of very large amounts of completely normal human stupidity. You can't even really get close.

Re:Rename the files f1, f2, f3, etc.

[SolitaryMan]

What is "WKRP"?

Re:Rename the files f1, f2, f3, etc.

[Anonymous Coward]

WTF? You need to brush up on your 70's/80's TV shows. WKRP was awesome. http://en.wikipedia.org/wiki/WKRP_in_Cincinnati

Re:Rename the files f1, f2, f3, etc.

[Anonymous Coward]

Herb? Is that you?

Re:Rename the files f1, f2, f3, etc.

[Maximum Prophet]

Unpromotable is not necessarily a bad thing, i.e. The Peter Principle http://en.wikipedia.org/wiki/Peter_principle

Ideally, you want to find your niche in the world where you can be most happy. Getting more responsibility than optimal is a bad thing, leading to stress and early death. Check out long term prospects of lottery winners to see what too much money will do to you.
A steady income stream, with no chance of promotion may or may not be exactly what someone needs to be most happy.

Re:Rename the files f1, f2, f3, etc.

[HornWumpus]

Assuming your fine with unpromotable, how are you with 'must be on call 24/365'?

I've lived it, the one thing worse for your stress then taking a promotion you don't want, is having 'them' put the worst air thief in the organization into the position to punish you for not taking the promotion. Now you have to manage the manager and the team without any authority just as self preservation.

Re:Rename the files f1, f2, f3, etc.

[Maximum Prophet]

Assuming your fine with unpromotable, how are you with 'must be on call 24/365'?

I've lived it, the one thing worse for your stress then taking a promotion you don't want, is having 'them' put the worst air thief in the organization into the position to punish you for not taking the promotion. Now you have to manage the manager and the team without any authority just as self preservation.

That's when you take a lateral, a job that you can do, but pays the same, either within the same company, or a new more exciting employer.

Going from a bad job to a higher position that you can't do, isn't healthy. But, if you are close to retirement, and can stick it out, the promotion might get you there quicker.

Basically, when faced with "Damned if you do", "Damned if you don't", choose the option that pays the most. However, if and only if you are comfortable, think long and hard about a job that pays more, but makes you unhappy.

Re:Rename the files f1, f2, f3, etc.

[HornWumpus]

I'm long gone from there. They still call my simulation results database loader 'the firehose' largely because they can't match it's performance with newer technology. It must suck that the fastest database for results analysis is MS Access in 2012.

The Story of Mel is instructive here.

[Anonymous Coward]

Most of you whom have seen this may have read it in the Jargon File. It's relevant. The short answer is "you don't":

The Story of Mel, a Real Programmer

This was posted to USENET by its author, Ed Nather (utastro!nather), on May 21, 1983.

A recent article devoted to the *macho* side of programming made the bald and unvarnished statement:

Real Programmers write in FORTRAN.

Maybe they do now,
in this decadent era of
Lite beer, hand calculators, and "user-friendly" software
but back in the Good Old Days,
when the term "software" sounded funny
and Real Computers were made out of drums and vacuum tubes,
Real Programmers wrote in machine code.
Not FORTRAN. Not RATFOR. Not, even, assembly language.
Machine Code.
Raw, unadorned, inscrutable hexadecimal numbers.
Directly.

Lest a whole new generation of programmers
grow up in ignorance of this glorious past,
I feel duty-bound to describe,
as best I can through the generation gap,
how a Real Programmer wrote code.
I'll call him Mel,
because that was his name.

I first met Mel when I went to work for Royal McBee Computer Corp.,
a now-defunct subsidiary of the typewriter company.
The firm manufactured the LGP-30,
a small, cheap (by the standards of the day)
drum-memory computer,
and had just started to manufacture
the RPC-4000, a much-improved,
bigger, better, faster --- drum-memory computer.
Cores cost too much,
and weren't here to stay, anyway.
(That's why you haven't heard of the company,
or the computer.)

I had been hired to write a FORTRAN compiler
for this new marvel and Mel was my guide to its wonders.
Mel didn't approve of compilers.

"If a program can't rewrite its own code",
he asked, "what good is it?"

Mel had written,
in hexadecimal,
the most popular computer program the company owned.
It ran on the LGP-30
and played blackjack with potential customers
at computer shows.
Its effect was always dramatic.
The LGP-30 booth was packed at every show,
and the IBM salesmen stood around
talking to each other.
Whether or not this actually sold computers
was a question we never discussed.

Mel's job was to re-write

Re:The Story of Mel is instructive here.

[gorzek]

I never, ever get tired of this story.

Re:The Story of Mel is instructive here.

[Anonymous Coward]

Mel Kaye should cock slap all the current breed of VMed language weenies and teach them a thing or two about efficient coding.

DVCS for the win

[AwesomeMcgee]

Create a git repository on 'production' and then a fork on your development machine. (Or a fork on a test machine would be better really, which you then fork to development)

Do your development, checkin and then pull to test, execute there, if all goes well, pull to prod and execute there.

Revision Control and Deployment

[MrSenile]

Before it gets out of hand, I'd look to set up four things.

1. Set up a proper split environment. Even if you don't have the hardware for it, set it up in such a way that when the hardware becomes available, you can move it appropriately. That being, a standard dev -> qa -> stress -> prod infrastructure.
2. Set up a good revision control. I've started to really enjoy using GIT for this, as there's other software like gitolite that can give you fine-grained access control to your repositories. However, feel free to use subversion or any other well contained revision control platform.
3. Set up a good method for deployment. My suggestion? Try puppet. It's free, and it's powerful, and if you get it configured, adding new systems to it is exceedingly easy to do.
4. Packaging for your deployment. If you are installing a bunch of software (scripts, job control, etc) package it and give it a revision, then it's easy to upgrade systems with the 'new package', or revert it to the 'previous package' instead of having to manually copy around files or (re)editing them.

Hope that helps.

Re:Revision Control and Deployment

[Anonymous Coward]

Second git recommendation. Bazaar has almost no tools for inspecting the history, no bisect, and no rebasing is allowed. As a result it is almost completely useless for analyzing existing code.

Re:Revision Control and Deployment

[cc1984_]

Please, if you're going to troll try a little harder next time.

Re:Revision Control and Deployment

[Anonymous Coward]

That plugin:
  * is out of date (does not work with the current version of bzr).
  * is completely undocumented and unsupported.
  * doesn't even do the things you need from rebasing.

The situation with the bzr-bisect plugin is exactly the same. Unsupported and broken. It looks like it does bisection only if you don't actually understand the purpose of bisection. In fact that applies to bzr as a whole. It looks like a version control system only if you don't understand the purpose of one.

Hmm

[jameshofo]

Yea that's interesting actually, I just ran into this myself. We're putting a project together and when something breaks I end up doing small fixes and losing the changes across deployments (we only have 3 active) so its very small. But I feel your pain, I'm not totally convinced that a full SVN system is necessary but once you break down the problems it likely is. Given your closed infrastructure you may want to consider adding some phone home features to your scripts, something intelligent enough to auto update smoothly in an automated way or manually. Make things easy for yourself so they're not difficult to work with and you will be encouraging yourself (and others) to use it.

The absolute best advice I can give is keep it simple, there are a million different ways to do it, try not to do a massive migration of everything all at once or you may find out later that some minute bug is hindering everything you do.

Lastly plan what you want it to look like and how, it will save you weeks of work.

Git + Vagrant + Puppet

[Anonymous Coward]

Use Git for source control.

Use Vagrant to create virtualized testing environments (via headless Virtualbox) that you can ssh into, develop in, and test... all running directly within your laptop. Use Puppet or Chef to create recipes for all of your servers, and you can virtualize all of them in a "pretend" network of virtual machines. All of that can be checked into Git too.

Store your central Git repositories in Github or some other reliable place (you can stick them on one of your own servers too). Code in your virtual machine, commit, and push up to the central Git repo. Then pull it down to your live servers to automatically update them.

You can use Puppet in server/client mode to automate the deployment of server configuration changes out to your live machines also.

And if you want to get REALLY fancy, just throw a third set of machines in there and use that as your "staging" environment, where changes go after your virtualized environment, but before your live environment (mimicking the live environment as closely as possible).

Re:Git + Vagrant + Puppet

[Anonymous Coward]

This.

Perforce and a VirtualBox/VMware

[scorp1us]

I know there are plenty of OpenSource tools out there, but I still prefer perforce. Also, recently (as of February) Perforce opened up its 2-user license to 20 users/20 workspaces! This is fantastic news!

Check in your mainline (or migrate) to perforce under /depot/mainline
Integrate to a non existent branch /depot/testing/VERSION, and check that in.
Integrate /depot/testing/VERSION to a non existent branch /depot/release/VERSION, and check that in.

Now with P4V, moving changesets from mainline to testing is as simple as drag and drop. Then move changesets from mainline to, then testing (and the changes found in testing) to release, and drag those back to mainline. (Dragging is in 'integrate' step) You now have come full circle and you have 2 places where you can make changes and have a release snapshot.

Now, get VirtualBox because it supports snapshotting. Set up perforce on that and take a snapshot. Then sync from perforce, run your tests, and deploy as needed. Then revert the snapshot to after you installed perforce.

Then you can make packaging/deployment scripts that only work on release branches.

Documentation

[Hjalmar]

Yes, set up a test environment. And implement some kind of versioning system, even if it's just "cp current_code old_code". You should always be able to fall back if you have a botched deployment.

But one of the best things you can do is to start writing documentation. I like to write my documentation assuming it will be my replacement reading it, and so I try to include everything. Justify every unusual implementation detail, explain why each task was down the way it was. List bugs, and any code you had to write to work around it. The best part of documenting your project will be that as you work through it, you'll find things that no longer make sense and make them better.

Any system, just use it.

[niftymitch]

If you work on a single server install RCS. You only need to
Learn ci & co to start.

If you work on many boxes you need a network friendly tool.
The obvious ones are git and mecurial (CVS too).

Simple cp works too.

More important may be version tags and date time hints in the scripts.

Git.

[blackcoot]

A great deal of the version wrangling you are facing is best done with a tool like Git.

The bigger problem (development discipline) is much harder to fix.

Chef & Jenkins

[terbeaux]

You want something to track changes, deploy changes, and test software. Bazaar will track your changes.

Chef is open source infrastructure management. The central server maintains a searchable database of your nodes and all of the scripts (recipes) that run on them. The nodes query this database and run the scripts that they are supposed to. This is similar to your environment now. You can also check your chef-repo into scm. This allows you to mess around with production and only commit back into scm when you are fairly certain that it works.

Jenkins has a similar setup but each node is ostensibly there to build and test software although we have used it for deployment and integration testing.

Chef & Jenkins can definitely help in deploying code and maintaining your infrastructure but you will need to take responsibility for testing your code somewhere along the process whether it be with on-commit with Jenkins or on deploy with unit or other tests. I definitely feel the value after investing time to learn these powerful tools.

Re:Chef & Jenkins

[undeadbill]

Definitely Jenkins for code pushes. Not only can you decide how to push the code (even build and deploy through RPM), you can also use the Jenkins interface to manage testing and QA as well. Build can be distributed through virtual machines, and automation can be tied into something like chef or puppet. That includes cleaning and restarting virtual host images during testing, automating deployment milestones, etc.

Also, the other benefit of using Jenkins is that you can manage future contributors through the Jenkins management interface, limiting their access privileges while preserving a lot of flexibility. This means you can start out someone with a few privileges, say for QA validation only, and then add more as they show they are trustworthy.

I have a bunch of personal code that I tote around

[Omnifarious]

I keep it in a Mercurial repository and use symlinks into the repository to deploy it. I also make free use of Mercurial's subrepo feature for tools that others wrote that are not yet found as packages on the Linux distributions I use.

Yes, there is still a testing issue. For most of this code it's not a big deal because I'm the only user. I test it as I write it with a few simple hand tests and then it's good to go.

If I were doing this for something where the code mattered to other people I would just add unit tests for various subsections as made sense. I would also start sectioning off the tools and making them into separate repositories of their own. I'd also make much sparer use of the sub-repo feature and instead have deployment scripts that handled making sure the correct version was in place.

You still need test environments though for integration testing. And as the code grows, ad-hoc test environments stop being very practical. You should dedicate a VM or two (or even a machine or two) to replicating miniature versions of the real-world setups the code is expected to work in.

Lastly, it's never too early to start using source control on your code. 98% of my code is under source control, even most stuff I think is 'throwaway' or ad-hoc.

I would also strongly recommend Mercurial (or git (if you must)) over Bazaar. It's faster, and the mental model those two tools encourage is a much more accurate representation of what they're really doing. Bazaar lets you pretend that branching is still a big deal and takes some effort to resolve. It lets you continue to think in the model of centralized source systems even though it's not. You will be doing yourself a huge favor in productivity (yes, even for a single developer) to not use it and go for something that doesn't let you pretend anymore. Of those tools, I think Mercurial has a far more carefully thought out and better set of commands and options than git does.

test motivations

[Eponymous Hero]

and now a versioning system would mean going through proper deployment/rollback in order to get real feedback.

not true. using a versioning system does not necessitate testing. just to be clear, testing is always necessary, and not enforced by any versioning system. you can use svn or git or cvs to keep versions of your files so when you do your testing on the production environment (shame on you) you won't have a stack of the same files with extensions like .bak, .bak.bak, .old, .delete, .undo, etc. sitting on your server.

test because it's the right thing, the proper thing, to do. not because you think some tech you choose to use is forcing you. you should be forcing you.

Git in production, version numbers are a nuance.

[Barryke]

Keep the files per project in whatever production directory you want and start a Git repository in it. Version numbers are irrelevant and only a nuance, you have every version of every file with any (commit) comment you want now! Then add scripted backup (such as FTP) to a central location of course to recover from disasters if your production files get damaged.

Add version number if you start rolling out to multiple sites.

It's possible to exchange files between git repositories, or merge back changes made in another production system.

Pretend you're a team

[slim]

Forget that you're a lone programmer. Set up a proper environment anyway.

This is going to seem like hard work, but once you've done the upfront effort, it will pay dividends.

Do *everything* that you'd do if you were a team. There are plenty of books / web sites on the subject.

Pick a version control system -- since you're starting from scratch, Git or Mercurial. Get your code into it.
Pick a continuous build system -- Jenkins is popular and free.
Write one unit test, and make Jenkins run it as part of the build process.
Decide on some sort of repository for your build artefacts.
Establish an integration testing box, and have your CI system deploy to that every build. Ideally use something like Puppet for this, and also use Puppet on your production machines.
Write one integration test, and make Jenkins run it after deployment.

You can dedicate a server to all of this, several servers, run it all on your laptop or in VMs; it really doesn't matter. But think ahead so that you can move it to dedicated machines later if you need to.

Lots of work, but now you have a nice, confidence inspiring build / code management system.

Once that's going, you can decide how to fix your lack of tests. One approach is to take a few weeks just writing tests. Another is to write tests as the need arises -- for new code as you write it; to demonstrate bugs before you fix them. Or somewhere in between.

Python isn't my area, but there is probably an ecosystem of pythonesque tools for a lot of this stuff. pyUnit, code coverage tools, etc.

You will have problems unit testing, since you won't have designed the code for testability. The choice is, live with fewer tests than might otherwise be possible, or refactor your design into something more unit testable. (IOC is unit testing's best friend)

GitHub

[the eric conspiracy]

Just get one of the inexpensive commercial subs for GitHub. This solves all sorts of issues. Remote backup, robust version system, issue tracking etc.

Re:GitHub

[Anonymous Coward]

And it adds all the problems of the cloud. This guy is a networking system admin working at an ISP. He should be able to handle running his own server, or at the very least, should learn how to do all those things.

Use Python's unittest framework

[Anonymous Coward]

I almost posted this exact question about 9 months ago. I ended up using git + GitHub for version control. There are enough comments posted already about version control so all I'll say is that even working by myself, using git for real branching is why I still have a job.

Python's unittest package is really great, at least for the small (10k lines of code) project I'm working on. Using no third-party code at all, you can set up a testing package for yourself so all you have to do is type "python test" before you commit to your repo (make a Python package called "test" and put a "__main__.py" module in it that calls "unittest.main()"). Python is great for glue code so you can write Python unit tests for your Bash code as well as your Python code, and make it all automatic using Python's unittest test discovery (start your file names with "test_*.py").

Perhaps unit tests aren't as good as having a whole box dedicated to testing the entire environment, but it's *dead simple* to maintain, and its simplicity will encourage you to test before every commit (or at least before you merge your topic branches back into master).

Take it out back

[dutchwhizzman]

Take it out back and shoot it. If it's rabid, there is no cure.

Bazaar Hate?

[Anonymous Coward]

What's with the Bazaar hate people? The summary says he's already migrating to bazaar so there's no reason to say 'switch to my XXX version control system because I use it thus it's better for you too'. All the free distributed version control systems (git, mecurial, bazaar, etc..) have the same feature sets (with slightly different names) and none of them have to be used in a distributed fashion. Like RCS, CVS, SVN, and everything else you'll only be using two main commands: check in and check out. The other main commands you'll use are tag, revert, and diff. All the control systems are equally easy to setup and maintain.

Both RCS and CVS don't track changes across multiple files (feature called atomic commits). They should be instantly dropped from consideration for that alone.

I would recommend Bazaar for a beginner as it has excellent step by step tutorials for many use cases and has good GUI tools (which reduce the learning curve until one's ready for custom command line scripts). SVN is fine, but a distributed control system would provide more flexible as you add more people.

Personally, I try to stay away from git and mecurial because of their communities. They tend to have a lot of people saying 'Git/Mecurial is the best versioning control system ever because it can do and the can't.' They never even try to explain the how their distributed system is better than any other distributed system. Those other ones simple don't exist in their warped fanboyism world. The less popular products tend to have a truer view of the world (and thus in my view are better overall).

Ignore any claims of speed. 20,000 lines of code isn't a large project. It might be large for you personally, but in the field of software development it is a largish small project. Large projects have millions of lines. While you're under a million lines and don't have a lot of binary files under version control (the commercial systems handle binary files much better than all the OSS version control software), ignore speed. The minor differences won't add up to the time you spend figuring out which ones are faster under your work conditions.

One last point, you don't need a dedicated computer to host your source code. You're not using 486s are you?

Jenkins for deployment

[THE_WELL_HUNG_OYSTER]

Use Jenkins for deployment. You can automate the entire process. For example, imagine automatically deploying after checking in a revision that contains the word "***DEPLOY***" in the commit comment.

Re:Jenkins for deployment

[turbidostato]

"Use Jenkins for deployment. You can automate the entire process. For example, imagine automatically deploying after checking in a revision that contains the word "***DEPLOY***" in the commit comment."

Now imagine how the hell you reach the confidence point when you can tag "***DEPLOY***" on a commit and then you will see why a CI tool (and I mean the CI *tool*, not a CI strategy) is of almost no value on the systems administration field (which is mostly what we are talking here) but to push to production in the largest environments.

Please pass this to your boss. (don't read it)

[HornWumpus]

You need to fire this cowboy. He doesn't think he needs to test his scripts.

I know he seems irreplaceable. That should be a big red flag.

Most interesting coder

[rfrenzob]

Proclaim yourself the most interesting coder thinkgeek style.

I don't often test my code, but when I do, I do it in production.

Re:Most interesting coder

[theshowmecanuck]

At the risk of being modded OT, I'd like to point out to others that the link is awesome.

Testing and modern Versioning System

[Morpf]

At first, please try to test your code. At least if you can't formally proof it's right testing is the only way to get the most bugs out of the code. Working test first can improve code quality substancially in function and in form, as you can refactor safely with tests in place. Try writing some mock-ups for things outside your own code.

I would choose some distributed versioning system. Not so much because it's distributed, but the most known ones (git, mercurial, bazar) behave way better than svn. The merging algorithms are better and checking out/in on svn with many small files is really, really slow, as it transfers one file after another. Bonus point: working on a local "copy" is fast as no network is going to slow you down.

stop writing code, you are dangerous

[arkowitz]

you are having too much fun dabbling and playing

Well ...

[gstoddart]

My question for the Slashdot community is: in the case of single developer (for now), multiple machines, and a small-ish user base, what would be your suggestions for code versioning and deployment, considering that there are no real test environments and most code just goes into production ?

If I'm the people who run the company, I start firing people. If I'm the developer, I run like hell before anybody realizes what a complete mess I've made.

No versioning, no test environment, live changes in production ... these are warning signs of something which has been cobbled together, and which continues working by sheer dumb luck.

I had a developer once who edited a live production environment without telling anybody and broke it even worse -- he very quickly found himself with no access to the machines and being told that we no longer trusted him with a production environment.

Having worked in highly regulated industries where the stakes are really high, I've had it drilled into me that you simply have no room whatsoever to be doing this kind of thing that ad hoc.

Glad you're starting to use something. But the risk to your employer of all of your stuff tanking and becoming something you can't recover is just too great. From the sounds of it, if you get abducted by aliens or hit by a bus, your company would come to a screeching halt.

Re:Well ...

[turbidostato]

"If I'm the people who run the company, I start firing people."

Unless, of course and as it is usually the case, it is the one running that small company the one that set the policy to start with.

"If I'm the developer, I run like hell before anybody realizes what a complete mess I've made."

Unless, of course and as it is usually the case, the guy is a professional, understands the trade-offs and such does (more or less) the boss that thinks the resulting mess is the most cost-effective way to run his business (and, up to a point, it usually is).

Re:Well ...

[JBHemlock]

It sounds like this is the case with this guy. He may not be the one running the company, but he's discovered (like so many) that what started out as a simple "I need to write a script to X" has turned into an actual department, with him in charge.

I won't bang on the same drum as everyone else. No, actually, I will. It bears banging on. For the past year and a half I've been developing Ghost SEED, a location-based MMO for Android. It's a fair-to-large sized project with basically only me as the active developer, so it's really very similar in situation to yours. I do all of these things (in fact, my day job is worse at this than I am, and it really shows in terms of increased fire drills and general chaos requiring late nights). In order of importance:

1. Set up a test environment. If you don't have the machines for it (and you can replicate your environment on it), sign up for an AWS account. Write good tests that actually exercise all the various code paths in your scripts. *Run your tests*.

2. Pick a version control system. Any version control system is better than none. Lots of folks (myself included) like Git. It's easy to use, reliable, and there are cheap hosting solutions for it that don't require you to have to think about backups or setting up a machine for it. The better hosts also have integrated ticket tracking (so you can keep track of problems and changes) which integrates into the repository (so you can just type something like '[Completed: 451] added support for gzipping responses' in your checkin comments, and it will automatically update the ticket with a link to the commit). Lots of folks like Github. I personally use CloudbaseHQ.

3. This is a distant third. If you can develop your tests to a point where you can run them with an automated system and have it give you an at-a-glance status, you'll be in really good shape. I have a Jenkins environment (an automated build system) set up to deploy my game server from my Git repository to my test server every night and then run all the unit tests on it. I can look at the Jenkins dashboard and see at a glance if it's okay or not. There's a big green light if everything ran fine, and a big red light if they didn't.

Setting all this up took me a couple of days, which were mainly just me learning how to use the various tools (Git, Jenkins, AWS, etc). In the intervening year it's saved me immense amounts of time. I have had only three minor and one major server issues during that year, and they all could be traced back to me being in a hurry and shortcutting the system I'd set up.

apt for deployment

[grewil]

In case you use debian, you can setup your own repository with reprepro. It's really easy to deploy stuff using apt.

Devops and CI

[dna_(c)(tm)(r)]

[...]the test environment would have to be a clone of the production environments. Good luck with that with the described environment![...]

There is stuff like Puppet (for declaratively deploying "services") and Vagrant to provision Virtualbox guests.

Downsides:

• It's only really efficient when your production environment can be provisioned with Vagrant/Puppet as well and no manual work is done on these guests. The way the question is formulated, I suppose that is not the situation.
• Virtualbox is only usable for desktop usage. I would love something similar and simple for KVM

Re:Devops and CI

[Anonymous Coward]

+1 for Vagrant...it's awesome. Vagrant can also be used with chef and chef should have no problems deploying to any environment in production that supports ssh.

Chef seems like the obvious answer to the guy's question. The ruby-based DSL is quite simple to pick up. The deployment cookbooks can be versioned along with the code to be deployed. And it works with a local virtualized test environment. What more can you want?

Re:Devops and CI

[Anonymous Coward]

Virtualbox is only usable for desktop usage. I would love something similar and simple for KVM

Nonsense. I use Oracle VirtualBox on production servers and the virtual machine instances are easy to manage and run smoothly. Of course I run Oracle VirtualBox and the VMs headless.

keep it simple and comment changes in the logs

[Anonymous Coward]

I personally keep all my admin config files, scripts, etc under RCS control. I want per file granularity of comments describing changes. git, mercurial & company solve a different problem than what you have doing system administration.

I'm working on a project right now that will probably have RCS for tracking individual files and mercurial for tracking the project at release level. I use RCS a lot during development. Write some code, test it, check it in, write some more, test it, check it in. Saves time if I find something doesn't work as expected. On one code I currently have 51 versions saved w/ comments describing the changes and reasons for making them. It scientific research code so lots of experiments in different methods of doing things.

SVN + Jenkins

[Gripp]

I commit everything to an SVN, then use jenkins to manage updates. Once you create the jenkins job all you have to do in the future is run it. and you can string jobs together to that if the change needs to be pushed to a number of servers it is still one click.

SQLite

[ericbg05]

My personal programming hero, D. Richard Hipp, works with a very small team on SQLite (which you may have heard of). He uses his own, home-grown SCM called fossil. It probably doesn't scale to a zillion contributors but, like all of Hipp's work that I'm aware of, it's super clean and easy to use. Sounds pretty great for your use case.

And, as other people on this thread have already said: your habit of throwing stuff into production without testing it is similar to playing Russian Roulette with your company. Stop that. Stop that right now.

Please tell me what company / product this is for

[bratmobile]

Because I never, ever want to rely on anything you build this way. You are headed for a disaster, unless you 1) set up a test environment, and 2) use a revision control system.

Really, anything less than that is just a complete waste of everyone's time.

Source Control Axiom

[QuickBible]

You won't need source control until you start using it and then you'll wonder how you ever lived without it. Then you'll start making wild changes because, hey, you have source control now so you can always roll back. This quickly leads to needing a testing environment.

Its all good

[Anonymous Coward]

dude don't worry; within a few weeks you'll be able to deal with all the spaghetti nonsense, since you'll eventually just learn the crazy, non-standard, and ridiculous code anyway...

and hey, once you're fired or quit, it won't be your problem anyway. So, short answer, there's NEVER a good time/reason to do tests unless required by regulations.

Open a bitbucket account

[Slackus]

Bitbucket supports both Git and Mercurial and has free accounts for unlimited private repositories. In addition to version control you get issue tracking, wikis etc.

Mercurial then

[Anonymous Coward]

with mercurial you can also uses the -r564 syntax, and as with git you don't need a repository sever to run anywhere (whether it's on the same machine or a distant server). So if your only objection to git is "I don't want large hex string", then use mercurial, not subversion. really.

as far as "most git advantages don't really apply a single developer" is concerned, once you get used to high speed versionning, full local history, rollback, topic branching and merging, trust me you never want to got back to svn, even when you're developing alone. never.

SVN Repository

[Anonymous Coward]

Check in all major changes individually. This allows for nice rollback and for later analysis if something goes wrong. Also, if you fuck up something while editing you can always roll back. SVN is quite easy after some usage experience and it can be scripted. Don't use the graphical B.S. tools for SVN. Use proper commit comments.

Also, Check in all important documents (e.g. important configurations, network topology plans, setup procedures etc). Protect the repository well, as it will probably contain passwords (I know that is not optimal, but the God Of the Mighty Dollar dictates that).

If his Boss Is An Idiot

[Anonymous Coward]

..he should continue that risky practice. The bozo will only ask him "why things have slowed down", if he does some proper testing. Apparently, it is not (yet) necessary.

There are situations where you can do this - especially if you have excellent people who can quickly patch the problems up. And if you don't operate these commercialware clusterfucks from HP to manage your network.

One Size Fits All ??

[Anonymous Coward]

This is probably a small telecoms operation. Not a billion-dollar bank. I once heard that you need a VP to sign off the smallest change on an Exxon mainframe. That is the opposite end of insanity.

You Must Be An Academic

[Anonymous Coward]

Nobody in the commercial world ever proves code correct. They only test it to various degrees. Same with hardware. And SVN will be the perfect match for this guy. After all, a one-man team will do all the checkins and merge ops.

Playing Devil's Advocate

[Anonymous Coward]

Maybe this is a very competent guy and has zero communications and bullshitting overhead ? Maybe he is a true genius and most of his decisions are actually sound ? Maybe his architecture makes sense ? Maybe he does not need the corporate bullshit bureaucracy ?

One guy of the caliber of a Stallman or a Thovalds will probably do much better than a team of Visual Source Safe users, even if that guy has no source control system.

Re:Playing Devil's Advocate

[slim]

>

One guy of the caliber of a Stallman or a Thovalds will probably do much better than a team of Visual Source Safe users, even if that guy has no source control system.

Linus Torvalds, author of Git?
Richard Stallman, author of GNU diff; without which many revision control systems wouldn't work?

Scalability

[tepples]

a test db box for testing your SQL scripts [...] can have the exact same software, OS and patches, and with equivalent database configuration and schemas, but on lower-cost hardware and with a fraction of the data.

I too maintain a test environment, but I've run into two problems with creating a useful "fraction of the data": First, testing code on a fraction has led to misconceptions about scalability to a far larger data set. Second, how would a substantial fraction data representative of real data be created if the real data contains people's shipping addresses or other PII?

Re:Scalability

[turbidostato]

"how would a substantial fraction data representative of real data be created if the real data contains people's shipping addresses or other PII?"

Do you really have to ask? You either clutter the fields or clutter their relationships:

Exhibit A:
* John Doe | Lexington Av.
* Betty Lamarr | Main St.
becomes
* John Doe | Main St.
* Betty Lamarr | Lexington Av.

Exhibit B:
* John Doe | Lexington Av.
becomes
* Nhjo Ode | Aevtginon Lx.

Re:Scalability

[theshowmecanuck]

testing code on a fraction has led to misconceptions about scalability to a far larger data set

This is real. The solution is to manage expectations. If people know that the tests just show functionality and not scalability, and that scalability testing is required (when warranted), you should be good. More importantly if the decision makers know this, you are good.

if the real data contains people's shipping addresses or other PII?

Scrub the data. Addresses are not personal information though. The fact a specific person lives there might. Open a phone book (if you can find one now-a-days. They have reams of addresses as well as phone numbers tied to real people. This is public knowledge. Personal information involves things more like name, age, finances, medical records, etc.

For the stuff that is real personal information, randomizing names to create fake people tied to real addresses is not hard at all (real addresses are often necessary when system tie into others where shipping or location are requirements). You can take real information and put it in a can and scrambled to make fake people. I think testers should be proficient enough to be able generate this kind of data.

As to one other comment made by the OP:

and now a versioning system would mean going through proper deployment/rollback in order to get real feedback.

Versioning systems do no such thing if you don't use them that way. If you want a "proper deployment and rollback cycle" you can do that. Or not. But at least you'll be able to go back in time to find the code that actually worked if you need to. No coder should work without the safety net of version control. Whether it be CVS, SVN, GIT, it matters less what it is than whether you have one or not. Pick one and use it.

Re:Scalability

[Anonymous Coward]

Let's not forget how easy it is to scrub names or generate test data. Just go to http://www.fakenamegenerator.com/ and you can get some pretty great test data, including names with accented characters (your name columns are defined as nvarchars instead of varchars, right?).

Nuclear Weapoms

[Anonymous Coward]

..also kill that noisy crow. He could have used an air-pressured rifle (SVN) to scare it off, but if he can use a nuke - it will be more fun. And it will also take care of all the ants.

vim

[trev.norris]

ggdG

Tried all the major free DVCS's and prefer bzr

[Anonymous Coward]

I prefer bzr, coming from SVN. It actually intrinsically supports various workflow models, including a very SVN-like central model.

When our corporate dev team of ~20 tried git it was an unmitigated disaster. Since we also moved to a "branch per feature" model simultaneously, this only exacerbated the problems.

May you never experience the "joys" of your devs hoarding commits locally because the DVCS allows this, thus reducing collaboration.

Or having the complex/confusing git push system cause a dev to rewrite history on the central git server and effectively "lose" another devs' previous commits by unhooking those from the branch history. (Why is the ability to rewrite commit history & orphan commits in branches on the central server considered a feature in git? God only knows...)

Or have your devs spend more time with git's two phase commit crap and so forth when all they wanted was to mimic the effects of a simple "svn commit" operation (hahaha, silly devs, committing should be *complicated*!), leaving them uncertain whether their crap is actually committed, committed locally, or actually pushed to the central server.

BTW, hope you think monotonically increasing revision numbers for commits in SVN are worthless, because git has no way to do anything like that. BZR has an approximation of revision numbers, on a per branch basis, though. Also, hope you have no use for having an empty directory checked into your repo, because git doesn't support that either. Naturally, you probably have no use for having renames be a supported operation in your VCS, because git doesn't do that either (it uses heuristics instead, though bzr *does* support true file rename capability/tracking).

Of course, the list goes on and on. Eventually, the team decided to go back to using SVN. Unfortunately, while there is a SVN-to-git importer, there's no way back. We decided it was better to write off years worth of commit history to get back to a useful VCS, so we merged all our git branches, stomped them flat, exported, and committed a rev 0 into a brand new SVN repo.

Thus ended that debacle.

Now, my view is extremely unpopular around the internet. Git zealots abound who are loathe to acknowledge any other VCS could be superior to git in any way, under any circumstance. They just don't seem to understand that not every project is the Linux kernel, which requires coordinating thousands of devs around the world in a distributed fashion.

That said, branching & file renaming blows in SVN. BZR does all those things very well, and "bzr commit" actually works like a SVN user would expect, etc, etc. I suggest you give it a try.

Oxymoron

[js_sebastian]

The scripts are irrelevant if not ran on the real environment,

Well, that's an oxymoron

It does not mean what you think it means.

Cold fire is an oxymoron, or dry water, or dumb genius. An oxymoron is an inherently contradictory combination of terms.

Re:Oxymoron

[Anonymous Coward]

Dumb genius isn't an oxymoron, it's just a genius who's unable to speak. Moron genius - that's an oxymoron.

Re:Oxymoron

[js_sebastian]

Dumb genius isn't an oxymoron, it's just a genius who's unable to speak. Moron genius - that's an oxymoron.

It appears you have out-pedenticed me...

20k?

[Anonymous Coward]

20K lines? Tiny!

What's your problem?

[Peter (Professor) Fo]

(1) You have job security. (2) You have delivered what is needed so far Now what happens... Your organisation could carry on as-is. It has worked in the past but you might get eaten by a rogue killer whale. Oho! A /real/ risk of you going AWOL for (100 genuine reasons). So how do you suggest this is is handled? In-line comments? Development Wiki? etc. IMHO I would try to list the things that could go wrong... ...Then put them into categories such that responsibility for a certain category can be handed to whoever. Two days later ask whoever is supposed to be 'in charge of foo' how they intend dealing with a certain detail. (This is Health and Saefty... Gone mad) if you ask people happily fumbling about their own business.ask to see the legal details.

a bit more info

[SethJohnson]

A good example is a test db box for testing your SQL scripts. Such a box can have the exact same software, OS and patches, and with equivalent database configuration and schemas, but on lower-cost hardware and with a fraction of the data.

I don't want to pick nits here because Luis is giving out a lot of very valid information and observations here. Just want to take it one step further.

Mirror the production environment DB with an identical amount of data. The data doesn't have to match row-for-row. But the test environment DB should have the same scale as the production DB. Here's why: If you want to run an alter table or alter index command, you want a sense of how much of an impact that is going to have on the database before running the operation in production. If you only have 10k rows in the table in test and run the command, it'll complete pretty quickly. You run that against a 2-million-row table in production, depending on the vendor, that could mean a table lock that takes down your production environment until it finishes (could be hours). Been there. Done that. It ain't pretty.

Seth

baby steps toward sanity

[beegle]

I've done a little bit of environment taming in my day.

Everybody's already told you the "right" things to do. They're all right. Thing is, you need to get there somehow, and you're looking for a path from here to there. At least, I think that's what you're asking.

You already have bazaar. Good tool. Don't worry about bzr versus cvs versus hg right now. You picked something. Run with it.

I suggest a quick shell script that replaces your editor with "edit; check-in; offer to push". Create another quick script (call it "oops") that asks you whether you need a local or global revert, then issues the relevant commands. Push those scripts to all machines (maybe as their own bzr project). Now, you basically have the same process with a much larger safety net. This isn't software that's being released to the world. Don't worry about version numbers or branching. If you EVER have to change a file, throw it into version control.

Now, you can start to synchronize your scripts. With an environment that wild, you probably have a script that's almost-but-not-quite the same running on a bunch of machines. Maybe there's a hard-coded hostname or directory or something. Come up with a version that's more universal (Use big "if hostname = foo" blocks if you have to), and get that new universal script added to a project and pushed to all machines. Once they're all using it, you can slowly clean it up.

Cool. You have unified scripts. Now let's talk about those configuration files. I'll bet that they're also 99% identical across all of your machines. Get them all into the same project (call them config-machine1, config-machine2, etc.). Get them as identical as possible. Now, think about how you might handle differences. For a quick fix, I like the "magic comment" ("## BEGIN foo.sh MANAGED SECTION" and "## END foo.sh SECTION") and a perl script that looks for those strings. m4 also works well, and isn't too hard to learn.

"Okay, smart guy. I have all of the common config scripts, but I have a bunch of single-purpose machines and scripts, too!" Yup. Awesome. Get them into version control, too. You never know when that machine's going to suddenly die or your boss will break out in a fit of generousity and get you that second server for load balancing. (Hey! It could happen.) When it does, setup will be a lot easier if you have all of the config files in a project.

At this point in the game, you'll be pretty comfortable with version control. You'll have been burned once or twice, and it'll have saved your butt a few times. You'll have some experience, and you'll be kicking yourself for the way that you first set it up. Now's the time to revisit those decisions. Is it time to split up some projects or roll some together? Maybe git or hg might make more sense. Maybe you hate your life and your coworkers so much that you want to go to Perforce, ClearCase, or some other commercial software. You'll have the experience to design it right.