Hardly. I'm living the dream. But to get there takes a fair bit of hard-eyed realism.
Besides, believe it or not, sometimes features are developed by *multiple* people. And those people need to work together and share their in-progress, possibly not-completed work through some mechanism. Some sort of... management system, if you will, for their source code.
I'm a big fan of working in teams. I just don't think it has to involve leaving a lot of half-finished stuff around. As Yoda would point out: "Do or do not."
Sorry, but some corporate clients are *not* happy with you dropping a new release every week.
Right. And why is that? Because they are used to software sucking. You're not thinking this through. If you could actually reduce your bugs in number and severity to a level where nobody cared, then they would be fine with releases.
All the advice I'm giving is pretty standard agile advice. You'll here it especially from the Extreme Programming end of the agile spectrum, but a number of Scrum people talk this way, too.
If you'd actually like to find out how that works, try, say, reading a book. Or drop by the Extreme Programming mailing list on Yahoo. There are a number of people expert with revision control and configuration management. There were entire presentations on the topic at Agile 2008 this year.
I've tried it, and it works just fine. I've also tried it your way, which is why I'm pretty comfortable saying which one I think works better.
That impossible for bigger changes or even whole features.
I think it's more accurate to say that you don't do that, and don't know how. Impossible means that you can prove that it could never happen. Proving a negative is hard, as somebody just has to come along with one counter-example. Perhaps you could consider what that example might look like?
You never worked on a feature that took more than a day to finish?
I work on things that never finish. Most people do; if your product is really finished, that means the company is closing.
Given that work is endless, the only question is what size chunk you work in. I commit to trunk every few hours, and I always try to keep trunk releasable. So although I have worked on things that took more than a day to finish, I now see that as a problem, and it rarely happens to me these days.
If you read the article, you would realize how foolish you sound, and you might learn what branching and merging are as well!
I am well aware that philosophically, checking out is effectively branching, and updating and committing are effectively merging. Some people really think of it that way. In my experience, a lot more people think of it the other way.
I'm going to stick with that language, partly because that's the default, and partly because local, private branches and merges are very different than public ones, and I am focusing on the public ones here.
Where do you keep things that you don't think are stable? [...]Either that, or you rarely have to check in a bug fix while you're in the middle of a "2-3 week" change.
I commit every few hours, and it's the rare bug fix that can't wait that long. (Actually, for my teams, any bug fix is relatively rare.) When it can't, I'll either use the "shelve changes" feature in my IDE, or create a new workspace. Either's sufficiently quick for me, especially when I do it maybe once a quarter.
If code isn't stable by the end of the day, then obviously I and my pair have lost track of what's going on, creating a bigger mess than we know how to manage. So usually we just throw out the change and start over in the morning. And when we don't, we generally wish we had. Untangling a giant knotted ball of string isn't worth it; it's cheaper just to buy more string.
Yes. Most of us agree that shorter and simpler branches are better. The teams I work with commit to trunk every few hours. You could call that short branches, or you could call that checking out, and I don't much care which.
It's hard to expect the unexpected, no matter how good a programmer you are.
Not really. A fall for a trapeze artist is always unexpected, but the fact that they will eventually fall isn't. So they have nets and other safety gear.
If you use test-driven development, automated end-to-end tests, pair programming, and promiscuous pairing, you can get your bug rates very low. Combine that with a skilled exploratory tester in the room, and they get very low indeed. And in my experience, you can develop much faster, so the work put into testing is essentially free.
You don't add a new feature in a bugfix release, you fix bugs!
And if you stop putting in bugs, you don't have any bug fix releases, therefore no bug fix branches. For a lot of situations, that means you can then work off trunk.
You do have a substantial chance of introducing bugs. Or maybe not, but that's how the customer will see it. Good luck convincing them otherwise if they have any experience with software.
My teams don't, but we work hard to keep it that way. My point is that a lot of people don't, and that's why they have to do a lot of monkeying around with stable versus dev branches. If you reduce your bug count to near zero, then nobody cares, and that need for branching goes away.
And yes, even customers come to think that. It's actually surprisingly easy to convince them. If you release new versions weekly and your bugs are small and very rare, then they come to see releases as easy and safe. It doesn't take long before you don't have to push them at all to release. There are new features they want to use, after all.
You'll need advance permission from the customer for a change like that.
Yeah, that's why I have a product manager in the room, often within arm's reach. They represent the customer fully, and are empowered to make those decisions.
Sorry, I replied to myself minutes later to clear up that false implication. Perhaps you missed that.
I was writing in the context of my parent post, where I said that branching was certainly necessary in some cases, but about 90% of the branching I see is due to failures to work effectively as a team. I still stand by that, but don't have any published data to back that up.
I think this was very true for some shops in Ye Olden Dayes. But it wasn't true for all of them; I've interviewed retired programmers who released frequently in manufacturing and office environments, for example.
Regardless, it's getting less true all the time. People are used to software changing more frequently on the web. That's true for both simple software and complex software. More importantly, interface and interaction designers are learning to change interfaces in ways that don't disturb normal workflows. And they're making applications more self-documenting, so that people can self train.
For the consumer web, building software like that is a necessity. You can't force people to deal with bad UI, and you can't force them to sit through training sessions. It has to work. People will eventually learn to apply these lessons to a lot of other software as well. Not everything can work this way, but anybody boldly asserting that it is impossible for their particular project is making a big gamble. A lot of other people have said that and been proved wrong lately.
It means that the Wikipedia "consensus truth" is balderdash.
It does?
I have a physics textbook from 100 years ago. It talks about the luminiferous aether, but mentions nothing about relativity. By your logic, that would mean that "science" is balderdash.
I think a better way to look at it is that what people call "truth" is always just a human understanding of the world. Limited, finite, and fallible, the best we humans can do is to continuously challenge and refine that understanding. Wikipedia has its flaws, but it does open that process up to the whole world, which seems like a plus to me.
Reality is that if you have project with more than one file, you already can run into conflicts. And if branching is cheap and easy - why not to use it?
Because branching is frequently easy, but not nearly as often cheap.
In my experience, the cost of finding a defect is larger the longer it sits, and the larger the merge, the more expensive it is, and in a way that's worse than linear. Ergo, I believe in doing everything possible to find defects as early as possible, branching as little as possible, and integrating as early as possible.
With your rate of 1 defect per developer per month, all our customers would have ran away already: after all we are doing charging software, we are responsible for managing customer's money.
Well, I did say "below". I have also done financial trading software, and there my defect rates are lower. All of my recent projects are consumer-facing web projects, though, and I can think of only once in the last year (so ~52 releases) that we've had to branch and do a special release because the bug was bad enough that people couldn't wait until the next release.
My point is not that people shouldn't branch. What I'm mainly objecting to in this thread is people solving problems that have nothing to do with versioning via version control systems.
Maybe in your situation, every branch really is necessary, in which case, bully for you. But it sounds like a lot of your branches are not driven by shipping different versions of the software, but by people who are having a hard time working together. For those, I think it's worth asking, "Hey, is there some other way we could solve this problem?"
Or, to rephrase, even _your_ team is unable to produce stable code every time they check in.
So?
The point is that you can get defect rates low enough that nobody cares, and you can therefore stop screwing around with branching just because you're scared of your teammates.
There are definitely good reasons to branch, but not knowing how to make reliable software isn't one of them.
That would be a different case than the "large team" discussion that triggered this sub-thread.
I agree that maintaining different released versions can sometimes justify branching (although there are other options, too).
But it's worth asking why your customers don't want to upgrade. It's often because previous upgrades have introduced new bugs, have some incompatibility, or aren't as good as older versions. In other words, that newer versions aren't better. You could fix that by branching, but wouldn't it be better to always send them software that is actually better?
Heck, you may even need to support the users who are running the old code for a long time, and have to keep doing bug fixes for them while most of the world is on the newer codebase.
That strikes me as a perfectly legitimate use of branching, just as long as the reason they aren't upgrading isn't to do with hard upgrades or fear of breakage.
I change the broken API to FooLib.
Personally, I solve this by a) when it's the same project, fixing every caller in one go, or b) when it's a separate project, treating FooLib as its own product. When a team is ready for a new library, they can upgrade.
Or I write the tests for the new refactor, commit them (all broken), and then another programmer pulls them down and does the refactor.
This is generally the kind of thing where I pair. If I don't know enough to do the job on my own and neither do they, then it seems to make more sense to work together.
That's even leaving aside the much more obvious cases of "this is a massive change that's going to result in several days' breakage as we all work through it"
Could you give an example of that? I just can't think of when that's happened to me; we just do the changes incrementally and check them in piece by piece.
On even a medium-sized project, you don't want to supplant the stable code until there's been a whole heck of a lot more testing than just one programmer who thinks he's written good tests and done plenty of his own integration testing.
Here I'd favor automating the acceptance tests, plus pair programming and possibly other quality-oriented practices. There's still some risk of a bug, but it's pretty low. Branching means a doubling of the test effort, as you have to test both before merge and after.
Then what is the nature of this cost you refer to?
It's a human cost.
If I'm keeping track of a bunch of branches in my head, then that limits the amount of other stuff I can keep track of. The time I and the people on my team spend talking about the different branches is time we can't talk about actual features. If my ops guy or my product manager has to figure out what's where and what's elsewhere, then that's time, attention, and energy that they can't put into what they're really there for.
It makes me think of juggling. Branching is throwing another ball in the air. Maybe it's a tiny rubber ball that I catch and put down again right away, in which case I don't care much. Or maybe it's a giant flaming chainsaw that we have to keep going for months, in which case it's a lot of effort. But either way, I have to keep it going.
Yes, yes, it's all totally impossible. Except that I do it and have done it. For years.
If you've got questions about how, feel free to ask. But until you can accept that what you call "RC 101" is just one approach to solving problems in software development, there's not a lot I can do for you.
But here's a hint: Try googling for "agile" or "extreme programming".
What about cases like Operational software where you may need to keep an Operational branch, a Development branch and perhaps even a Prototype or Test branch.
Could you say more about the business case you're thinking about?
If you are attempting to ship a solid product and you need a critical bug fix ASAP, you really want to start from EXACTLY the code you shipped last time and make the minimum change to fix the critical bug.
This is true only when you think your changes have a substantial chance of introducing bugs. If you are not worried, this is not a problem. See my reply a couple up thread for more on that.
Ship early ship often is not a recipe for being taken seriously in a lot of businesses.
As far as I can tell, this is mainly because they are used to software sucking a lot. If your software is sufficiently reliable nobody cares. What company insists that you can't use the new version of Google until it has been vetted by IT?
When adding a new feature that might take 2-3 weeks to be "stable", do you use a different system to back up your work each day?
I don't check in things that I don't think are stable. I do that primarily through test-driven development and pair programming, plus automated acceptance testing and showing new versions to product management and/or QA every few hours.
Bugs only get harder to find over time. It seems a lot cheaper to me to kill them dead as soon as possible, rather than storing them up and trying to find them later.
Yeah, I completely agree. If I were starting an open-source project today, especially one that I hoped to become giant, I'd definitely use something like git.
The best way is to build features iteratively, so that you start extremely small and release continuous improvements. The internet has made users very comfortable with this when done right. Nobody knows or cares what Google's release schedule is, for example.
Next best is to work in ways which hide the change. If you're adding a new flow, for example, you can build it all out of sight of the user, and connect it to the rest of the app as your final commit. It might still be released, but nobody sees it. But you can easily test it internally, and because integration happens in small steps, it's much easier.
You can also make the feature switchable. Google's a good example there as well; by making a feature conditional, it lets them test features on a small segment of the general public before making something generally available. Launching a feature is flipping a switch, rather than related to the release process.
Some people also make things conditional in a different way, so that they can create multiple builds of the app from the same source code base.
But I never use any of those tricks if I can just start small.
Did you ever tried to code yourself? You look/sound like idiot here.
Heh. My parents are programmers. I started programming for fun in the very early 80s, and for money towards the end of that. I haven't written any code this week, but about half my time over the last couple of years is still coding.
The teams I have run recently have defect-in-production rates below one per developer-month. What are yours?
Slashdot Mirror
Rose-coloured-glasses, much?
Hardly. I'm living the dream. But to get there takes a fair bit of hard-eyed realism.
Besides, believe it or not, sometimes features are developed by *multiple* people. And those people need to work together and share their in-progress, possibly not-completed work through some mechanism. Some sort of... management system, if you will, for their source code.
I'm a big fan of working in teams. I just don't think it has to involve leaving a lot of half-finished stuff around. As Yoda would point out: "Do or do not."
Sorry, but some corporate clients are *not* happy with you dropping a new release every week.
Right. And why is that? Because they are used to software sucking. You're not thinking this through. If you could actually reduce your bugs in number and severity to a level where nobody cared, then they would be fine with releases.
All the advice I'm giving is pretty standard agile advice. You'll here it especially from the Extreme Programming end of the agile spectrum, but a number of Scrum people talk this way, too.
If you'd actually like to find out how that works, try, say, reading a book. Or drop by the Extreme Programming mailing list on Yahoo. There are a number of people expert with revision control and configuration management. There were entire presentations on the topic at Agile 2008 this year.
I've tried it, and it works just fine. I've also tried it your way, which is why I'm pretty comfortable saying which one I think works better.
Yes. If you stop making things unstable, then you don't need that extra branch. See the rest of my posts here for how that might happen.
That impossible for bigger changes or even whole features.
I think it's more accurate to say that you don't do that, and don't know how. Impossible means that you can prove that it could never happen. Proving a negative is hard, as somebody just has to come along with one counter-example. Perhaps you could consider what that example might look like?
You never worked on a feature that took more than a day to finish?
I work on things that never finish. Most people do; if your product is really finished, that means the company is closing.
Given that work is endless, the only question is what size chunk you work in. I commit to trunk every few hours, and I always try to keep trunk releasable. So although I have worked on things that took more than a day to finish, I now see that as a problem, and it rarely happens to me these days.
If you read the article, you would realize how foolish you sound, and you might learn what branching and merging are as well!
I am well aware that philosophically, checking out is effectively branching, and updating and committing are effectively merging. Some people really think of it that way. In my experience, a lot more people think of it the other way.
I'm going to stick with that language, partly because that's the default, and partly because local, private branches and merges are very different than public ones, and I am focusing on the public ones here.
Where do you keep things that you don't think are stable? [...]Either that, or you rarely have to check in a bug fix while you're in the middle of a "2-3 week" change.
I commit every few hours, and it's the rare bug fix that can't wait that long. (Actually, for my teams, any bug fix is relatively rare.) When it can't, I'll either use the "shelve changes" feature in my IDE, or create a new workspace. Either's sufficiently quick for me, especially when I do it maybe once a quarter.
If code isn't stable by the end of the day, then obviously I and my pair have lost track of what's going on, creating a bigger mess than we know how to manage. So usually we just throw out the change and start over in the morning. And when we don't, we generally wish we had. Untangling a giant knotted ball of string isn't worth it; it's cheaper just to buy more string.
Yes. Most of us agree that shorter and simpler branches are better. The teams I work with commit to trunk every few hours. You could call that short branches, or you could call that checking out, and I don't much care which.
It's hard to expect the unexpected, no matter how good a programmer you are.
Not really. A fall for a trapeze artist is always unexpected, but the fact that they will eventually fall isn't. So they have nets and other safety gear.
If you use test-driven development, automated end-to-end tests, pair programming, and promiscuous pairing, you can get your bug rates very low. Combine that with a skilled exploratory tester in the room, and they get very low indeed. And in my experience, you can develop much faster, so the work put into testing is essentially free.
You don't add a new feature in a bugfix release, you fix bugs!
And if you stop putting in bugs, you don't have any bug fix releases, therefore no bug fix branches. For a lot of situations, that means you can then work off trunk.
You do have a substantial chance of introducing bugs. Or maybe not, but that's how the customer will see it. Good luck convincing them otherwise if they have any experience with software.
My teams don't, but we work hard to keep it that way. My point is that a lot of people don't, and that's why they have to do a lot of monkeying around with stable versus dev branches. If you reduce your bug count to near zero, then nobody cares, and that need for branching goes away.
And yes, even customers come to think that. It's actually surprisingly easy to convince them. If you release new versions weekly and your bugs are small and very rare, then they come to see releases as easy and safe. It doesn't take long before you don't have to push them at all to release. There are new features they want to use, after all.
You'll need advance permission from the customer for a change like that.
Yeah, that's why I have a product manager in the room, often within arm's reach. They represent the customer fully, and are empowered to make those decisions.
Sorry, I replied to myself minutes later to clear up that false implication. Perhaps you missed that.
I was writing in the context of my parent post, where I said that branching was certainly necessary in some cases, but about 90% of the branching I see is due to failures to work effectively as a team. I still stand by that, but don't have any published data to back that up.
I think this was very true for some shops in Ye Olden Dayes. But it wasn't true for all of them; I've interviewed retired programmers who released frequently in manufacturing and office environments, for example.
Regardless, it's getting less true all the time. People are used to software changing more frequently on the web. That's true for both simple software and complex software. More importantly, interface and interaction designers are learning to change interfaces in ways that don't disturb normal workflows. And they're making applications more self-documenting, so that people can self train.
For the consumer web, building software like that is a necessity. You can't force people to deal with bad UI, and you can't force them to sit through training sessions. It has to work. People will eventually learn to apply these lessons to a lot of other software as well. Not everything can work this way, but anybody boldly asserting that it is impossible for their particular project is making a big gamble. A lot of other people have said that and been proved wrong lately.
It means that the Wikipedia "consensus truth" is balderdash.
It does?
I have a physics textbook from 100 years ago. It talks about the luminiferous aether, but mentions nothing about relativity. By your logic, that would mean that "science" is balderdash.
I think a better way to look at it is that what people call "truth" is always just a human understanding of the world. Limited, finite, and fallible, the best we humans can do is to continuously challenge and refine that understanding. Wikipedia has its flaws, but it does open that process up to the whole world, which seems like a plus to me.
Reality is that if you have project with more than one file, you already can run into conflicts. And if branching is cheap and easy - why not to use it?
Because branching is frequently easy, but not nearly as often cheap.
In my experience, the cost of finding a defect is larger the longer it sits, and the larger the merge, the more expensive it is, and in a way that's worse than linear. Ergo, I believe in doing everything possible to find defects as early as possible, branching as little as possible, and integrating as early as possible.
With your rate of 1 defect per developer per month, all our customers would have ran away already: after all we are doing charging software, we are responsible for managing customer's money.
Well, I did say "below". I have also done financial trading software, and there my defect rates are lower. All of my recent projects are consumer-facing web projects, though, and I can think of only once in the last year (so ~52 releases) that we've had to branch and do a special release because the bug was bad enough that people couldn't wait until the next release.
My point is not that people shouldn't branch. What I'm mainly objecting to in this thread is people solving problems that have nothing to do with versioning via version control systems.
Maybe in your situation, every branch really is necessary, in which case, bully for you. But it sounds like a lot of your branches are not driven by shipping different versions of the software, but by people who are having a hard time working together. For those, I think it's worth asking, "Hey, is there some other way we could solve this problem?"
Or, to rephrase, even _your_ team is unable to produce stable code every time they check in.
So?
The point is that you can get defect rates low enough that nobody cares, and you can therefore stop screwing around with branching just because you're scared of your teammates.
There are definitely good reasons to branch, but not knowing how to make reliable software isn't one of them.
That would be a different case than the "large team" discussion that triggered this sub-thread.
I agree that maintaining different released versions can sometimes justify branching (although there are other options, too).
But it's worth asking why your customers don't want to upgrade. It's often because previous upgrades have introduced new bugs, have some incompatibility, or aren't as good as older versions. In other words, that newer versions aren't better. You could fix that by branching, but wouldn't it be better to always send them software that is actually better?
Heck, you may even need to support the users who are running the old code for a long time, and have to keep doing bug fixes for them while most of the world is on the newer codebase.
That strikes me as a perfectly legitimate use of branching, just as long as the reason they aren't upgrading isn't to do with hard upgrades or fear of breakage.
I change the broken API to FooLib.
Personally, I solve this by a) when it's the same project, fixing every caller in one go, or b) when it's a separate project, treating FooLib as its own product. When a team is ready for a new library, they can upgrade.
Or I write the tests for the new refactor, commit them (all broken), and then another programmer pulls them down and does the refactor.
This is generally the kind of thing where I pair. If I don't know enough to do the job on my own and neither do they, then it seems to make more sense to work together.
That's even leaving aside the much more obvious cases of "this is a massive change that's going to result in several days' breakage as we all work through it"
Could you give an example of that? I just can't think of when that's happened to me; we just do the changes incrementally and check them in piece by piece.
On even a medium-sized project, you don't want to supplant the stable code until there's been a whole heck of a lot more testing than just one programmer who thinks he's written good tests and done plenty of his own integration testing.
Here I'd favor automating the acceptance tests, plus pair programming and possibly other quality-oriented practices. There's still some risk of a bug, but it's pretty low. Branching means a doubling of the test effort, as you have to test both before merge and after.
Then what is the nature of this cost you refer to?
It's a human cost.
If I'm keeping track of a bunch of branches in my head, then that limits the amount of other stuff I can keep track of. The time I and the people on my team spend talking about the different branches is time we can't talk about actual features. If my ops guy or my product manager has to figure out what's where and what's elsewhere, then that's time, attention, and energy that they can't put into what they're really there for.
It makes me think of juggling. Branching is throwing another ball in the air. Maybe it's a tiny rubber ball that I catch and put down again right away, in which case I don't care much. Or maybe it's a giant flaming chainsaw that we have to keep going for months, in which case it's a lot of effort. But either way, I have to keep it going.
Yes, yes, it's all totally impossible. Except that I do it and have done it. For years.
If you've got questions about how, feel free to ask. But until you can accept that what you call "RC 101" is just one approach to solving problems in software development, there's not a lot I can do for you.
But here's a hint: Try googling for "agile" or "extreme programming".
What about cases like Operational software where you may need to keep an Operational branch, a Development branch and perhaps even a Prototype or Test branch.
Could you say more about the business case you're thinking about?
If you are attempting to ship a solid product and you need a critical bug fix ASAP, you really want to start from EXACTLY the code you shipped last time and make the minimum change to fix the critical bug.
This is true only when you think your changes have a substantial chance of introducing bugs. If you are not worried, this is not a problem. See my reply a couple up thread for more on that.
Ship early ship often is not a recipe for being taken seriously in a lot of businesses.
As far as I can tell, this is mainly because they are used to software sucking a lot. If your software is sufficiently reliable nobody cares. What company insists that you can't use the new version of Google until it has been vetted by IT?
When adding a new feature that might take 2-3 weeks to be "stable", do you use a different system to back up your work each day?
I don't check in things that I don't think are stable. I do that primarily through test-driven development and pair programming, plus automated acceptance testing and showing new versions to product management and/or QA every few hours.
Bugs only get harder to find over time. It seems a lot cheaper to me to kill them dead as soon as possible, rather than storing them up and trying to find them later.
Yeah, I completely agree. If I were starting an open-source project today, especially one that I hoped to become giant, I'd definitely use something like git.
There are a lot of ways to deal with it.
The best way is to build features iteratively, so that you start extremely small and release continuous improvements. The internet has made users very comfortable with this when done right. Nobody knows or cares what Google's release schedule is, for example.
Next best is to work in ways which hide the change. If you're adding a new flow, for example, you can build it all out of sight of the user, and connect it to the rest of the app as your final commit. It might still be released, but nobody sees it. But you can easily test it internally, and because integration happens in small steps, it's much easier.
You can also make the feature switchable. Google's a good example there as well; by making a feature conditional, it lets them test features on a small segment of the general public before making something generally available. Launching a feature is flipping a switch, rather than related to the release process.
Some people also make things conditional in a different way, so that they can create multiple builds of the app from the same source code base.
But I never use any of those tricks if I can just start small.
Did you ever tried to code yourself? You look/sound like idiot here.
Heh. My parents are programmers. I started programming for fun in the very early 80s, and for money towards the end of that. I haven't written any code this week, but about half my time over the last couple of years is still coding.
The teams I have run recently have defect-in-production rates below one per developer-month. What are yours?