Interview With BitKeeper Author Larry McVoy

Jeremy Andrews writes "KernelTrap has spoken with Larry McVoy, BitMover founder and primary BitKeeper author. BitKeeper, a distributed source control system, has been adopted by Linux kernel creator Linus Torvalds and condemned by free software icon Richard Stallman. In this interview, Larry looks back through the years, describing his exposure to computers and Linux. He also discusses the history of BitKeeper, from writing NSElite for Sun (which turned into their still used SCM, Teamware), to his desire to keep Linus from burning out, to the present day solution. The choice to not license BitKeeper under the GPL is also explained. Larry discusses much beyond Bitkeeper as well, exploring some of his other interests. Find the full interview on KernelTrap."

Comments

Congratulations to Linus

[WzDD]

For years, I've been skeptical of the Linux kernel development model, and specifically the its lack of source control. While it seemed to be "working", Linus has showed the strain several times on the kernel list. As far as I'm concerned, I'm glad he's found a tool that works for him - I totally agree with Linus' attitude of "use the tool that works for you, not its ideologically better, but otherwise inferior competition".

Hopefully this will at least alleviate some of the "Linus doesn't scale" criticisms, too.

free... as in freedom?

[TechnoVooDooDaddy]

Stallman can't get over the fact that Bitkeeper is NOT licensed under the GPL, and that Linus chooses to use it anyway. Presumably Linus just likes it better, and he's free to do as he sees fit. Freedom, that's an interesting word, because the mere notion of it means it must apply equally and unilaterally to everyone, or it doesn't exist. Stallman has repeatedly tried to exert pressure on people including McVoy to license things under *his* GPL, and complaining loudly when it doesn't happen. In other words Stallman is making an effort to limit their freedom with their own product.

Freedom applies to everyone, or it applies to no one.

Re:free... as in freedom?

[abe ferlman]

It's suddenly become fashionable to bash Stallman on Slashdot. How odd.

Anyway, your point is wrong-headed. Think about what gets said when people discuss GPL'd game software- people say "GPL is important for infrastructure, not for entertainment". They say when it counts open standards are important but when it's trivial it's ok to be closed.

Well, this is a serious infrastructure issue and Stallman has every right to be upset about it.

Your twist of the word freedom is easily parried- the freedom to take away others' freedom is not a freedom at all. The GPL position is internally consistent.

And finally, your attack, like so many here today, is premised on an ad hominem argument- you're attacking Stallman, not his argument.

The only part of the interview that matters....

[mshiltonj]

This is his response to the GPL issue. It's a good point. The choice is: Enough money to feed the family and pay the mortgage, or give back to the community and make a fraction of what you could have otherwise. It's not always this black and white, but many times it is.

Larry McVoy: I've never bought into the open source model as a self sustaining model for all software. It works in some places the software is tied to some other source of revenue, such as hardware, but in general, it stinks as a business model. It's fantastic if your goal is to have a lot of free software out there, but it starts to fall apart when building that free software costs more than you can extract from it in revenue.

BitKeeper is in that camp. There is about 25 man years of effort in BitKeeper so far, with no end in sight. We pay Bay Area salaries, so our cost for an engineer is about $160K/year. That's at least 4 million dollars no matter how you look at it, and that's a lower bound. I took a hard look at the Cyclic people who tried to make a business out of supporting CVS and they pulled in $145K in their best year. It would take 27 years to make $4 million at that rate, and that assumes we stop drawing salaries today. In this product space, if people can use it for free, they will. People have tried to argue with me that BitKeeper is a better tool and it would generate more support revenue. That's nonsense, exactly because it is a better tool. At least with CVS, there are enough broken or missing features that you could generate revenue to fix them. Maybe.

So I took a hard look at the situation and decided that I wanted to maximize value to everyone. I divided the world up into 3 camps: the free users, the commercial users, and the vendor. The goals were to provide maximum value to everyone and have everyone provide value back in return. Here's how it works:

Free users: these users don't pay in money, but they do pay. They pay by using the product and pointing out bugs. BitKeeper is a dramatically better product because of the free users. The BKL, the free usage license, insists that you are running the latest images, because that's where the free users provide value. It doesn't help anyone to get bug reports on problems we've already fixed. The job of the free users is to help debug the latest.

Commercial users: these users pay in money which funds further development. As a commercial user, they can pick which release they want to run, which sometimes means they stay back for stability reasons, perceived or otherwise. They benefit from the free users running a new release first, and it's typical that they wait for the timestamps in the download area to be a few weeks old before upgrading.

Vendor: we provide value in the form of the product and support. We get the bug fix value from the free users and financial benefit from the commercial users. The money is turned right around into additional development.

While BitKeeper is hardly a get rich quick scheme, it is self supporting. We've taken no outside investment, the company is built on the backs and wallets of the people who work here, and that's cool. It means there is no outside board of directors in the form of VC's telling us to stop wasting time giving it away. I know that giving it away has helped make it a better product, which is good for everyone, but I'd hate to be in the position of having to justify that decision to a VC before the fact. It's easy to see that things worked after the fact, it's much harder to see that they will work ahead of time.

The bottom line on the licensing scheme is that it was designed to give as much and get as much as possible to and from all parties. Licenses such as the GPL give more to the free users, but give dramatically less to both the original author and to the commercial users. Using GPLed software for everything is like living in a world where the answer for when you have an illness is "here are the plans for the hospital, you can finish building it and check yourself in. Oh, and here's the medical instruments you'll need, you can slice yourself open and poke around. You can do it, good luck!".

Licensed software is more like the insurance model. Nobody pays what it cost to develop the software, that's way to expensive. So everyone pays a little bit and the cost load is spread out. Yeah, for consumer applications like what Microsoft ships, they can get very rich because there is a very large market. But for applications like BitKeeper, it's a tiny market, about a million seats world wide, and there are about 300 different SCM tools out there. Hardly the area to go try and do a free product and hope that support revenue will work. It's just not realistic. There is absolutely no chance that BitKeeper would be anywhere near as good as it is today if we had chosen to GPL it.

Re:The only part of the interview that matters....

[arivanov]

Mandrake has just renamed the classic software business model of support contracts into a marketing entity called club.

This model is the oldest model out there. It created IBM, DEC and the other old greats and it is keeping some of them alive till now.

Yes, it is different from the MSFT style "pay now what we give is what you get, and no support" model. But this does not mean that it is by any means less economically sound. It is actually more sound on the long run.

So I do not think that Mandrake will have any problems with the Club. It is likely that the other linux vendors will take it as well.

Pragmatism

[jdh28]

Linus' approach to BitKeeper (and to everything it seems) is a purely pragmatic one. He has said that if there is a GPL'ed SCM that is at least as good as BitKeeper then he will switch. Until that happens he refuses to let idealism stand in the way of progress.

I think the BitKeeper license is an interesting innovation. My only problem with it, is that if I am using it for free, I am _forced_ to upgrade when new versions become available. Even on an open source project I wouldn't want to be changing something as fundamental as my SCM very regularly. If it aint broke and all that.

john

Re:Pragmatism

[Mr_Silver]

My only problem with it, is that if I am using it for free, I am _forced_ to upgrade when new versions become available.

So pay up and quit whining about it.

You're getting something for free, nothing, nada, zip that someone else has spent time and effort on, why are people not appreciative of this fact any more?

You don't go whinging about a free beer your mate gives you do you? And yes, even that beer will come with conditions (like you're supposed to drink it with him).

Re:Pragmatism

[AxelBoldt]

You're getting something for free,

No you don't. You pay for it with the considerable hassle and expense of being forced to upgrade your basic infrastructure software whenever some company tells you to.

If you download Linux, that's free in every sense of the word. Bitkeeper is not free. Not even free as in beer.

RMS condemning non-free, not BitKeeper itself

[Seth Finkelstein]

Agree or disagree, I believe the phrase BitKeeper ... condemned by free software icon Richard Stallman mis-states the issue. It's not the merits of BitKeeper _per se_ which are at issue. Rather, it is the very idea of the use of it - even if it's technically better!

What RMS actually said was:

Bitkeeper issue
The use of Bitkeeper for the Linux sources has a grave effect on the free software community, because anyone who wants to closely track patches to Linux can only do it by installing that non-free program. There must be dozens or even hundreds of kernel hackers who have done this. Most of them are gradually convincing themselves that it is ok to use non-free software, in order to avoid a sense of cognitive dissonance about the presence of Bitkeeper on their machines. What can be done about this? ...

Linux, the kernel, is often thought of as the flagship of free software, yet its current version is partially non-free. How did this happen? This problem, like the decision to use Bitkeeper, reflects the attitude of the original developer of Linux, a person who thinks that "technically better" is more important than freedom.

Value your freedom, or you will lose it, teaches history. "Don't bother us with politics," respond those who don't want to learn.

That's a very profound statement. It's easy to sneer at it, to dismiss it ad hominem. But he raises important points which deserve to be addressed in depth.

Sig: What Happened To The Censorware Project (censorware.org)

Re:RMS condemning non-free, not BitKeeper itself

[reverse solidus]

RMS's philisophical point is that it's a bit odd to be using a proprietary product as the source control system for a GPL'ed kernel. Especially a GPL'ed kernel that has benefitted so much from existing Free infrastructure.

The practical risk is that Linux, by implicitly endorsing non-free competitors to free products, is helping to cut off it's own air supply.

For example, I suspect that there would be many fewer Free and Open Source projects if CVS were proprietary. Free (speech) infrastructure is a huge boost to software development.

Developers have to make a choice between the short term expediency of choosing a product strictly on its current technical merits, and the long term benefits of encouraging a healthy Free software ecosystem.

You can argue specific cases, but to deny there is a tradeoff is disingenuous.

Extremism and Source Code Control...

[ArthurDent]

RMS is an extremist. Always has been. In that role, he has fought for the complete freedom of software. IMHO, it's an admirable, idealistic view of the world. Having said that, there are times when his extremism causes him to fall right off the deep end, for refusing to make the smallest compromise for the good of the movement.

McVoy is hardly anti-free-software. The very fact that he gives away *anything* for free symbolizes that. (He doesn't have to give anything away.) He makes the simple requirement that the free users use the newest versions for bug reporting reasons. Not a bad idea IMO.

But RMS bristles at even the association with a software product that is ever sold for money. That extreme view causes great debate in the community (this discussion for one) and that's a *good* thing! However, RMS needs to be less beligerent about this one. The kernel needs a stable base in terms of source control. If Linus determines that the best solution is BitKeeper, then that's his decision. RMS has the right to his opinion, but not to insult the intelligence of all of us by tring to tell us that we're all compromising our values by allowing this.

In a similar vein, am I the only one who is sick of RMS whining about the naming of Linux? The accepted name is Linux not GNU/Linux. It's out of your hands, RMS. Live with it.

Ben

Re:Extremism and Source Code Control...

[ryants]

Yet, how does that stated view of commercial software jibe with his beligerent response to the idea of Linus using BitKeeper?

Why do people have such a hard time understanding this?

RMS believes there are certain freedoms, listed at www.gnu.org, that must be met before a software package may be called "free". In English it is unfortuante that "free" has two meanings: in Latin, they are "libre" and "gratis".

The issues is not, never has been, nor never will be about the issue of cost (gratis). The issue is about liberty/freedom.

BitKeeper doesn't even come close to satisfying the requirements for being free (libre), regardless of its cost. Hence RMS' problem with it.

I hope these words were small enough for you.

Re:Linus could not accept CVS/RCS

[tao]

CVS has a lot of short-comings. Most people that use CVS are ready to admit that, I think. Maybe CVS works for gcc, but gcc has multiple persons with write access to the tree. The kernel bitkeeper repository has only one single person with write-access; Linus himself. Linus himself decides what goes in and what doesn't, by pulling from other developer's trees, or by applying normal patches.

Then again, CVS would probably be able to handle the kernel too. But Linus doesn't like CVS one single bit, and since he doesn't, the choice wasn't CVS vs BK, but rather no source-management system vs BK.

Oh, and Bitkeeper is in no way mandatory for kernel-development. Alan Cox and Alexander Viro, for instance, don't use BK, and have no trouble getting their contributions into the kernel. In fact, Viro commented that it got easier, since Linus got relieved of some work.

The fact that gcc was a prerequisite for the Linux-kernel doesn't change the fact that CVS is far inferior to BK. Oh, and it's not like BK is totally non-free. As long as you accept to have your changelogs logged to a server, you get the program for free. Furthermore, the source-code is available, and you are allowed to modify it, as long as it still passes the regression-tests and does nothing to circumvent the open-logging. To finnish off, the program will become GPL if BitMover ceases to exist (can't remember the exact terms, sorry...)

I'm not saying that I'm thrilled with having one important part of the kernel-development process non-free, but I agree when Larry McVoy says that people should spend their time coding a replacement instead of complaining.

I am a CM (configuration manager)

[ACK!!]

The funny thing is that I have to be a part time programmer to get any sort of baseline control out of CVS. Why? Because it is what it says it is.

CVS is versioning control not a tool for complete configuration management.

It is not difficult to keep baseline control over a project with tagging models and proper procedures. However, your cm better come to you with a proper background in scripting at the very least. I am a former sysadmin myself.

For CVS to go beyond its parameters and become a tool for real software control takes some scripting and working.

My goal in terms of giving back to the community is to come up with a standard set of tools for tagging and tracking source code files over a large project. Currently my tools are far to project specific to be of use to the general community. My thought was to expand upon a tool like cvsweb for ease of use.

I have no idea if bitkeeper is any better than CVS for total software control but I will be doing some research as soon as the interview is not slashdotted.

________________________________________________ _

Ah, there's the rub

[Nygard]

The system is no longer GNU + Linux kernel. If it's about giving credit in proportional amounts, then X, perl, Berkeley, Apache, Netscape, and many other major contributors should also be recognized.

I do understand that some of these can be stripped away without impairing the OS, but some of them cannot.

While calling the whole ball of wax "Linux" may be overstating the importance of the kernel, calling it GNU/Linux understates the importances of all the other contributions.

Re:Re-inventing the wheel

[cjstephen]

The effectiveness of the source control system used to develop any non-trivial, multi-developer project is critical.

Creating a good one isn't trivial.

In my opinion, CVS and RCS are far from the best solutions available. CVS behaves like an automation layer on top of the single file version control of RCS, not a change management system for a whole project. It can do most things other systems can do, but you may have to waste effort working around its weaknesses.

I've not used BitKeeper but I have used Perforce, another commercial product which can be used for free on Free software projects, and found the benefits to the development process significant. Making simple things trivial and hard things quite easy helps developers to do the right thing without getting in their way.

Minimising the cost of using the source control system to its fullest effect leads to many improvements throughout the software development process. I'd argue it leads to better, more maintainable code.

As a Linux user and free software advocate I'm glad that Linus is using BitKeeper, rather than CVS. I'd love to see a better free software source control system, but until we've built one I'd rather not sacrifice quality and efficiency in other important projects.

Show me the money

[nagora]

Someone has come up with a way of earning a living under the following circumstances:

1. A lone programmer has a written a new program. Lets say it's an industrial level Cad system for Linux based on 20 years experience as an architect.
2. S/he releases it under the GPL but also writes a nice thick manual which is available as a PDF (let's assume s/he can't afford a minimal print run for this) to go with it.
3. The program works and works well and the manual is good enough to actually use the program; support is not a major issue.
4. The programmer continues on development of the program into new versions which are also GPL'd.

How does this programmer buy food to eat?

This is the flaw in the Free-Software model that McVoy is getting at. If you are a programmer who releases quality work which is distributed for free, how the hell do you survive?

The fact that the GPL does not prevent trying to sell software does not change the reality of distribution of such software in the Internet age.

I don't want to hear solutions based on using the software; the model here is someone who wants to be a programmer, not to remain an architect.

I believe in Free Software; I just can't see how I could ever be involved beyond it being a hobby funded by my real job.

TWW

What about FSF machines' BIOS & firmware?

[dstone]

In reference to the FSF, RMS says:
We have no non-free systems or applications on them now, and our principles say we must keep it that way.

So is this to say that their motherboard BIOS and all supporting microcontroller code, EPROMs, firmware and controller code in their video cards, ethernet cards, etc. all comes with source code?! Impressive. Where do they shop?

BitKeeper is proprietary, and that's okay

[_|()|\|]

The choice is: Enough money to feed the family and pay the mortgage, or give back to the community and make a fraction of what you could have otherwise. It's not always this black and white

Larry's company sells proprietary software. Nothing wrong with that, although I question the wisdom of putting a startup in an area that demands $160K salaries. Also, speaking for myself, I'd take a pay cut to work on free software. (I think I could get by on $100K.)

The complaint over the license has less to do with Larry than with Linus. Linus isn't fanatical about free software. He'd like the world to believe that Linux is successful because he's such a great manager, never mind the GPL.

Mozilla and Evolution are good enough. I won't abandon them just because IE and Netscape are freeware. Linus thinks that CVS is not good enough. RMS would have him resist the temptation of BitKeeper freeware, because it lessens the incentive to improve CVS (or replace it with something better). After all, where would Linux be today if its users and developers had been tempted away by non-commercial SCO or freeware Solaris? But Linus isn't fanatical.

proprietary vendors cloud free, not FSF

[_|()|\|]

Why do FSF advocates insist on clouding the difference between the right to free-use and the right to change the source?

On the contrary, I think the FSF explains the different types of free software very clearly. The common response to a question such as yours is: would you buy a car with the hood welded shut? You're not a mechanic, so what do you care?

Take, for example, Intel Solaris. It was distributed free of charge from its web site, ISOs and all. I download it, install it on my web server, happy as a clam that I have a free, enterprise O/S behind MyPuppySam.org. Oops, the web server crashed, and I have to reinstall. Hmm, the Solaris CD-R is scratched. "Hey, buddy, can I borrow your Solaris disk?" I ask my friend. "Sorry, dude, didn't you read the license agreement? I'm not allowed to lend you my copy. You'll have to go download it again," he replies. Oops, it's not available for download any more, because it's been "deferred," whatever that means. Guess I'll have to pay $50 and wait for the mail man. I guess I should count myself lucky that I can buy the media kit, at all.

Re:Role models

[JordanH]

• Good. I like a role model who has the sense to use the best tool for the job, and who doesn't waste his time with inferior tools.

Best has a context associated with it. RMS believes that best is always free as in GPL'd. Anything else, to him, is inferior.

Linus himself was quoted as saying "Making Linux GPL'd was definitely the best thing I ever did. "

I'm concerned about what I see as extreme pragmatism on Linus' part. Surely, better software, in terms of features and useability, isn't the only criteria for determining it's selection. Price is obviously a major criterion with such internet-developed projects as the Linux Kernel.

As far as I can tell, the BitKeeper license doesn't insure that future versions, perhaps even versions necessary to run on future OS releases, will still be free of charge and without source, we can't be sure that we could make it work on those releases.

Maybe this is just paranoia and there's really nothing to worry about, or maybe not.

RMS is insistent and consistent. Somebody has to be.

I like Barry Goldwater's statement "I would remind you that extremism in the defense of liberty is no vice." To RMS this is about liberty and he doesn't compromise.

It seems to me that there are two choices: The GPL is adopted for purely pragmatic reasons because we can outcompete closed source development. The other is that the GPL is adopted as a principled position.

If we take the first position, then will we stop using Linux the moment something better comes along in terms of features, useability and stability? After all, the existence of BitKeeper proves that, at least in some contexts, that closed source development is superior to Open Source Development models. Doesn't it?

Let me ask you. Should we adopt MicroSoft software if it offers better features and useability? Or... are there other concerns than the narrow "best tool for the job" consideration?

Re:ClearCase is not centralized

[arivanov]

One site is always a master, the rest are slaves for a specific branch. For another branch roles may be swapped of course.

There are also some specifics dealing with the conflict resolutions and obvious races in them. Usually the lusers never see that. It is for the cm to sweat over and sort them out.

Anyway, Larry is right, you are wrong.

RTFM again please.

Absolute nonsense

[FreeUser]

The system is no longer GNU + Linux kernel. If it's about giving credit in proportional amounts, then X, perl, Berkeley, Apache, Netscape, and many other major contributors should also be recognized.

That is just flatout nonsense, and has been rebutted so many times, so effectively, that one seriously wonders how people can still say that with a straight face, much less get modded up to +5 for it. The reasons, of course, are ad homonim ... people dislike RMS for various reasons, his sometimes abrasive personality and lack of tact almost certainly among them.

The core UNIX/Linux operating system consistes of the kernel, various file and binary utilities, a few core libraries, and (arguably) a compiler. It most certainly does not include a GUI windowing system (Microsoft's confusion as to what constitutes an operating system aside), nor does it include a web browser, much less a web server.

The core Linux operating system consists of the Linux kernel and a collection of critical components that were written by the GNU folks long before Linux came along. You may not like RMS's request, or argument, that Linux systems ought to go by the moniker of GNU/Linux, but only someone completely ignorant of operating system design, and of the internals and components of the Linux operating system, would ever argue that "if we call it GNU/Linux we should call it Berkely/GNU/X/Apache/Netscape/Linux." That, or someone who knows better, but has a political ax to grind and is willing to bend the truth more than a little in order to do so.

As an excersize, remove GNU glibc from your (GNU/)Linux system and reboot. If that doesn't make clear the fallacy of your argument, then I suspect nothing ever will.

Call it GNU/Linux, or just call it Linux if you prefer, but please cease and desist spreading absolute nonsense about what is and is not a part an operating system v. what are user space utilities that run on top of an operating system.

Re:Absolute nonsense

[Jorrit]

Well... With some effort you can make a linux distribution that runs on one floppy alone by removing semi-non-essential stuff. Would that mean that everything that got removed is not part of the OS?

Not all OS's have a GUI included. That's true. However that doesn't mean that a GUI is not part of the operating system when it is included. I really see no fundamental difference between a filesystem and a GUI system. Both provide services to the user of the OS. And for a desktop user, both are essential.

Greetings,

Re:Absolute nonsense

[OWJones]

As an excersize, remove GNU glibc from your (GNU/)Linux system and reboot. If that doesn't make clear the fallacy of your argument, then I suspect nothing ever will.

Call it GNU/Linux, or just call it Linux if you prefer, but please cease and desist spreading absolute nonsense about what is and is not a part an operating system v. what are user space utilities that run on top of an operating system.

I enjoy how you contradict yourself here. Funny, but I thought glibc was for userspace applications. Every single textbook, paper, and non-MS-influenced publication I can find -- and there are about 5 of them at my desk, including the Tannenbaum book you claim to cite in another post -- defines an operating system as the (as in singular) program which serves as a layer between the hardware and the application author, and presents a consistent system call API to these applications. They say nothing about it being "the program ... oh yeah, plus some libraries and utilities." Nothing. They do talk about the API the OS provides for those libraries and utilities, but they are separate.

I agree with you that we shouldn't call it "Berkely/GNU/X/Apache/Netscape/Linux". I just don't see how you can selectively ignore your own arguments when they stop being convenient for you.

-jdm

Re:No....

[ncc74656]

Given what happened the last time all the guns were rounded up in your country, I would think you would've learned your lesson by now.

You probably hint on the founding myth of the United States, where an armed militia fought the English colonial army.

It's no myth. Bellesiles was a fraud, in case you haven't heard yet.

I don't see how armed people would have fought the nazi regime

IIRC, didn't the Nazis have some trouble dealing with the Warsaw ghetto, where the locals had managed to procure some small arms for themselves?

when that regime was supported by a majority of those people of that time

Was it? I'd think the people didn't have much of a choice in the matter, once they gave up the ability to defend themselves: "do as der Führer says, or we'll put a bullet in your head." Besides, when is a majority opinion ever of any relevance anyway when it comes to your rights? I don't know if prewar Germany was a democracy, but the United States sure as hell isn't—and I'm grateful that it isn't. Democracy, after all, is two wolves and a sheep deciding what to have for dinner.

And a recent event, where someone got his training and weapons via a shooting club, rather speaks against the general availibilty of guns.

My experience is that you won't find a more polite or upstanding group of people than at the gun range. Since everybody is in possession of firearms of varying degrees of potential lethality, people tend to be on their best behavior—an illustration of the axiom that "an armed society is a polite society."

I don't deny that there are psychopathic individuals in society whose rights ought to be restricted. It does not follow, however, that the rights of law-abiding citizens should be infringed on account of a few "bad apples." Should your free-speech rights be infringed because somebody might say something that would cause offense? Should your right to peaceably assemble be infringed because a few rabble-rousers might go on a rampage through downtown and bust up a few shop windows? If your answer to these questions is "no," then you logically cannot justify infringing the right to keep and bear arms.

Well Well

[evilviper]

Stallman verses the world? I can't believe it! You could knock me over with a feather.

I've heard from several leaders of many highly visible GPLed projects who have essentially said that the biggest problem with the GPL is Stallman. Not that that's not my personal opinion, so don't flame the messenger.