PostgreSQL CVS-git conversion
on
The Rise of Git
·
· Score: 4, Interesting
I had a small role in getting the PostgreSQL project to convert from CVS to git. There's a good summary of what happened at Lessons from PostgreSQL's Git transition. With a pretty conservative development community, the bar for converting from CVS to git was set pretty high: the entire CVS repository had to come through, such that every single release ever tagged could be checked out and get exactly the same files as checking it out of CVS (a little binary diff tool was used to confirm). With around 15 years of history in there, that took some upstream fixes to the cvs2git tool to finally accomplish; it took just over a year to work out all the details to everyone's satisfaction. My checked out copy of the current repo is 272MB right now, so neither small nor giant.
I would say that everyone who works regularly on the code is at least a little bit more productive than they used to be, with the older CVS experts having seen the least such improvement. But some people are a whole lot more productive. I'd put myself in that category--my patch contribution rate is way up now that it's so much easier to pop out a branch to hack on some small thing and then submit the result for review.
And the conversion seems to have improved the uptake of new developers getting involved in working on the code. Having to deal with CVS was a major drag for younger developers in particular, and Subversion is equally foreign to most of them now. As suggested in the article, anyone under 25 will only touch a corporate style CVS or Subversion repo if dragged kicking and screaming into it. As more of that generation rises through IT, old style repos will continue to get shredded at a good rate every year. It could have been any of the DVCS systems that ended up in this position, but git was the one that got the right balance of feature, innovation rate, and publicity. Now that it's got such a wide user base, too, I don't see any of the other VCS software options competing with it successfully in the near future.
I wrote a blog entry on The rise and fall of CentOS that talks at some length (including long into the discussions) about why CentOS has failed to prosper specifically because they are not an open, growing community, and about how they might emulate more successful open-source projects instead.
I just found my Yggdrasil Linux install CD and manual when unpacking some old storage today. I'd happily donate it to this cause, I just need to dig up my Soundblaster card and one of the CD-ROM drives compatible with it.
It's not so much the distance as the traffic. I have been stuck in regular (read: not unusual, happens every work day) hour long traffic jams on the roads around (in decreasing order of nastiness) New York, Los Angeles, Washington DC, Boston, Philadelphia, Baltimore, and San Francisco. Having split my life between owning manual and automatic transmissions during that time, I feel that an automatic is far safer in these extended stop and go traffic conditions. There's about 20M people just in those cities, so I'd estimate about 1/3 the total population of the UK is exposed to this issue here.
A true idiot driver can head full-speed into a roundabout and T-bone you anyway, seen it. The level of idiot drivers where I live is such that I get more nervous about people doing that than T-boning me at a stop sign, because those are at least familiar to them (there is only one roundabout in my regular driving area, and it's a nasty one).
And I expect idiots to blow through stop signs, am looking for them to and normally have more room to dodge them in an open intersection. One badly converted roundabout here is so narrow I have no options if someone fails to yield when entering. I can only speed up rather than also turn away from them, converting the T-bone into a slightly safer rear-end collision. I'm perfectly fine navigating the things myself and like the concept, but assuming you can make an idiot driver smart by putting an obstacle in their way is not a guaranteed success.
The people who donate systems to the PostgreSQL buildfarm go out of their way to include some weird architecture/compiler combinations whenever possible for this same reason. Some of the bugs that show up on these platforms more spectacularly exist on the most popular AMD64 version, too, they're just harder to come across. Processor diversity is great for flushing out some types of subtle bugs.
Yeah, but if you upgrade your 2005 era server to newer hardware, you have to lube up for what your new Oracle license for the more powerful hardware is going to cost you. And if your server from that era is fast enough for running a small database, why go through all that pain?
There is no more OpenSolaris; Oracle already kicked that project in the nads back in August. You might use the derived OpenIndiana distribution instead, but there's a whole different path to uncharted territory.
Basically this means everyone on older hardware will be stuck with Solaris 10 on it until they can plan a migration to something else, probably a whole new server running Linux instead. After all, what kind of idiot would make the mistake of buying new Sun hardware now that they've seen how things are going to work? All of the database server customers I deal with are replacing what used to racks full of Sun boxes running Solaris with Dell + Linux as fast as they can afford to replace the hardware. And my PostgreSQL conversion business is really picking up too. Go Oracle!
I'm with you on the compliance vs. security angle. Recently I've started working with people who want me to certify I comply with HIPAA guidelines for touching private health care data. The emphasis on paperwork over real security practices there is mind-boggling. I'm been put into an uncomfortable position because I can't morally agree to these policies unless I really mean it--which means I'm facing a huge security expense overhead added to my business--while my competitors do a shady job but mark all the checkboxes that they are compliant. Guess who wins the bids?
I work with people who have built successful web application with PostgreSQL as the database store using Ruby + Rails, Java + Tomcat, and Python + various additions; at a really high level, building extensions to Django is even a popular option. Which is preferable really depends on where you intend to go with this. If you want a spot with a smaller web company, Ruby would be more likely, and giant companies tend toward Java. I can't think of a good way to stereotype the companies that prefer Python, but they are out there too.
Oh, you think watching Python on VHS is the hard way, do you? Why, in my day, if we wanted to see a good Monty Python skit we had to go kidnap the performers, construct a set, and buy enough booze to get them drunk enough to perform. VHS? Luxury.
Java as a base language is just fine. The problem is that the sort of problems people like to solve with Java involve things like database interaction, web applications, and user interface construction. And doing all those things turns Java programming into a giant library exploration exercise.
Many other languages end up falling into the same trap if you try to push them toward the same things Java aims at. Python for example has a pretty weak database interface layer. If you want to build a non-trivial DB app, you're likely to add both a database driver plus an ORM solution to make that work sensibly. As a PostgreSQL developer I run into psycopg2 + SQLAlchemy as an example combination. The resulting code is arguably no less "library call sequencer" than a similar solution built using JDBC + Hibernate. And the Java one also fits together into all these other "enterprise" app widgets--application servers and database connection poolers for example. You can do all that in Python, too, but you'll find yourself wandering into the same scale of library mess in the end.
Building an application development framework toolkit that doesn't feel like your language has been turned around to suit the needs of the library is a hard problem. I think one of the reasons Rails has become so successful is that it did a better job than most of avoiding that problem (albeit while giving you a whole different set of problem trade-offs to worry about instead).
The hard way part is where you're given exercises that require extending upon the example given. Providing exercises that are just the right difficulty level for someone learning, easy enough to solve and internalize the learning but not frustrating, is a very difficult thing to do as a technical author. I haven't looked at the book enough to comment on how well it does it that everywhere, but the few examples I checked look well constructed so far.
Also, to address your other comment, Python is one of those languages where looking at code produced by people much better than you can be overwhelming. If you're trying to learn how to do simple work iterating over a list for example, you probably want to just learn that directly--even though someone far better than you might approach the same problem by using a lambda function.
Actually, just by keeping me away from real work I'm quite sure that playing Centipede costs me far more per hour spent trying to master it now than I used to spend in quarters. Once you get decent, a game of that will easily last 5 or more minutes. That's $3 an hour of playtime. I lose a lot more than that when I blow off my job to play with MAME nowadays.
Centipede is a more popular title than all these, and that also falls into the category where it's not the same with a trackball that doesn't feel like the one you'll find in a computer trackball. There are some reasonable arcade trackballs for emulation you can use to make it and Misslle command workable though, as well as some other big trackball games (Marble Madness, Crystal Castle).
Emulating the Tempest spinner is much harder but still possible, by purchasing the same type of hardware and calibrating it obsessively. Never 100% right, but 95% is close enough to enjoy the game, and for the adjustment period to playing a genuine cabinet to be short.
Battlezone, though, the feel of that game is impossible to recreate anywhere else. A sit-down vector Star Wars is in that category too.
Mega Man is also my pick for the appropriate counter-example. The recent re-releases used the old engine, so the controls felt exactly the same. And they perfectly recreated the feel of the old title. The difficulty level, the unfairness of play, everything was just right--just with all new levels and enemies. The way I wanted to throw the god-damn controller at the screen was exactly the same as when I first discovered the game in 1990. The only difference was that I'd be far more likely to damage my slim DLP panel if I actually did it, the old CRT survived the few times I gave in to that anger and let it fly.
The most amazing part is that they eventually released one of the new Mega Man games for the PS3, too, and even that one ended up being the same experience. Didn't even require any Nintento hardware, yet it was once again 1990 and I was pissed at the level designers and alternated between fury to beat them and despair that I should just give up. If Sonic hasn't managed to accomplish the same retro yet updated feat, it's not because it's impossible; they just didn't do it right.
Debian's emphasis on being emphatically free software has made several parts of it more difficult than their Ubuntu counterparts. The first thing I have to do after bringing up a new Debian install (or during installation itself in really bad cases) is go turn on the "non-free" repositories. Then I can find things like the absolutely necessary firmware blobs for my hardware, without which nothing functions.
In the same situation, Ubuntu includes most of that in the default installation. And in the cases where there is something really objectionable from an open source standpoint--such as the Nvidia driver--there is a handy "Install hardware drivers" wizard that makes the "give me easy and fast and I don't care about free" a few clicks away.
Some of the other things done in the name of freedom are less annoying, but even harder to resolve. The renaming of Iceweasel and Icedove are ridiculous. And some very hardcore and questionable decisions made by Debian about the legality of building software that links to multiple libraries has broken some packages in very tricky to resolve ways. See PostgreSQL, OpenSSL, and the GPL as the one I've been stuck trying to get resolved. I wrote a summary of the root problem that documents some history about how Debian has wandered into the position of making life harder for end users by enforcing license trivia. It's not really their fault--the OpenSSL developers are really to blame--but Ubuntu just doesn't care about how free their software is, and in many cases that makes life easier for the end user.
You can't calibrate the drive, what the hell are you talking about. You need to know the type of metal used in the CD to figure out how the photons are going to bounce off of them. Why do you think I pay so much for gold CDs instead of the aluminum ones? There's so much less drift from disc to disc on those, saves me a lot of recalibration time.
There is no such thing as a "direct digital rip". The CD standard doesn't provide one, there are no boundaries on the CD for one to work against, and as stated rip jitter is inevitable. The only question is how the software and hardware involved handle it. The post you were objecting to talks about one of the pieces of the magic used to help with this fundamental problem that you're not aware of, and there are some others too.
Drives that support what's called AccurateStream will guarantee you that they always pick the same spot every time you ask it to seek somewhere, which is the first part of the problem. If you drive doesn't do that, you end up needing to do the overlapping read shuffle described above to figure that out. See EAC Drive Options for more about all that.
Even if you have AccurateStream, there's a second problem: the spot will be the same every time, but exactly where that is can't be guaranteed--it varies based on the drive model. The way AccurateRip copes with this problem to collect a database of CD Drive Offsets. If your drive isn't in their database, what you can do is use a known music CD that AccurateRip has good data on, then calibrate your drive using it to figure out how much you're off by. People submitting those test results is how they compiled the database.
If you have AccurateStream hardware, and you know your drive offset, you can get the same rip every time and match against the checksums that AccurateRip provides. But this is only happening because several pieces of the chain know how to compensate for the limitations of audio CDs encoding, there is no way to get digital data straight off of them usefully.
Slashdot Mirror
I had a small role in getting the PostgreSQL project to convert from CVS to git. There's a good summary of what happened at Lessons from PostgreSQL's Git transition. With a pretty conservative development community, the bar for converting from CVS to git was set pretty high: the entire CVS repository had to come through, such that every single release ever tagged could be checked out and get exactly the same files as checking it out of CVS (a little binary diff tool was used to confirm). With around 15 years of history in there, that took some upstream fixes to the cvs2git tool to finally accomplish; it took just over a year to work out all the details to everyone's satisfaction. My checked out copy of the current repo is 272MB right now, so neither small nor giant.
I would say that everyone who works regularly on the code is at least a little bit more productive than they used to be, with the older CVS experts having seen the least such improvement. But some people are a whole lot more productive. I'd put myself in that category--my patch contribution rate is way up now that it's so much easier to pop out a branch to hack on some small thing and then submit the result for review.
And the conversion seems to have improved the uptake of new developers getting involved in working on the code. Having to deal with CVS was a major drag for younger developers in particular, and Subversion is equally foreign to most of them now. As suggested in the article, anyone under 25 will only touch a corporate style CVS or Subversion repo if dragged kicking and screaming into it. As more of that generation rises through IT, old style repos will continue to get shredded at a good rate every year. It could have been any of the DVCS systems that ended up in this position, but git was the one that got the right balance of feature, innovation rate, and publicity. Now that it's got such a wide user base, too, I don't see any of the other VCS software options competing with it successfully in the near future.
I wrote a blog entry on The rise and fall of CentOS that talks at some length (including long into the discussions) about why CentOS has failed to prosper specifically because they are not an open, growing community, and about how they might emulate more successful open-source projects instead.
I just found my Yggdrasil Linux install CD and manual when unpacking some old storage today. I'd happily donate it to this cause, I just need to dig up my Soundblaster card and one of the CD-ROM drives compatible with it.
There is no plausible explanation for the Joppa/York roundabout that doesn't involve drugs.
It's not so much the distance as the traffic. I have been stuck in regular (read: not unusual, happens every work day) hour long traffic jams on the roads around (in decreasing order of nastiness) New York, Los Angeles, Washington DC, Boston, Philadelphia, Baltimore, and San Francisco. Having split my life between owning manual and automatic transmissions during that time, I feel that an automatic is far safer in these extended stop and go traffic conditions. There's about 20M people just in those cities, so I'd estimate about 1/3 the total population of the UK is exposed to this issue here.
A true idiot driver can head full-speed into a roundabout and T-bone you anyway, seen it. The level of idiot drivers where I live is such that I get more nervous about people doing that than T-boning me at a stop sign, because those are at least familiar to them (there is only one roundabout in my regular driving area, and it's a nasty one).
And I expect idiots to blow through stop signs, am looking for them to and normally have more room to dodge them in an open intersection. One badly converted roundabout here is so narrow I have no options if someone fails to yield when entering. I can only speed up rather than also turn away from them, converting the T-bone into a slightly safer rear-end collision. I'm perfectly fine navigating the things myself and like the concept, but assuming you can make an idiot driver smart by putting an obstacle in their way is not a guaranteed success.
It's also important they don't get involved in a land war in Asia.
You started with a universe having elements in it, did you?
The people who donate systems to the PostgreSQL buildfarm go out of their way to include some weird architecture/compiler combinations whenever possible for this same reason. Some of the bugs that show up on these platforms more spectacularly exist on the most popular AMD64 version, too, they're just harder to come across. Processor diversity is great for flushing out some types of subtle bugs.
Yeah, but if you upgrade your 2005 era server to newer hardware, you have to lube up for what your new Oracle license for the more powerful hardware is going to cost you. And if your server from that era is fast enough for running a small database, why go through all that pain?
There is no more OpenSolaris; Oracle already kicked that project in the nads back in August. You might use the derived OpenIndiana distribution instead, but there's a whole different path to uncharted territory.
Basically this means everyone on older hardware will be stuck with Solaris 10 on it until they can plan a migration to something else, probably a whole new server running Linux instead. After all, what kind of idiot would make the mistake of buying new Sun hardware now that they've seen how things are going to work? All of the database server customers I deal with are replacing what used to racks full of Sun boxes running Solaris with Dell + Linux as fast as they can afford to replace the hardware. And my PostgreSQL conversion business is really picking up too. Go Oracle!
I'm with you on the compliance vs. security angle. Recently I've started working with people who want me to certify I comply with HIPAA guidelines for touching private health care data. The emphasis on paperwork over real security practices there is mind-boggling. I'm been put into an uncomfortable position because I can't morally agree to these policies unless I really mean it--which means I'm facing a huge security expense overhead added to my business--while my competitors do a shady job but mark all the checkboxes that they are compliant. Guess who wins the bids?
I work with people who have built successful web application with PostgreSQL as the database store using Ruby + Rails, Java + Tomcat, and Python + various additions; at a really high level, building extensions to Django is even a popular option. Which is preferable really depends on where you intend to go with this. If you want a spot with a smaller web company, Ruby would be more likely, and giant companies tend toward Java. I can't think of a good way to stereotype the companies that prefer Python, but they are out there too.
Oh, you think watching Python on VHS is the hard way, do you? Why, in my day, if we wanted to see a good Monty Python skit we had to go kidnap the performers, construct a set, and buy enough booze to get them drunk enough to perform. VHS? Luxury.
Java as a base language is just fine. The problem is that the sort of problems people like to solve with Java involve things like database interaction, web applications, and user interface construction. And doing all those things turns Java programming into a giant library exploration exercise.
Many other languages end up falling into the same trap if you try to push them toward the same things Java aims at. Python for example has a pretty weak database interface layer. If you want to build a non-trivial DB app, you're likely to add both a database driver plus an ORM solution to make that work sensibly. As a PostgreSQL developer I run into psycopg2 + SQLAlchemy as an example combination. The resulting code is arguably no less "library call sequencer" than a similar solution built using JDBC + Hibernate. And the Java one also fits together into all these other "enterprise" app widgets--application servers and database connection poolers for example. You can do all that in Python, too, but you'll find yourself wandering into the same scale of library mess in the end.
Building an application development framework toolkit that doesn't feel like your language has been turned around to suit the needs of the library is a hard problem. I think one of the reasons Rails has become so successful is that it did a better job than most of avoiding that problem (albeit while giving you a whole different set of problem trade-offs to worry about instead).
The hard way part is where you're given exercises that require extending upon the example given. Providing exercises that are just the right difficulty level for someone learning, easy enough to solve and internalize the learning but not frustrating, is a very difficult thing to do as a technical author. I haven't looked at the book enough to comment on how well it does it that everywhere, but the few examples I checked look well constructed so far.
Also, to address your other comment, Python is one of those languages where looking at code produced by people much better than you can be overwhelming. If you're trying to learn how to do simple work iterating over a list for example, you probably want to just learn that directly--even though someone far better than you might approach the same problem by using a lambda function.
I was hoping they'd used some Python fork where the lexer was replaced by one using APL characters or something.
Actually, just by keeping me away from real work I'm quite sure that playing Centipede costs me far more per hour spent trying to master it now than I used to spend in quarters. Once you get decent, a game of that will easily last 5 or more minutes. That's $3 an hour of playtime. I lose a lot more than that when I blow off my job to play with MAME nowadays.
There are lots of classic games where you take a little yellow pill to help you on your way.
Centipede is a more popular title than all these, and that also falls into the category where it's not the same with a trackball that doesn't feel like the one you'll find in a computer trackball. There are some reasonable arcade trackballs for emulation you can use to make it and Misslle command workable though, as well as some other big trackball games (Marble Madness, Crystal Castle).
Emulating the Tempest spinner is much harder but still possible, by purchasing the same type of hardware and calibrating it obsessively. Never 100% right, but 95% is close enough to enjoy the game, and for the adjustment period to playing a genuine cabinet to be short.
Battlezone, though, the feel of that game is impossible to recreate anywhere else. A sit-down vector Star Wars is in that category too.
Mega Man is also my pick for the appropriate counter-example. The recent re-releases used the old engine, so the controls felt exactly the same. And they perfectly recreated the feel of the old title. The difficulty level, the unfairness of play, everything was just right--just with all new levels and enemies. The way I wanted to throw the god-damn controller at the screen was exactly the same as when I first discovered the game in 1990. The only difference was that I'd be far more likely to damage my slim DLP panel if I actually did it, the old CRT survived the few times I gave in to that anger and let it fly.
The most amazing part is that they eventually released one of the new Mega Man games for the PS3, too, and even that one ended up being the same experience. Didn't even require any Nintento hardware, yet it was once again 1990 and I was pissed at the level designers and alternated between fury to beat them and despair that I should just give up. If Sonic hasn't managed to accomplish the same retro yet updated feat, it's not because it's impossible; they just didn't do it right.
Debian's emphasis on being emphatically free software has made several parts of it more difficult than their Ubuntu counterparts. The first thing I have to do after bringing up a new Debian install (or during installation itself in really bad cases) is go turn on the "non-free" repositories. Then I can find things like the absolutely necessary firmware blobs for my hardware, without which nothing functions.
In the same situation, Ubuntu includes most of that in the default installation. And in the cases where there is something really objectionable from an open source standpoint--such as the Nvidia driver--there is a handy "Install hardware drivers" wizard that makes the "give me easy and fast and I don't care about free" a few clicks away.
Some of the other things done in the name of freedom are less annoying, but even harder to resolve. The renaming of Iceweasel and Icedove are ridiculous. And some very hardcore and questionable decisions made by Debian about the legality of building software that links to multiple libraries has broken some packages in very tricky to resolve ways. See PostgreSQL, OpenSSL, and the GPL as the one I've been stuck trying to get resolved. I wrote a summary of the root problem that documents some history about how Debian has wandered into the position of making life harder for end users by enforcing license trivia. It's not really their fault--the OpenSSL developers are really to blame--but Ubuntu just doesn't care about how free their software is, and in many cases that makes life easier for the end user.
That's at least a 10 minute argument.
You can't calibrate the drive, what the hell are you talking about. You need to know the type of metal used in the CD to figure out how the photons are going to bounce off of them. Why do you think I pay so much for gold CDs instead of the aluminum ones? There's so much less drift from disc to disc on those, saves me a lot of recalibration time.
There is no such thing as a "direct digital rip". The CD standard doesn't provide one, there are no boundaries on the CD for one to work against, and as stated rip jitter is inevitable. The only question is how the software and hardware involved handle it. The post you were objecting to talks about one of the pieces of the magic used to help with this fundamental problem that you're not aware of, and there are some others too.
Drives that support what's called AccurateStream will guarantee you that they always pick the same spot every time you ask it to seek somewhere, which is the first part of the problem. If you drive doesn't do that, you end up needing to do the overlapping read shuffle described above to figure that out. See EAC Drive Options for more about all that.
Even if you have AccurateStream, there's a second problem: the spot will be the same every time, but exactly where that is can't be guaranteed--it varies based on the drive model. The way AccurateRip copes with this problem to collect a database of CD Drive Offsets. If your drive isn't in their database, what you can do is use a known music CD that AccurateRip has good data on, then calibrate your drive using it to figure out how much you're off by. People submitting those test results is how they compiled the database.
If you have AccurateStream hardware, and you know your drive offset, you can get the same rip every time and match against the checksums that AccurateRip provides. But this is only happening because several pieces of the chain know how to compensate for the limitations of audio CDs encoding, there is no way to get digital data straight off of them usefully.