Ohloh Tracks Open Source Developers

eldavojohn writes "The startup company Ohloh has a database listing 70,000 developers working on 11,000 open source projects. Their aim is to 'rank' open source developers, which raises some interesting questions about exactly how useful this tracking company is. Questions like, 'Is there an accurate way beyond word of mouth to measure the importance and skill of a developer?' I found it slightly alarming that, to this site, the number of commits (with input from the number of kudos) tells how good a developer you are."

Comments

Accurate? Not for me

[mrslacker]

I don't know how representative it is, or if it might improve over time, but I looked myself up.

I found mentions in 5 projects - _except_ they're all just versions of 2.6 kernel source with the same contribution for an obscure TV card cx88 variant I did. In practice, I'm sure I'm hardly alone in having contributions (mostly in small ways, but sometimes very considerably) to over 100 projects over the years. I guess I have to go through and add some of those projects.

Naw, CBA. At least I can make sure my resume is accurate.

So What Metrics Do You Suggest?

[eldavojohn]

I'm sure I'm hardly alone in having contributions (mostly in small ways, but sometimes very considerably) to over 100 projects over the years. I also don't think you're alone in finding that metrics fail to measure good programmers. My boss constantly asks me for lines of code count from developers. No matter how many times I express this to him, this is not a measure of success or of how good a coder you are.

I tried to think of metrics to relay up the chain (a special thank you to the stat-scm goal in maven) but I come up with some pretty lame ones:

• Code to comment ratio is desired at 1:1 (at least in the commercial world)
• A class/method/function/procedure/module desired size should be defined and rated
• # of Unit tests

As you can see these are the ones that I found could be automatically gathered. And even these have exceptions. Anything else I think of either takes too much time to gather or is subjective. This is tough, I would like to default to peer review but oftentimes I find teammates voicing their personal hatred for an individual or taking into account personal qualities when ranking a developer. Real Life Example: Teammate A is from MIT and teammate B thinks everyone from MIT is a god. Unfortunately Teammate A hasn't done anything but criticize everyone's code without any constructive comments to make it better.

I submitted this story hoping it would open dialog on measuring coding abilities in a semi-automated way.

Re:So What Metrics Do You Suggest?

[JonTurner]

# of penguin stuffed animals in your possession.
# of kernel builds
# of ICQ shouting matches ending in "Nazi!!!"
# of cans of Jolt consumed
# of steps from mom's basement to side door.

Or the old standby, lines of code (including comments, of course).

In short, there's no way to automatically judge the value of a programmer based on silly metrics. How would one score for "bugs not written" or "elegance of solution"/"nasty kludge avoided"?

FOSS is worth whatever the users and the coders say.

Re:So What Metrics Do You Suggest?

[The End Of Days]

What about some slightly deeper tracking into the repository? Figure out how much code a developer committed that had to be changed later.

Re:So What Metrics Do You Suggest?

[morgan_greywolf]

What about some slightly deeper tracking into the repository? Figure out how much code a developer committed that had to be changed later. Not bad, but it depends a lot on the maturity of the project. Many young projects start out with several complete rewrites before they find a workable solution that performs well, is scalable, robust and reliable. And even on mature projects -- a lot of times and entire codebase is scrapped to accommodate new features: think Samba 3.x -> 4.x or GNOME 1.4 -> GNOME 2.0.

Re:So What Metrics Do You Suggest?

[timewasting]

I always hated the lines of code metric because it was so useless. A really good architecture with a good coder leads to much LESS code that looks simple to the casual observer. Similarly, the number of comments might not be very useful as well. Spaghetti often has lots of extra comments, while good code is often self-commenting (at least in my commercial realm). Lots of unit tests are often unhelpful if there are no automated system tests. The best coders balance design, coding and testing in the scope of their deliverables and can effectively estimate how long it will take ahead of time. The art is coming up with a good architecture based on standard patterns and following up with concise, self-documenting code that is easy to maintain and enough tests to know when it works and more importantly when it doesn't. If you can do that in a reasonable amount of time, and less time than what is expected, you're good. If not, you need to be refactored.

Re:So What Metrics Do You Suggest?

[ohlolo]

Software development metrics are not worthless. They are, however, seriously misunderstood. This is partly why we built Ohloh to focus on Open Source: it's the world's largest testbed of available software development metrics.

One challenge to interpreting development metrics is having a clue about what is 'normal'. Just knowing your FOOBAZ count is X doesn't help much. Once you can compare your FOOBAZ count to 100k other developers, it may begin to give you some helpful perspective. Of course, relying on a single metric is myopic - which is why we offer comment ratio, language breakdown etc...

Btw, I agree that human opinion plays a vital role. That's why we also enable people hand out 'Kudos' to their peers - to acknowledge human judgement as well. The Kudo scores are then evaluated using a PageRank-inspired algorithm across all open source contributors.

sounds familiar

[Anonymous Coward]

This reminds me of how academics are increasingly judged. It is more about how many papers and how many other people link to it rather than the quality of each paper's work or the note of the linking party. Accordingly, many authors inflate their 'impact' scores by splitting up papers and publishing nonadvancing science, no-one can blame them for this as many are trying to justify themselves to their departments or are still doing the postdoc merry-go-round looking for new jobs every 18months.

Same here

[jaaron]

Most of my contributions were on website documentation, wikis, or mailing lists, which aren't included in these metrics. At the moment, a lot of my commits are done on repositories not directly available to the public. While I don't really need Ohloh to tell me if I've contributed to a project or not, it's still a little annoying.

And what about contributors who submitted patches that had to be committed by someone else? Or people who contribute by providing help on IRC channels, blogs, forums, or other mailing lists?

While ohloh metrics can be useful, they also need to be taken with a grain of salt, particularly the contributor metrics. They're a bit more useful on measuring a project as a whole (but they still miss a lot of activity).

Would this discourage contributers to open source?

[softwaredoug]

Would this discourage contributers to open source projects? Now if I put on my resume that I've contributed to an open source project, somebody is going to want to look me up. I have to deal with all that baggage when I just wanted something to do in my spare time. Also, I really am not sure I feel comfortable being given an absolute rank. People always bring different skill/approaches to different jobs and I don't think you can arguably say one is better than another. I've worked in teams where everyone respects the different capabilities and limitations of each member. Its sort of like arguing there is an absolute thing known as "intelligence". Is there really such a thing or do we just all bring different skills/perspectives/approaches to the problems we solve? I'd prefer to think the latter, that everyone contributes what they can but has their own limitations. Talking about absolute "intelligence" or "value" seems condescending and elitist.

Number of commits?

[Tarlus]

So in other words, I could commit some of my own code to a CVS repository, find some errors that I missed, fix them, commit it again, decide to add more comments, commit it again, find one more thing I probably could have done differently and then rewrite it, commit it again...

And I would be ranked highly as a great developer?

Re:Number of commits?

[rucs_hack]

Certainly as a diligent one who actually cares about code quality.

Anyone who thinks they can produce bug free code first try is an idiot. Anyone willing to accept that there are always going to be bugs, and actively looks for them is a good coder.

Flawed, but interesting.

[urcreepyneighbor]

Wouldn't such a system assume that everyone uses only one handle - or, their real name - all the time for every project? If so, then a lot of people - who contribute under multiple handles, nicks, whatever you want to call their identities - are going to missed or severely under-rated.

I would rather not have my real name attached to most of what I've contributed. One, because my code is so damn sloppy that it's embarrassing. Two, because I don't want the hassle of my real life - you know, offline - and my, uh, "digital lives" conflicting with each other. Three, if I was easy to find - online - I run the risk of being pestered with silly tech support questions.

UrCreepyNeighbor, while an accurate description of my personality, is one of many identities I have. Same could be said of almost everyone. I'm sure "HotChic17CA" doesn't use that username when she's talking with her grandmother, for example.

L337 development skills!

[hellfire]

So in other words, I could commit some of my own code to a CVS repository, find some errors that I missed, fix them, commit it again, decide to add more comments, commit it again, find one more thing I probably could have done differently and then rewrite it, commit it again...

Your willingness to fix errors, add comments, and do code rewrites puts you in the pantheon of programming gods! The next thing you are going to tell me you actually write your own legible "how to" user guides in PDF!

Re:commits

[krog]

It's as good a measure as any.

You might be right, but it still sucks. And in the case when all your solutions are crap, I think it's dishonest to present any of them as actual, workable solutions.

But what's the point?

[EmbeddedJanitor]

Making OSS dev is hardly a competition sport, so why do this? Kudos is one thing, but that should come from the community, not from some database.

Some people will get a shiny glory and some will feel annoyed bbecause their projects/contributions have not been tracked.