Slashdot Mirror
IBM Seeks Patent On Judging Programmers By Commits
theodp writes "How'd you like to be deemed unworthy of a job based upon a scan of your GitHub updates? That's what proposed in a newly-published IBM patent application for Automated Analysis of Code Developer's Profile, which proposes weeding out developer candidates for certain roles based on things like the amount of changes one typically makes with each commit, how frequently and regularly one makes commits, what hours of the day one makes commits, the percentage of commits with conflicts that one resolves, and the 'depth' of one's commit comments ('shallow', 'mid-range' or 'deep'). Big Blue explains that commit or repository interactions can be used to produce a 'conclusion report' that compares a developer to others who have profiles on the repository, which helps management 'avoid wasted time with ineffective developers."
The guy that spends a week finding a five year old memory bug, that no one has every been able to find is now ineffective, whereas a dweeb performing trivial refactoring is classed as a genius?
The number of commits or lines of code written is not indicative of how good a programmer is. What matters is whether they can deliver a working, efficient program to spec to the deadline. Aside from that - what happens if said candidate doesn't commit any code on public repositories - will IBM just bin his resume without even bothering to interview? This is just so absurd I had to check my watch to make sure it wasn't April 1st.
This is a tool for idiot managers who don't understand programming and who have no clue how to manage programmers, and who want to look like they're in the loop. There's no way I'd stick around in a job where I was judged based on this absurdity.
No sig? Sigh...
They're both full of people who are afraid to commit.
So how many one line changes can I commit per day before someone yells "ENOUGH!!"?
I know too many 'hero' developers that would surely take advantage of this setup.
What if my commits are simply being done because I'm fixing bugs put in by someone else who's doing commits that are bug-ridden?
Yes I know a 'true' developer won't check in code that hasn't been unit tested....guess what, it does happen, and it's those developers who like to see themselves as pounding out 1000s of lines of code that make the difference, not the quality.
I assume this is just a theoretical analysis. Well, eventually a team of coders will get to do it... and guess what, they'll use it of themselves.
:-)
I expect this team to have lots of backstabbing fun
Let IBM have the patents that way no other vendor will add theses to thier products.
"which helps management 'avoid wasted time with ineffective developers.'"
Does it also help developers avoid wasting time with ineffective management who use stupid metrics?
Business methods are NOT inventions, they do NOT advance the sciences or useful arts, and SHOULD NOT BE PATENTABLE!
"I disapprove of what you say, but I will defend to the death your right to say it." - Evelyn Beatrice Hall, re Voltaire
How do you measure the value of a commit when tracking down a one-line bug can take a very long time to find? I don't get paid because of my ability to write massive amounts of code... I get paid because if there is a bug, I am the only one who can track it backwards and find where to fix it.
That sums up the majority of IT managers I've ever worked for. Most of them were admin/business types who'd been moved sideways from other areas and they generally would have had trouble spelling "computer" , much less programming one. But yes, this will be perfect for those sorts of numbnuts who need an easy way to "measure performance".
The guy that spends a week finding a five year old memory bug, that no one has every been able to find is now ineffective, whereas a dweeb performing trivial refactoring is classed as a genius?
The examples are endless so I'll provide another that I 1) witness monthly and 2) have been on either side of. You have the programmer that, when presented with implementing a solution to a complex problem, sits down and draws out class diagrams on paper and erases and redraws while that is cheap and writes the code once many days later and makes small changes to it as it is tested/refined. You also have the programmer that dives right in, may well discover that this was a really expensive solution though easy to code and has it constantly sent back to him after testing only to have to rewrite major portions of it and/or realize then that he/she is reinventing the wheel leading to major changes to try to use another library and, well, this run-on sentence like their work could go on forever while your first programmer was done weeks ago. And who gets the major commit and repository score?
My work here is dung.
"IBM Seeks Patent On Poor Management"
I wouldn't want anyone else to use this idea...
I know fantastic developers with wide ranging practices on commit size, frequency, length of commit comments. While I disagree with many of them, they still produce great software.
Then the programmers will just change how they comment their code.
When I was doing consulting I got dinged because as part of my time breakdown I put Bug Fixes as a line item. After I got dinged for that (Because they didn't want to pay for faulty coding) I changed bug fixes to specification changes. They were happy with that. For the most part good employees what to be honest about their work, however if you make a system where honesty is punished, then people will start stretching the truth.
No one has really found a good way to evaluate a programmers efficiency, mostly because if they do their job right they are not doing the same thing every day so their output will very. There were attempts of measurements like Lines of code which meant programmers started to write more verbose code and doing more copy and paste and less objects and procedures. Then they try to measure by number of projects or sub projects they produced, so the experience developers will jump on and take up the quick and easy projects leaving the ones that require more work to the newbies. Then there reporting by bugs tracked later on, now this had the problem where the developers would take so much time to make sure there weren't any bugs reported that the product never left the shelf or put try and catches that jumped over errors cases and hid the fact the bug happened only to appear in someone else section of code.
Software Developers/Programmers do the job that computers cannot do. So it is difficult to use computers to monitor their performance just because their work is a lot less calculated.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
Considering that IBM has initiated a global outsourcing program, starting in Germany, it is easy to see how automated judging of code quality can go wrong:
Outsourced coders tend to code much more to spec, not using their brain and being sensible, and if automated judging of code quality results in an increase of payments, they will add 10 lines of comments to a x+=1 if the automated judgement likes that.
In addition, when I'm finished finding some bug, very often the resulting code will be shorter than the offending code. This is consistent with the true and tried concept that lines of code are proportional to the number of bugs. I wonder whether automated analysis is smart enough to detect such activity.
Hey don't blame me, IANAB
Years ago I heard of a group of C developers who were judged by a metric which was Reported Bugs / Lines of Code. Lines of Code was determined by a compile time count of lines compiled. Developers started adding extraneous .h files which increased the LoC value and lowering the metric to a releasable state.
You want more commits? Here are more commits.
I changed one character. Commit!
This sounds like it belongs on thedailywtf.com. So would anyone who writes copypasta look good according to these statistics? Would people who write short, carefully considered, effective, reusable code look bad? This type of application has the effect of forcing people to start optimizing their code to maximize the metrics, rather than working to produce an excellent product.
My invention is a system and method for the Automated Analysis of Patent Filings to determine which companies are submitting things that are obvious, anticompetitive, practically worthless, and/or generally inclined to stifle rather than spur innovation. These factors will be used to produce a "conclusion report" that compares a company to others in the field to help job-seekers 'avoid wasting time with trolls and bloated relics from days gone by.'
This reminds me of metrics that awarded lines of code, as if programming were factory piecework (well, maybe at IBM, I don't know).
The only metrics that are important are:
-Does the software work
-Is it maintainable
-Is it done on time
Everything else is balls, or y'know fuelling management paranoia to sell IBM metric tools...
crazy dynamite monkey
Posting anon, sorry. But I did work at Sun Microsystems for a good number of years and it was very much a system that rewarded those that committed (cvs, "putbacks/bringovers", whatever tool of the day was used) often. I always felt that my code commits were being judged in frequency and comparitively.
I don't know if things got better or worse when Sun "merged" with Oracle. I was not there at that point; but before then and just before then it kind of started to suck in terms of code devel politics.
But we prize form over function. Server down - that's okay. Form not completely and correctly filled out? That's a FIRING offense.
.. many tiny changes in many files indicates that concerns have been incorrectly separated in the pre-existing code, which might have been written by the same developer.
Hey don't blame me, IANAB
Remember, if The computer company, who happens to hire a great many programmers, were not granted a monopoly on this technique for hiring programmers, then they would have no incentive to develop the technique.
If you don't grant this patent, why should IBM innovate ways to figure who to hire? What's in it for them?
It is only by forcefully preventing the other companies in the tech industry from doing things like this, for two decades, that technology as a whole can move forward.
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
How do you patent this type of shit. It's an idea, a process, not a product. Even if it were a "product" how does it even come close to being original. Management has been using these bullshit techniques to attempt to measure "performance" since the damn 60s.
Of course it will get a number, because the USPO has lost touch with reality. Everything is original, there is no prior act, no idea is far fetched enough to not get a patent. Hell, I remember when "lines of code" was the measurement of work. As I recall, 25 lines a day was the average. Tripe!
If management looked at development, not as a manufacturing job, but what it really is, craftsmanship then we could do away with bullshit patents like "count the number of times Joe/Jane programmer commit changes". We craft/build programs based on specifications and were we to meet those requirements, then we did our job. Measure completion rates, over/under completion times, quality of the product, but for Heaven's Sake, grow some brain cells and get out of the 19th century mentality of counting widgets made per day as productivity.
Life is a great ride, the vehicle doesn't matter
Everything you do on the Internet is available for inspection and analysis. Sooner or later, someone will come up with a methodology for analyzing photographs on Facebook and Picasa that indicate a person's personality type. And this will be used to filter job applicants.
If you post it online, it will be used to rate you in some fashion. It may be just an HR tool for sorting resumes, or it may be something that is used to select between candidates.
Someone I know just went through about four months of a job working its way through a large company where the job changed from a contractor job (that he was approved for) to a employee position. There were four candidates and four different managers each rated a different person as the top contender. HR took this as an indication that there was no "outstanding" candidate and nobody got the job. The hiring manager was pissed but there was nothing that could be done about it. This is how hiring works in large companies with unions. The actual hiring manager has zero control over the process.
But if you haven't been paying attention, everything you do online will be collected, collated, inspected, analyzed and reported on. There will be services (probably are already) for researching job candidates, prospective spouses, potential roommates, etc. that will use this information to build up a profile that will be useful to some. Is it all BS? Maybe, but people will pay money for it and therefore it is going to happen.
Computers are stupid, much like the folks at IBM that conceived this idea. In the end, all programming metrics are subjective.
Low scoring (i.e. few commits): A new, unique and briliant algorithm that models molecular interaction 10000x more accurately and only requires 10 lines of code and some trivial unit tests.
High scoring (i.e many commits): Changing the labels in 1000 dialog boxes to correct for proper case, making a commit for every 10 changes.
There is simply no way, absent of meaningful artifical intelligence, to measure whether the changes were difficult or meaningful. And, sorry to say, guys, Watson isn't there yet.
Please do not read this sig. Thank you.
Does the analysis software scan it's own source code repository, firing the programmers who have contributed too little to its creation?
+1.
Getting rid of crap and making things more consistent is worth a lot more than adding tons of code that trigger Coverity warnings or need to be refactored or even rewritten later. Programmers who produce tons of LOC are rarely good in my experience. A few weeks ago I spent about a week figuring out a truly hairy bug and added about 10 lines.
I found that longevity is a good metric. Code that doesn't need to be touched again because it just works is good code.
thegodmovie.com - watch it
I've encountered this thinking frequently: "we can't measure everything objectively, so we'll just measure something somewhat related - that's got to be better than not measuring anything."
Sounds reasonable, but is actually wrong and dangerous. Engineers will identify what you are measuring and will change accordingly - which skews the results. And worse some will not change out of a sense of obligation or pigheadedness - which also skews the results because others do. Even if the thing you measured originally had some correlation with what you *wanted* to measure - it will no longer do that once you start measuring.
At least they are patenting it, which should prevent others from introducing this dumb idea.
I figure that this class of mind numbingly stupid patent results from the bastard mating of lawyers who want to patent all human activity with managers who want to have so many patents that they will never be blamed for not having a legal rabbit to pull out of the hat.
So if someone comes up with a meaningless way to judge the "output" of lawyers and makes their lives a living hell, it would be appropriate "justice". They would get to experience the kind of crap that they have been shoveling into the lives of normal humans.
Why is Snark Required?
Infoworld had an article about the fallacies of IT metrics which I copied and have saved. If you want to read why measuring code in lines per day/month is the wrong metric (even if you already know why it's wrong), this article gives more reasons.
The link to the article is this one.
It should be mandatory reading for anyone who utters the term 'metrics' in a meeting.
We will bankrupt ourselves in the vain search for absolute security. -- Dwight D. Eisenhower
Hmm..and in keeping with it's sometimes hard to tell the difference between a feature and a bug, one could also say that the number of lines of code is proportional to the number of features.
SJW: a person who perceives an injustice, and while correcting it, commits a greater injustice.
Every time I read a story like this about performance metrics at IBM, whether it be git commits/KLOCs written/whatever, I have to wonder what kinds of practices engineers there develop. Does engineering quality suffer as a result of gaming the metrics? Are really good engineers helped in their professional development by these metrics, or are they hampered? Are they driven away? Personally, this kind of environment sounds dreadful, but I'm curious what IBMers would have to say about it.
A witty saying proves you are wittier than the next guy.
Code that doesn't need to be touched again because it just works is good code.
The flip side of that is code that doesn't *get* touched because everyone is afraid to modify it.
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
If you think these methods of rating programmers are faulty then this patent is good. Nobody will be able to use these faulty metrics (except IBM) because they are patented ;-)
I found that longevity is a good metric. Code that doesn't need to be touched again because it just works is good code.
Quick, patent your process.
This sounds like a great idea that you could even base bonuses on. You get a point for every line of code that is still in production. After the points are added up, the bonuses are based on the relative percentage of all points.
Question: Do you care if you have someone that has amassed so many points that he doesn't have to work anymore?
If bonuses are based on a percent of sales, then consider the end case where you've got a profitable product that none of your coders have touched in years. Do you care? If sales fall, so will bonuses, providing an incentive to invent something new.
Under this system, newbies would have to code new apps that provide additional revenue.
All ideas^H^H^H^H^Hprocesses in this post are Patent Pending. (as well as the process of patenting all postings)
Seriously, can you really patent the idea of taking metrics in order to evaluate performance? Even with the idea of software patents, that seems overly broad.
So, in other words, the idea is not to say "There Is One Ideal Way to Do Commits!" but rather to use the metrics this algorithm generates to see if a developer's way of working will integrate well with an already-established group. If IBM actually uses the system this way, it might actually make some sense.
Maybe one day they'll be able to patent managers that actually manage and get to know their programmers.
But really I mean by then there will be no such thing as managers and they'll all be Scrum Masters and power will go to the development team itself.
In IBM there's a religion in software that says you have to count K-LOCs, and a K-LOC is a thousand lines of code. How big a project is it? Oh, it's sort of a 10K-LOC project. This is a 20K-LOCer. And this is 50K-LOCs. And IBM wanted to sort of make it the religion about how we got paid. How much money we made off OS/2, how much they did. How many K-LOCs did you do? And we kept trying to convince them - hey, if we have - a developer's got a good idea and he can get something done in 4K-LOCs instead of 20K-LOCs, should we make less money? Because he's made something smaller and faster, less K-LOC. K-LOCs, K-LOCs, that's the methodology. Ugh! Anyway, that always makes my back just crinkle up at the thought of the whole thing. -- Steve Balmer.
Not much has changed at IBM since the 70s, apparently.
Outsourced coders tend to code much more to spec, not using their brain and being sensible, and if automated judging of code quality results in an increase of payments, they will add 10 lines of comments to a x+=1 if the automated judgement likes that.
If it gets Outsourced coders to use comments, then I'm all for it. Too often that tends to be the thing they skip the most. They are too involved in writing the code, too green, or under too much pressure to write and be done, that comments often are thrown under the bus.
This space for rent. All reasonable inquiries will be entertained at proprietors discretion.
I really don't see anything wrong with this, as long as it is used for what it is - a tool. A tool to have a rough gauge of what developers are up to. We don't throw out weigh scales because it isn't an accurate measure of a person's health, do we?
If you have a good and competent manager she/he can use this tool as a rough measure to see what people are doing. Trends probably matter too. If you see a developer suddenly drop off in terms of commits, this is cause to look at other metrics and perhaps view some commits to see what's happening, or to watch a bit more closely in meetings. There are many metrics (lines of code, style errors, poor commit messages, the way a developer interacts with others, work habits, hand inspected code quality, problem solving approach) that each contain a small picture of how well a developer is doing his/her job. A lot of these on their own are worthless, but if you are able to properly interpret the data then the data can be useful.
IMO, the bottom line is that a crappy manager will do a crappy job no matter what tools he/she has. A good manager may be able to use a tool like this to help her/him do her/his job better.
I'd just add and then delete the same file so many times that the company would go bankrupt paying my bonus. Then, when everyone is packing up their things, I'd traverse the office, leaping from desk to desk, shouting "I won, I won!"
This is brilliant
No no, that should read This is brillant!
Certainly on par with a lot of The Daily WTF postings, only now on a managerial level. Joy!
"What in the name of Fats Waller is that?"
"A four-foot prune."
15 years or more ago, I had a friend in a high-level consulting company (as in contract programming stars, not powerpoint pushers). Their usual m.o. was to pay 2x the going rate and expect 5x to 10x the productivity and brains.
They had a contract with IBM fixing bugs in I think AIX or something (lucrative and endless). They shared the bug data base with IBM's own workers, who were working on the same project.
Clever gits that they are, they mined the bug data base algorithmically to look for old bugs which were opened, had many people work on them unsuccessfully, but were eventually closed. They identified one IBM employee (out of dozens/hundreds) who was able to fix demonstrably hard bugs (that other employees had tried and failed to address) consistently.
So, they made him an offer he couldn't refuse.
See: http://www.freepatentsonline.com/crazy.html
The question isn't whether it's a valuable or meaningful metric, but rather, can they make money from it (otherwise why would they patent it)? Maybe it'll end up as part of some application lifecycle tool that they hope to foist on the marketplace (profit!).
Patents aside, it definitely seems like an ineffective metric. A large percentage of my work consists of measuring progress and defects in the software development lifecycle. Before I even consider measuring something, I ask myself, "What problem am I trying to solve?" From reading TFA, it seems like IBM found a metric that was easy to collect/calculate, and is trying to reverse-engineer it into a process or in this case, a patent.
There is no measure of productivity that does not have huge flaws. We'd like to measure quality but there isn't even a reliable definition of quality.
Other posters have mentioned that developers will game the system. That is true. Even if the metric is simply personal observation, people will exceptional and sometimes devious social skills will game the system. That is what office politics and the good old boy network are about.
If trick is to combine subjective and object metrics (even if flawed). You need the objective metrics to catch the social slackers. You need subjective measurements to catch deliberate distortions and account for variations in working styles.
Granted, there will always be idiots who will a tool inappropriately but that does not mean it is a bad tool.
Also leads to a situation where incumbents have a financial reason to be defensive about their code. Try suggesting to your boss that something he wrote years ago has to be re-written if his bonus is riding on it.
"In America, first you get the sugar, then you get the power, then you get the women..." -H. Simpson
I guess we'd have to refactor the immortal words of Wally, "I gonna commit me a new minivan."
Apple tried this a long time ago with predictable results. (well, predictable to most of us anyway)
and I know that lines per day, commits, Mountain Dews chugged don't work as a valuable metric. I know it's different for programmers, but in the rest of the world it boils down to "Get X done by this date." Sure there are status updates, but not daily monitoring of networks routed, cables pulled, systems setup, etc. It's like management is treating coders like data entry people. Just push some buttons and magic comes out! Woo hoo! Don't know how you guys deal with that crap.
Vote monkeys into Congress. They are cheaper and more trustworthy.
The guy that spends a week finding a five year old memory bug, that no one has every been able to find is now ineffective, whereas a dweeb performing trivial refactoring is classed as a genius?
Heaven forbid that you should RTFA, or even the summary, so plow on with your prejudice. Nowere does its say that frequent check-ins would be used as indicating a good programmer. Quoth the patent:
The data analysis component 433 may also include a comparison component 435 for comparing analyzed profile attributes of more than one developer... Determine any other valid statistical comparison between this user profile and other user profiles to find the standard deviation of a profile from the mean profile values. This finds users who may be at the extreme side of the average who may need to be highlighted for support or acclaim
So, Alice and Bob are good programmers and check-in once a week. Charlie is a bad programmer and checks-in ten times a day. Perhaps his manager says to him Why don't you work more like Alice and Bob? They make less frequent changes. Perhaps you need to slow down a little?. Once you have data to analyse, you will be able to find correlations. This is how spam filters work, after all. They do not a priori decide that Viagra indicates spam: the data indicates that is so.
Ne mæg werig mod wyrde wiðstondan, ne se hreo hyge helpe gefremman.
Since his bonus is only a part of the current sales, perhaps that part of the code shouldn't be rewritten.
Instead of spending your time rewriting code that is making paying customers happy, start with the current codebase and make a new product that will make you a large bonus. Re-invent the wheel only to get new customers. If you mess with the existing product, you might impact current customers, even if you make it better. The are many instances where a customer is dependent on a "bug", has worked around it, and would rather live with it vs. something unknown.
All ideas^H^H^H^H^Hprocesses in this post are Patent Pending. (as well as the process of patenting all postings)
They *should* be afraid to modify it. Even a bug fix can adversely affect some customers. They may have worked around and come to depend on a particular bug, and "fixing" it would adversely affect them. Even a code speedup can cause problems in the real world.
If the paying customers are happy with the code, leaving it alone is a sound business decision. If you want to fix it, create a new version (or even a new product) that fixes all the known problems with the current code. After most of the customers of a code base are gone, and it's not worth maintaining it, open source it. (Although, if you are efficient enough, maintaining a code base for exactly one remaining customer shouldn't be to hard.)
All ideas^H^H^H^H^Hprocesses in this post are Patent Pending. (as well as the process of patenting all postings)
What will the "BRAINIAC MBAS" think of next? We live in a world of irony.....based on this logic, We would all read by candle light, as at 10,000 "commits" Tom Edison's people would have never have perfected (or discovered for that matter) the electric lightbulb. What a moronic and short sighted approach to innovation. The MBAs btw- same folks that brought us OUTSOURCING and th ehousing bubble (hows THAt working for you????) are the single greatest force against American Exceptoinalism that exist. Short term profits do not a strong economy or industry make. How about we quantify how much time is wasted by redundant middle level mgmt, and needless meetings, and frankly if the mgmt chain doesn't understand a new technology in 15 minutes, (without a helpdesk person) then send them packing. Same logic. Why waste time trainign peope to drive cars, just have an off shore person with a two way radio for the driver to ask question to (i.e. what does the octagon shaped red sign mean......) wouldn't that be cool. NOT!!! "BAN THE MBA SAVE THE WORLD!"
"Any sufficiently advanced technology is indistinguishable from magic." - Arthur C. Clarke
The backend for GitHire [githire.com] almost certainly constitutes prior art...