Finding New Code

tabandmountaindew writes "Too much time is wasted re-implementing code that someone else has already done, for the sole reason it's faster than finding the other code. Previous source code search engines, such as google codesearch and krugle, only considered individual files on their own, leading to poor quality results, making them only useful when the amount of time to re-implement was extremely high. According to a recent newsforge article a fledgling source-code search engine All The Code is aiming to change all of this. By looking at code, not just on its own, but also how it is used, it is able to return more relevant results. This seems like just what we need to unify the open-source community, leading to an actual common repository of unique code, and ending the cycle of unnecessary reimplementing."

Comments

Java Only for now, more coming soon

[jsnipy]

I wonder how fast we will see other types of code.

Re:Java Only for now, more coming soon

[niconorsk]

From the FAQ, they will start adding new types of code starting Summer 2007, and will have a poll to discuss which languages to support.

Re:Java Only for now, more coming soon

[b100dian]

I hope they will support Ruby!

I call bullshit on this

[analog_line]

I'm not a coder, but my impression of the vast majority of coders is that they reinvent the wheel because they believe that everyone screwed up their wheel implementation and if no one is going to do it right, they should.

Re:I call bullshit on this

[introp]

There's an old adage in the racing business: if you're building a parade float, buy your wheels. If you're racing a 300 kph Formula One car, consider building your own. If you place very high demands on a component because it is at the core of what you do, the stock component may not be good enough.

So, in software, if the "wheel" is at the core if your product, you may have to re-invent it to get exactly what you need. This is not because everyone else screwed it up, just that the stock "wheel" serves 90% of the features for cheap. Look at the Mac iPhone's OS, Cisco's move to (cheaper, memory-wise) VxWorks on the WRT54, etc.

Re:I call bullshit on this

[Anonymous Coward]

What I tend to find is that a lot of existing code is not sufficiently abstracted enough from the original implementation, or contains too many dependencies on either the rest of the original codebase or has external dependencies that I am either unable or unwilling to satisfy. If I have to import 3000 lines of structure definitions, macros and constants and modify my code just to be able to re-use a couple of functions, it's likely going to be less work and effort in the long term to write my own implementation.

Re:I call bullshit on this

[ajole]

The first thing you have to do is get people to READ code! How many times have you pushed a door that had a big sign that said "Push"?

Go Python!

Re:I call bullshit on this

[max.capacity]

I don't think that is true in all cases. Often others' code has subtle bugs, or implementation details that cause problems later on. While you might get a boost in productivity using someone else's code, that can quickly be lost when you have to go back and either fix your code due to some side effect or fix the other person's code as a result of bugs, short cuts, etc.

Moreover, with differences in coding styles, it can be hard to wrap your head around someone else's code. A solution is to use standard idioms to make the code more readable.

Re:I call bullshit on this

[j00r0m4nc3r]

I think a system like this might work if it has a user feedback system, where particular authors get a good reputation by positive feedback. So you know that an implementation is probably good if that author has good ratings. Think about system libraries. Nobody (well, mostly nobody) writes their own implementation of system libs, because they trust (usually) the implementation provided by the OS or the compiler. Why do they trust? Why is a Microsoft routine more trustworthy than S00p3rC0d3r's? Just because those functions have been tested by users over and over again. And they're (usually) well-documented.

Re:I call bullshit on this

[ajole]

er.... a sign that said "Pull".

It's early here :)

Re:I call bullshit on this

[thisIsNotMyName]

Coders typically reinvent the wheel because it is usually easier to rewrite something than it is to learn how the existing code works. That actually leads into my main problem with using code search like this to try to promote more code reuse, namely, trust. The search engine is going to need to provide some way for me to make a judgment on how well written and bug free I can expect the code to be. Joel Spolksy of Joel On Software has said that if you are writing an application that is mission critical to your business you need to control not only the application layer, but also the layer below that, whatever it may be. If I'm going to be giving up control (of creating) of pieces of even my application layer, I need some assurances that the code is high quality.

Re:I call bullshit on this

[tb()ne]

I'm not a coder, but my impression of the vast majority of coders is that they reinvent the wheel because they believe that everyone screwed up their wheel implementation and if no one is going to do it right, they should.

I've seen that happen numerous times but it is often just an excuse to cover the fact that they are too lazy to read and comprehend someone else's code.

Re:I call bullshit on this

[NotQuiteReal]

I often Push on the Pull doors, just to see if they work.

Often the Push/Pull sign is just some control freak placing arbitrary rules on things. So what if you clock a little old lady on the other side once in a while.

Freedom to swing both ways has its price!

Re:I call bullshit on this

[homey of my owney]

It seems to me, the real problem is this; http://developers.slashdot.org/article.pl?sid=07/0 2/03/2147222/ discussed here just last week.

Re:I call bullshit on this

[Anonymous Coward]

True, but many times they are justified! When software is done right, people use it! Coders re-use it! How many programmers today are working on reimplementing BLAS or ATLAS compared to how many are using those products? For Windows programmers, how many are using .NET today? That's all re-usable code done well! If there were more great reusable code out there, people would be re-using it, that's common sense. Another problem is that many of the things re-usable software handle are very abstract, such as vector/matrix operations, but at some point, code needs to be specialized for a particular task.

Re:I call bullshit on this

[matt74441]

I've never really coded anything overly complicated outside of a school environment, but when I do I usually like to avoid using someone else's code. If I do, I usually only ever use it for reference purposes, so I know if I'm on the right track. Probably not the best way to do it, but for me, its not enough for me to just take someones code and integrate it into my project, but I need to understand it. Its considerably easier for me to debug my own code, then to try and debug someone elses, especially when their code doesn't completely suit my needs.

Re:I call bullshit on this

[maxwell demon]

When following your link, I get "Nothing for you to see here. Please move along." While I agree that not seeing the code is indeed a major obstacle to reusing it, I strongly doubt that it's what you meant to link to.

Re:I call bullshit on this

[wizbit]

I agree, but with a slightly less hostile view to coders, since I am one:

I like to code. I'm a programmer because I like to write code. If given the choice between learning and re-implementing somebody else's solution - for non-trivial tasks, this is usually the best way to go, but also takes a long time - and writing my own solution, I tend to gravitate toward the latter. Why? It usually is faster development-wise. Probably not in the long-run, since well-known code is also usually well-tested and my solution could invariably have bugs that the other would not. But I'd be intimately familiar with whatever I come up with, and it lets me write code immediately, instead of learning somebody else's and trying to meld our two styles/approaches.

It's not arrogance; it's a decision with a slight bias against delayed gratification.

Re:I call bullshit on this

[Lazerf4rt]

I don't think people are giving programmers enough credit for having common sense, and this project to reduce code re-implementation sounds pretty idealistic. I don't know if I call bullshit on it, but I smell a few flawed assumptions.

First flawed assumption seems to be that the hard part in re-using code is simply to find it. But that's crap. When code is in the form where it can even be re-used, it's called a module, and a big chunk of this code is the module's interface. The interface is what lets you re-use it. But there are huge differences between interfaces. There are different calling conventions, different parameter orderings, different limitations on thread re-entry, different permissions on order of things you're allowed to call, and entirely different approaches to specifying the interface. A streaming library can have a public function Read() or it can have a pair of public functions BeginRead()/EndRead(), and there are many valid reasons for both cases.

Point being, you often have to refactor a module's interface before you can fit it into your project, and depending on the size and purpose of that module, the refactoring might be just as slow as writing a new implementation.

Second flawed assumption is that developers aren't even able to find the code which they need to find today. But that's crap too because there are a lot of great, re-useable libraries that programmers already commonly know about, or can easily find through Google or Sourceforge. First of all, the standard C/C++ libraries give you a lot. Then there's zlib, curl, glut, Allegro, and hundreds (thousands, depending on your standards) more, depending on what you're doing.

Come to think of it, when you really want to re-use code, you look for libraries, not source code. Searching for source code mainly helps people who want to learn programming.

I know that the site linked in the summary only contains Java code right now, and I'm mainly focusing on C/C++ here but I think a lot of what I said applies. (Don't tell me that Java's automatic turning of each class into a monitor solves the thread re-entry problem, because that really just substitutes one problem for another.)

Re:I call bullshit on this

[homey of my owney]

Indeed, it is not. Why Software is hard: http://developers.slashdot.org/article.pl?sid=07/0 2/03/2147222

Re:I call bullshit on this

[oliverthered]

The iPhone isn't really a good example other mobile oses (e.g. Linux) could have done the job just as well if not better. There are only two reasons for apple to use their own OS branding and vendor lockin.

Re:I call bullshit on this

[Anonymous Coward]

I'm pretty sure the number of people in F1 racing who build their own wheels is on the low side of zero.

Re:I call bullshit on this

[Anonymous Coward]

I'm not a coder, but my impression of the vast majority of coders is that they reinvent the wheel because they believe that everyone screwed up their wheel implementation and if no one is going to do it right, they should.

I am a coder, and having seen a decent amount of open source code, I certainly do believe this to some extent. Sturgeon's Law definitely applies to source code. In fact, I'll often reinvent the wheel multiple times, because even I can't get it right the first time, and because the requirements often change slightly.

More to the point, actually writing the code is the easy part. I'd much rather see a search engine for algorithms. (say... http://www.nist.gov/dads/) Once I've written the code one time, once I know how it needs to work, writing code to do it is easy. It just takes a little time.

I really do think that the whole "rewriting code is bad" thing is bullshit below a certain level. If the entirety of the algorithm will fit in your head all at once, there's no harm in rewriting it.

Re:I call bullshit on this

[cant_get_a_good_nick]

There was a Joel On Software post a while back that explained this.

Most applicable quote:

If it's a core business function -- do it yourself, no matter what.

Re:I call bullshit on this

[Anonymous Coward]

I believe programming is an art.
If I'm a painter and I want to create a painting with a women wearing a hat I wouldnt grab someone else's painting that has women in it and draw hat on it.
I'd rather make my own version from the scratch because painting is my passion not an obligation and I enjoy doing it.
Thats why people re-invent the wheel most of the time because they just love doing things their ownself :)

Re:I call bullshit on this

[pla]

I've seen that happen numerous times but it is often just an excuse to cover the fact that they are too lazy to read and comprehend someone else's code.

Although I agree with you in essence, I find it odd to call someone "lazy" for doing more work...

The real problem here comes from how much time it takes to really understand someone else's non-trivial code. A good coder, regardless of some mythical level of "laziness", needs to ask whether it will take longer to verify someone else's code, or to roll one from scratch.

Take something like a DCT. You can do the core calculations a million different ways, all with their own caveats and performance implications. You can do the overall whole as a set of functions, or recursively, or as a single function with a set of fairly unintelligible loops (or even as a wrapper to a known-good FFT). If the author of some code I wanted to use took a fairly novel approach to the problem, it might take me more time just to figure out how it works in general (nevermind actually verifying that it works correctly) than to write my own version.

Re:I call bullshit on this

[tanstaafl4.5]

Oblig Seinfeld: "Not that there's anything wrong with that."

Re:I call bullshit on this

[Tim Browse]

Often others' code has subtle bugs, or implementation details that cause problems later on.

I hate to be the one to tell you this, but your code does too.

Re:I call bullshit on this

[Anonymous Coward]

Thank you for this comment. Its definitely something I'm going to keep in the back of my head. I've been looking for a good way to verbalize exactly what you said.

Re:I call bullshit on this

[arivanov]

You are misplacing your blame here mate. The blame is with pseudomanager wannabees, not coders.

In order to use someone's else code you have to perform some research, evaluate it and even do a few proof of concept runs. Once this has been done you generally have to freeze the third party code as a part of your platform and develop on the frozen version. You also have to reevaluate this third party code before planning every release cycle. Research, evaluation, poc and maintenance of said third party code takes resources. It also takes some serious knowledge and experience as the person who does that needs to be able to tell good code from bad as on him depends the entire project.

The telltale signs of a wannabee beginner promoted into management who does not have a clue are:

• They do not allocate the time for research and poc work at all or allocate clearly insufficient time.
• They do not freeze the platform components so that the developers have a clear understanding on top of what they develop.
• They do not allocate time to maintain the frozen components and perform evaluation of newer versions before planning new releases. As a result old, buggy and bad third party code is used release after release.
• They consider any code but the one produced by them (or their project) inferior and look ait it with derison. To demonstrate that, instead of assigning the most senior and experienced coders to platform/third party evaluation they assign the most junior ones. These usually do not have enough knowledge and experience to distinguish good code from bad.
• Worst of all, after failing with third party code once due to their managerial incompetence they fall back to "not invented here, let's reinvent the wheel" instead of correcting themselves (or being fired, which they rightfully deserve).

Re:I call bullshit on this

[Jerome H]

I hate to be the one to tell you this, but your code does too.

I agree with you but it is much more easier to debug your own code than that from a stranger and thus faster. I think that was grandparent's point.

Re:I call bullshit on this

[tb()ne]

Although I agree with you in essence, I find it odd to call someone "lazy" for doing more work...

Ok, I can see how that might seem odd. Laziness stands in contrast to motivation. And many programmers will be much more motivated to write their own code than to read someone else's, even if it takes longer for them to write it from scratch.

The real problem here comes from how much time it takes to really understand someone else's non-trivial code. A good coder, regardless of some mythical level of "laziness", needs to ask whether it will take longer to verify someone else's code, or to roll one from scratch.

I agree. That is the question that should be asked. The problem is that the answer tends to be easily biased toward rolling it from scratch.

Take something like a DCT. You can do the core calculations a million different ways, all with their own caveats and performance implications. You can do the overall whole as a set of functions, or recursively, or as a single function with a set of fairly unintelligible loops (or even as a wrapper to a known-good FFT). If the author of some code I wanted to use took a fairly novel approach to the problem, it might take me more time just to figure out how it works in general (nevermind actually verifying that it works correctly) than to write my own version.

Out of 10 hits on "DCT algorithm", you could probably eliminate the 5 horribly documented ones in about 2 minutes. Of the remaining 5, it probably wouldn't take much more than about 10 minutes to determine which ones aren't the algorithm you want and to estimate how long it would take to read/verify the one that looks like the best match.

Obviously, the utility of reusing code depends on how non-trivial it is to both read and write. Although, for some large code bases, you may be better of just running unit tests against the existing code that trying to dissect it first.

Oh, and wrt verifying the existing code - you would need to verify your own code too, right?

Re:I call bullshit on this

[Matthew Bafford]

I often Push on the Pull doors, just to see if they work.

Often the Push/Pull sign is just some control freak placing arbitrary rules on things.

You're modded funny (and that might be your intention), but I do the exact same thing. Arbitrary rules piss me off. What annoys me more, though, is handles (that are easily pulled) on the side of the door you can only push. Talk about poor user interface design!

Re:I call bullshit on this

[Anonymous Coward]

Searching for source code mainly helps people who want to learn programming.
Actually, with today's error-flagging and correction-suggesting IDEs, lots of newbies search for code because they don't want to learn programming. They just want to stitch some modules together. Unfortunately, they don't realize this, so they wind up cutting and pasting huge wads of code from various sources together and the IDE guides them through stitching it together. The end result is often a module that doesn't work because the "developer" doesn't know the difference between JDBC and JNDI, so they can't figure out why they can't get a connection to the database. Then they wonder why, when they plop their 4000-line EJB down in front of somebody, the answer is usually "Uh, sorry Prakesh, I'm afraid I can't help you with that..."

Re:I call bullshit on this

[lhand]

"I often Push on the Pull doors, just to see if they work...."

Yup. Same here.

I once walked up to the doors in a parking garage. The sign above the door knob said turn and push. So I did a pirouette and gave the door a shove.

It opened. You should have seen the looks on the faces of the people around me.

Anyway, back to the main topic.

I suspect the "not invented here" problem is why these code repositories aren't used. Most of us programmers don't want to use someone else's code because we have to first figure it out, and then use/interface with it. We tend to think it's easier to just re-write the thing. You know, just like the complete re-write for version 3.0 of a program.

Not that I agree that's the best way to go, mind you.

Re:I call bullshit on this

[try_anything]

Searching for source code mainly helps people who want to learn programming.

Ugh.... That makes me sick just thinking about it. Because there is no hyperlinking, there is no way to build human quality judgments into the search. The search results will be of average quality, which is poor.

That means that programmers with good taste will refuse to wade through the sludge returned by code searches. They will much prefer to wade through lots of good, irrelevant code looking for something relevant than to wade through lots of bad, relevant code looking for something good.

The availability of code search tools will have no significant consequences, because programmers who do who prefer to use code search tools instead of the alternatives (books, documentation, and carefully selected examples from high-quality open source projects) will produce very little of value. Talented programmers would stunt their growth by immersing themselves in mediocrity -- but they won't consent to do so. Average programmers produce code at a much lower level of quality than the resources they use, so average programmers using other average programmers as resources will devolve into helpless, useless idiots.

Of course nothing so dramatic ever happens. People will quickly learn to ignore these tools and keep using their current methods.

Re:I call bullshit on this

[StikyPad]

First flawed assumption seems to be that the hard part in re-using code is simply to find it.

Second flawed assumption is that developers aren't even able to find the code which they need to find.

I guess the third flawed assumption was mine, that the first and second assumptions were going to be different?

Re:I call bullshit on this

[jonadab]

> I think a system like this might work if it has a user feedback system,
> where particular authors get a good reputation by positive feedback.

Unfortunately not -- or, at least, not unless something non-trivial is done to make it work that way.

Most of the feedback is generated by brand new programmers fresh out of their first semester of high school web design class looking to get up to the next level. Those are the people who are looking for sources they don't know about, so they're exploring, using the search engines, finding what they can, and giving feedback. Established programmers *know* where to find good code, but new programmers are still looking around, and consequently still finding what's out there. They don't yet have the ability to tell good code from bad, so sites like Matt's Script Archive get plenty of good feedback. (Don't believe me? Today, a Google search for perl scripts turns up that site *first*. Before search.cpan.org, before perlmonks, before everything.)

Sure, if you ask anybody who knows the language in question pretty well, he'll steer you in a better direction. But the problem is that Google, at least with its current technology (which, as search engines go, is state of the art at this point in time), can't tell the difference between informed feedback and uninformed feedback.

Remember, nothing naturally falls into a well-organized state. Serious effort has to be put into *getting* it into a well-organized state, or it'll be chaos. This law of entropy was first stated in the realm of thermodynamics, but it is true generally: in the absense of a deliberate directed effort to the contrary, everything in the universe tends toward worthlessness. Wherever you see order, it's the result of hard work on someone's part.

Some kind of feedback on the feedback is needed -- but it needs to work a good deal better than slashdot's meta-moderation. I'm not sure exactly how it needs to work, but it needs to be able to assign value to different sources of feedback, and possibly to sources of the feedback on the feedback as well. And some reliable seeding method is needed, a bootstrap if you will, to anchor the thing so that certain reliable sources of good feedback can be established.

Re:I call bullshit on this

[jonadab]

> Although I agree with you in essence, I find it odd to call someone
> "lazy" for doing more work...

That's false laziness. True virtuous laziness drives a programmer to create tools that make his work easier over the long term, and a collection of personally evaluated known good sources of quality code is as good an example as any. When a programmer doesn't *bother* to do something that would make his work easier over the long term, that's the vice of false laziness.

A programmer must be careful to avoid false laziness. Also false impatience and false hubris.

Re:I call bullshit on this

[Watson Ladd]

I would like to point out that system libraries can contain coding errors themselves. One example is Apple's malloc(). It opened a file for debugging with the programs permissions, leading to a security hole in all setuid programs that need to allocate memory.source

Re:I call bullshit on this

[AaronLawrence]

Unlike Programmer Bob's reimplemented malloc, which contains 12 new security holes, degrades to O(n) performance and fragments memory badly.

The point is: system libraries are ALMOST always better than anything you will be able to write yourself, unless you already specialise in writing such libraries.

Re:I call bullshit on this

[kaidadragonfly]

This doesn't quite apply if you have the source code, since his point was that you should rely on nobody else for your core business functions, and if you have the source code, you don't need to rely on anyone else.

Re:I call bullshit on this

[SageMusings]

Standard QA, am I right?

Re:I call bullshit on this

[epine]

The point is: system libraries are ALMOST always better than anything you will be able to write yourself, unless you already specialise in writing such libraries.

If it were that simple, theory and practice would not sleeping in separate beds.

I recall an interview where Theo claimed that a substantial number of bugs discovered in the OpenBSD code audit were created because the programmer failed to understand the library or system API they were programming against. We're talking about simple libraries here, such as the C string library and basic system services such as network programming. Now that I think of it, I have a 1006 page reference entitled Unix Network Programming. The simple API and implied infrastructure is rarely as simple as it is made out to be. The library might be correct in itself, but the program won't be unless the correctness of the library is correctly understood. Not even counting the cases where the API is stupidly designed by those who should have known better, such as the C string functions.

It's rare that a piece of code does exactly what you want it the moment after you download it. If it proves unsatisfactory, do you maintain your own patch branch, or do you try to push your required changes to the upstream maintainer? You might find one library that you can live with without any changes, but what if it conflicts in some way with one of the other five libraries you also need to incorporate? Do all the library components have compatible memory management? Are error conditions handled consistently? Do they all use different string functions?

Some of the best code around has been authored by serious do-it-yourselfers such as Knuth and DJ Bernstein, both of whom wrote complex systems essentially bug free. In Bernstein's case, his strangeness and personality make it hard to use his binaries, much less incorporate his code.

A good example of code seriously engineered for re-use is the Boost C++ library. What most programmers seem determined to ignore is that re-use imposes a serious abstraction penalty. Source code composes nicely only when it was designed to do so, but that design represents a complexity of its own kind.

A final factor I'll mention is that the computational infrastructure improves exponentially. Perfectly sensible coding decisions made when sendmail or vi was authored to cope with extremely limited hardware make far less sense five or ten years later. Why would a programmer wish to adopt a block of code who design reflects constraints which no longer exist (unless the original coder took this explicitly into account). Even Knuth made a bad choice regarding character encoding to satisfy a performance constraint related to machine word size, which created a lot of extra complexity later in grafting in the Asian languages.

It's time we put to rest the notion that code reuse is somehow trivial or automatic. Any simplicity that matters is achieved by hard work.

Re:I call bullshit on this

[Raenex]

Funniest quote:

The only exception to this rule, I suspect, is if your own people are more incompetent than everyone else, so whenever you try to do anything in house, it's botched up. Yes, there are plenty of places like this. If you're in one of them, I can't help you.

Re:I call bullshit on this

[snathaidmhor]

Sounds like we need a publicly-editable module search repository, ala a wiki, but specific to programmers' needs.

Let's see, it could include abilities to:

• add pseudocode
• detail the specs for the interface (e.g. parameters and return values)
• add different ports for specific programming languages

Of course, this would have to be free (supported by ads?) and really easy to use.

If someone were to create this and it took off just like Wikipedia, us coders would not have to reinvent the wheel all the time...

Any takers??

Re:I call bullshit on this

[Anonymous Coward]

I use existing code all the time, but mainly as a learning tool.
I generally avoid code re-use from public sources. There are 3 main reasons for this in order:

1) Copyright issues: I cannot slap any code I found on the Internet into a client's project. What if in 2 years it is revealed that the "collaborator" distributing code freely without any attachment was actually working for SCO? ;)

2) Learning: When facing a new field, I will sometime re-code modules, just to know exactly how it works. If a problem occurs in or near that module I like to be able to say "Oh yeah, I know exactly where that exception is causing problem". Also, by human nature a less known module will always be the prime suspect, since it is not yours.

3) And finally but lastly, yes in my experience too often "free code" (easily found) is done by junior coders or students eager to please or add to their resume. Such code will often have parts specific to the project being studied. I will bypass "bad code" comments, many others made great comments here on that. Variables names, coding style, architecture, comments; all can be corrected easily if the algorithms are well done.

I am currently working in a small project constructed at 90% of re-usable code. It is obviously a bunch of patches put together.
Changing coding styles in each file.
Bad variables names too often created for another field (a network packet named "ClientBlueTeamChatData" when used for inventory).
Numerous cryptic and misleading comments, since code was changed for the local project.
Entire functions commented out without any explanation (// Works if in comment)
Numerous memory leaks (this is recurring in free code).

Re:I call bullshit on this

[eggywat]

Could this problem be partly attributed to the wide use of the imperative programming model.

Checkout the channel 9 interview in which such luminaries as Anders Hejlsberg (designer of C#), Erik Meijer (designer of LINQ), Herb Sutter (C++ expert, and author of the seminal DDJ article "The Free Lunch is Over: A Fundamental Turn Toward Concurrency in Software") and Brian Beckman (one of the first members of Microsoft Research). They discuss subject of the interview was software composability and the future of programming.

Neil Bartlett http://neilbartlett.name/blog/?p=11 says

"As you would expect from such a meaty subject and heavyweight participants, the video is fascinating. A theme that runs through the interview is the difficulty of composing software into large systems, due to problems that seem to be inherent in the imperative programming model. A large part of the problem is side effects."

"Software reuse can be risky business for an imperative programmer. If you call someone else's method or procedure (or even one that you wrote yourself a little while ago) you are never really sure what background effects are going on. It may work most of the time, but what happens when you combine it with another method that also has unknown effects? I'm reminded of the Joker's evil plot in the 1989 Batman movie: ordinary household items like shampoo and toothpaste, while harmless on their own, would combine into a deadly toxin when used together. Using libraries and code from multiple sources creates bugs - many of them hidden until they are combined for the first time in the user's PC. I believe that this phenomenon goes a long way to explaining various programming pathologies, such as the endless re-invention of the wheel and the "not invented here" syndrome. It can actually be worthwhile to rebuild whole swathes of functionality already created by somebody else, just so that you can understand and control it yourself."

I think he has a good point.

Add to that the fact that code you intend to reause may have bugs, or have been created for a slightly different purpose to your intended use etc and you have a recipe for reinvent the wheel.

http://bymyreckoning.com/

Three things that make this article suspicious

[knightmad]

1) "Java Only for now, more coming soon!"
2) "Alpha"
3) The linked article is a "product announcement" on Newsforge

This is slashvertisement for a vaporware product. Although this is promising, there is nothing concrete there to call it "what we need to unify the open-source community", not even an alternative to Google codesearch.

Btw, is alpha the new beta?

Re:Three things that make this article suspicious

[kinglink]

No alpha is the old beta.

An unworking game should never have been called alpha. When companies beta test software it tends to be what early alpha testing is. When companies release software that's what used to be called beta testing (and in the old days there was gamma testing in addition).

However beta (and gamma) took time and money... and what company wants to spend that.

Re:Three things that make this article suspicious

[fithmo]

Btw, is alpha the new beta?

No, this is a true alpha - see, you can tell because the logo already has a gradient, but it lacks the difinitive shiny table reflection that today's modern beta sofwate requires.

Are we really making it better for us, or worse?

[hacker]

If we create this grand, uber code-searching portal, which can search the context of the code, aren't we making it easier for commercial entities to go ahead and and pick and choose those bits of code to use in their products, knowing full well that they're going to violate the GPL (or other OSS licensing models) by doing so?

I've talked to NO LESS THAN a dozen commercial companies in the last 2-3 years where they're actively taking FOSS source and incorporating it into their products, because.. (and I quote) "..Its freeware, so we can use it however we wish."

The licensing differences between "freeware" and "free software" seem to escape them. Just google around and you'll see thousands of FOSS projects listed on sites like TUCOWS, download.com and others, as "freeware" and not the proper "free software" that they are. There are also people who think "free software" means just that (lowercase "F" there).

Let's be sure that if we have a search engine that let's brainless developers look like experts by cutting and pasting bits of OSS code from here and there together to make their software work, that they know what the license is and that they must be in compliance with it to use it.

Please?

Dependency Rejection

[aprilsound]

I'm of the opinion that most frameworks and libraries don't reimplement enough. In the FOSS Java world, everyone has a dependency on some version of Apache Commons Lang. For what? Just so we don't have to write StringUtils.isEmpty?

Don't get me wrong, if you're developing a stand alone project that wont be a dependency for someone else, then you absolutely want to rewrite as little code as possible. Let someone else maintain as much of your codebase as possible. But if you are writing something that other projects will be using as a dependency, don't you dare make me download four other libraries just to run your code. Write your own dang StringUtils or, if you're lazy and your project is GPLed, just copy the code.

Re:Dependency Rejection

[ComputerSlicer23]

Or better yet, ship a "minimal dependency library". That has all of the minimal sets of dependencies inside of it (either of real code or replacement code). Thus if you are using "Apache Commons Lang" already, it'll pick and use that. If not, you can use the drop in replacement code.

Kirby

Re:Dependency Rejection

[Anonymous Coward]

The C/C++ equivalent is that festering pile of crap Glib, which manages to bring huge masses of code to the party without doing anything you can't already do with standard C, C++ or POSIX and a tiny amount of brains.

Re:Dependency Rejection

[anarxia]

Most apache projects use maven so dependencies are handled automatically for them. The learning curve for maven is steep but it's well worth it.

Phew

[Anonymous Coward]

real programmers use CPAN

Re:Phew

[Fist! Of! Death!]

but only enlightened Monks can become one with CPAN...

Re:Phew

[Anonymous Coward]

Funny? FUNNY!!!! Who the hell modded the parent funny?

This is quite possibly the highest signal to noise ratio post EVER on Slashdot :)

I heartily endorse the above product, service, person or message.

Doesn't work

[Timesprout]

I just ran a search for "the 500,000 lines of code I need to finish by friday all the stupid extra features the PHB wanted after we had set a deadline based on the original spec".

0 results, rather disappointingly.

Re:Doesn't work

[mattpointblank]

In actual fact, it breaks the search!

http://search.allthecode.com/search/results?zequer y=%22the+500%2C000+lines+of+code+I+need+to+finish+ by+friday+all+the+stupid+extra+features+the+PHB+wa nted+after+we+had+set+a+deadline+based+on+the+orig inal+spec%22&bSubmint=Search

Maybe we're being told something.

Re:Doesn't work

[Anonymous Coward]

I just ran a search for "the 500,000 lines of code I need to finish by friday all the stupid extra features the PHB wanted after we had set a deadline based on the original spec".

0 results, rather disappointingly.

Don't fret. Nobody ever got fired for buying MS software.

Re:Doesn't work

[dodobh]

Try asking here http://www.thedailywtf.com/

Re:Doesn't work

[curunir]

You might try searching either this search engine or this search engine...you might find better results.

Laziness?

[vladsinger]

So what if a particular solution has been implemented before? Continuously writing code should keep people on their feet, and ready for when they have to respond to a unique situation, right?

Re:Laziness?

[tb()ne]

So what if a particular solution has been implemented before? Continuously writing code should keep people on their feet, and ready for when they have to respond to a unique situation, right?

Except that most people have jobs with deadlines. Besides, there's always more code to write. Even if I pull available code out of a repository, I'll still be continuously writing code. Furthermore, there's value in seeing how other people implement a solution because it is probably not exactly the way you would have implemented it and you just might learn something from their solution.

Java Only for now,,ASTs coming soon

[Anonymous Coward]

Maybe they should store annotated AST's instead of raw code.

Great!

[dasOp]

Now we'll see the One True Wheel (tm) which will fit on all cars.
Screw different tires for different conditions, screw differing performance.
The One Wheel will rule us all.

Re:Great!

[kalirion]

No, not quite.

1. Use masking tape (generic) to fit The One True Wheel(tm) to your axle (proprietary)
2. Use a knife (kitchen) and some cement (Home Depot) to give The One True Tire(tm) the correct grooves for the road condition.
3. Profit!

not all SW can be reused.

[apodyopsis]

I am an embedded engineer. Various firms I have worked for have tried to implement some kind of "reuse" code store. But every time real-time considerations and platform specifics have derailed it (thankfully) in the early stages. At the low-level so called "code reuse" is (IMHO) a nothing short of a right royal pain in the neck. It looks good on paper, managers like the concept - but it is impossible to implement without large amounts of hardware abstraction. Maybe it makes more sense further up the SW tree, at a higher level when things are not so resource critical.

Re:not all SW can be reused.

[Anonymous Coward]

I am an embedded engineer An embedded engineer? Like, say, Han Solo was embedded in carbonite? But what are you embedded in?

Not Quality - Comprehension

[Dareth]

The reason people often roll their own "trivial bits" of software is not so they get the best quality. It is so the developers have full comprehension of the code and that it follows the guidelines they use for their projects.

There are enough faulty assumptions about code you developed internally. Why should one keep trying to cram the square peg into the round hole. Sure you can shave it down, refactor the block all you want. But it may well prove to be faster and better to craft the piece you need.

Diversity does not hurt in the long term. If everyone just followed the status quo, and reused the same blocks of code, how boring would that be?

Re:Not Quality - Comprehension

[RonBurk]

Exactly what Dareth said! And "follows the guidelines they use for their projects" does not just refer to how the code is indented/commented. It can mean substantive issues like: pluggable into a test harness, memory management policy, thread safety, error handling policy, avoiding recursion, etc.

Re:Are we really making it better for us, or worse

[gravesb]

We shouldn't ignore a good idea just because it makes it easier for someone to do something illegal. There are laws to protect the code. I think the benefits will outweigh any loss from commercial companies stealing the code. I hope this does work out as well as it looks like it could.

Licensing

[maxwell demon]

I think in order to be really useful for not reinventing the wheel, it should allow intelligent searching for licensing. That is, it should allow to restrict your search to codes with certain licenses, or even better, to code under a license compatible with any given license (or set of licenses).

For example, if you are working on code which you want to release as BSD, it's not much help if you find code licensed under the GPL, even if that code on its own is great. Likely, if you are writing GPLed code, you are not interested in code under licenses incompatible with the GPL (like e.g. the MPL).

Of course, the search engine cannot make a guarantee that the license will fit your needs, but then, it cannot guarantee that about the code's functionality either.

Re:Licensing

[oliverthered]

You can still find out how the GPL guy done something that may be hard for you to do and then reimplement it under whatever license you want.

Re:Licensing

[Anonymous Coward]

When I worked at SCO we used to do that all the time, works great... I can specifically recommend the linux kernel, it's just so wel documented. (Yes, I am joking and probably flamebait)

Re:Licensing

[Anonymous Coward]

How many GUIs do we have? Windows, Xwindows, SWING, Thinlet, wxWindows, FOX, GTK, MFC, and many other solutions exist. Each of these, however, has a different license/language/API requirement. In terms of software development, this means that you must have significant software expertise to become productive. Some of this knowledge is expected in an employee, but not all of it.

As to stability of code, this has something to do with the coder, software maturity, and goals. For example, libjpeg has it latest version at "6b" which was done 27-Mar-1998. It is still around. It is the most popular solution, partly because of its non-restrictive license. There are others who develop their own JPEG solution, but that is increasingly rare. For the most part, such decoders are custom created if additional speed is needed and it is often tied to specific hardware.

Re:Licensing

[Anonymous Coward]

Google code search already does this:

license:regexp Search only for files with licenses matching regexp.

license:bsd int\ printf -license:gpl heapsort

Well....

[VE3OGG]

While the article mentions that too much time is spent re-implementing new code, I disagree that this is necessarily a bad thing (tm). Re-inventing the wheel can often cause evolution of code, as opposed to the stagnation that can occur if something remains static. Now, of course people will say that this is GPL code, and people can then modify it -- this is of course true, but modification on that level seldom equates to evolution per se, sometimes because the changes as specific to the application, sometimes because you are trying to do something with code that simply wasn't designed for (I guess you could equate it to trying to run a a web server from Windows 95).

Re:Well....

[AaronLawrence]

How can code by "evolving" if you're throwing it away and doing it again?

Perhaps you mean that people can look at existing code and take the good ideas and do it better.

But that's not evolution. And the problem is that thinking you can do it better, doesn't actually mean you can. At most, it will probably be "more suitable for the way I think and the constraints I do/don't care about".

Re:Are we really making it better for us, or worse

[MobyDisk]

I'm curious about these companies because un my experience, companies take licenses very seriously. Were these small companies or large companies? Were they clueless idiots in general, or just on licenses? Was this done out of malice or lack of understanding? Did you press the issue to them? If so, what was their reaction.

I would love to hear more.

Too much time?

[jmagar.com]

I wonder if writing it yourself is a time saver. Some reuse is okay, IF you have intimate knowledge to the code in question... but finding code that "might" fit the current problem is risky as the Dev is unable to truly assess whether or not the code it appropriate. The undocumented bugs, or system assumptions, will lead to using code and then countless hours debugging problems you didn't expect. Now you are debugging code you didn't write, and likely taking more time understanding than it would have taken to build it yourself.

There is probably a point at which the system complexity of the resued code becomes great enough that the re-use is valuable. But how big, and how mature the reusable codebase, affects this decision.

Re:Too much time?

[tppublic]

I wonder if writing it yourself is a time saver.

Like most questions, the appropriate answer is "it depends". Take an example: I just spent yesterday rewriting a single class to fit into a standardized library. After 20 minutes of coding, 1 hour of documenting, and 2 hours of writing tests, I actually have something that meets the library standards. Could I have used the original class? Sure. But it had problems and inconsistencies. The main problem is that most open source code goes through the coding, but never gets the documenting and aggressive testing, because it's too much overhead.

The undocumented bugs, or system assumptions, will lead to using code and then countless hours debugging problems you didn't expect.

This is so true. This even happens in the hardware world. A T1 connection seems like it's pretty simple. By modern networking standards, it's really slow. However, finding a T1/E1 LIU (Link Interface Unit) that actually works is surprisingly hard. An engineer familiar with the specification can break the vast majority of implementations in less than 5 minutes. I actually know of only one that works (not built by my company, I might add).

There is probably a point at which the system complexity of the resued code becomes great enough that the re-use is valuable. But how big, and how mature the reusable codebase, affects this decision.

For big problems, the 'reusable code' simply becomes a framework that is built upon. This could be anything from subversion to packages like Joomla. Reuse of such large items certainly makes sense.

On a smaller scale, maturity is important, but a lot of the issue centers around the available documentation, test code, AND documentation of the test code (1500 lines of JUnit tests don't do that much good if you can't tell what each test is actually doing). There are very FEW libraries that are actually well written, well documented, and well tested. That's why so much code gets rewritten.

Dependency Rejection-Take a picture.

[Anonymous Coward]

That's one of the things I like about image-based languages. Code snippits can easily be downloaded and integrated on a running system.

Just more results, not more content

[hsa]

I tried searching for more general level stuff.

"Radiosity"

gave me several pages of "Radix Sort".

For inexperienced reader: these are not related at all. One is a general sorting algorithm in computer science and one is lightning algorithm used in computer graphics, and games like Quake.

So in my guess it just does a fuzzy search and yields more results. Getting more results, which are not the ones you want won't help you one bit. Useless for me.

Re:Just more results, not more content

[Anonymous Coward]

Right; this search engine is awful. Absolutely nothing I tried came up with relevant results.

Re:Are we really making it better for us, or worse

[Anonymous Coward]

I've talked to NO LESS THAN a dozen commercial companies in the last 2-3 years where they're actively taking FOSS source and incorporating it into their products, because.. (and I quote) "..Its freeware, so we can use it however we wish."

Then publish their names and the software that is incorporating said FOSS software. If you, or anyone, knows about companies or people using code in specific products that violates a license agreement, then we need to know about it so we can bring it to their attention, to the attention of the developer, and to the community as a whole. Without pressure from the community on companies/people who violate license agreements, nothing will ever be done.

Isn't that what...

[OneSmartFellow]

...GCC is. A Compiler and a complete set of libraries for most mainstream languages, along with the source. You have everything you need to write your application.

No need to reinvent basic_string or itoa()

But of course this is little help to those in my profession who insist upon wrapping up (obfuscating) everything they do in layers of 'abstraction'.

Most of thier stuff is legacy code before it is even released. Me, I stick to the KISS philosophy. That way, when the FNG comes along he can understand what my code is doing, and fix bugs with confidence he isn't going to break something else. But hey, that's just cynical me, YMMV.

Re:Isn't that what...

[hoggoth]

> But of course this is little help to those in my profession who insist upon wrapping up (obfuscating) everything they do in layers of 'abstraction'.

Several years ago I had my 'real world' introduction to object oriented programming when I worked on a financial management application developed and used by a large Wall St. brokerage house.

The C++ code had classes 15-20 layers deep, each layer adding just one tiny bit of functionality to the layer it inherited from. It was a nightmare, taking 15-20 minutes to figure out what any function/method call would do by tracing the classes up and down the hierarchy.

Code throughout the project called on classes at any level in the hierarchy's depending on the precise behavior it wanted at the given point. But there were so many classes with so little to differentiate what they did that it was impossible to remember the difference between FixedIncomeStreamTreeNodeHelper and FixedIncomeStreamTreeLinkedNodeHelper for example.

SourceForge Features

[Doc Ruby]

This system would be a great feature of SourceForge. Finding all the common components in different projects to be factored out and share instead. I was always disappointed in SourceForge's lack of intelligence about the related contents of its different projects. This thing could find relevant code and import it into the integrated navigation.

Re:GPL nightmare

[Anonymous Coward]

BSD rules !

Oh really? Which part of the market does it rule again? None, you say?

Re:Are we really making it better for us, or worse

[Anonymous Coward]

If these companies release public binaries, and you feel what they are doing is morally wrong, you could consider anonymously blowing the whistle on them to people who are capable of analyzing the binaries and who can take it from there legally speaking. FSF or some similar organization would seem to be a prime candidate. Some people seem to be very skilled at recognizing open-source code even in binaries - mostly, I believe, by looking for embedded strings and similar - but they probably would be more effective if they could look for specific libraries in specific binaries instead of searching at random.

Corporations probably won't learn the difference between "freeware" and "Free software" until it becomes apparent that not doing so will get them in legal troubles.

too complex to work

[fuliginous]

It all just seems to task intensive.

I'm committed to the better solution being better languages. The likes of python, ruby, boo (no lisp related debates please) add better features making coding denser and faster. Better tools and to some extent in the areas of GUI widgets components mean there is less re-work.

The big gain is not having to search for what has been done already, learn it, tailor it.

it's the license, stupid.

[Anonymous Coward]

If I download open source code and combine it with my software, I have to agree to some conditions which I might understand and which the "author" might not even be aware of (maybe he cut and pasted from some GPL'd code). I risk having the FSF or some company come after me with hotshot lawyers like Eben Moglen or David Boies, after I've distributed my code.

If I write the code myself, it takes longer, but it's likely to be better - faster - cheaper - more relevant to the task and tools at hand. And it's unencumbered (well, apart from the patent nuisance which is orthogonal to this subject).

We need a catchy phrase to describe "unencumbered software", which is what everyone thinks of when they first hear the phrase "free software" until Richard Stallman explains it to them a half dozen times.

Are we really making it better for us, or worse

[Anonymous Coward]

"I've talked to NO LESS THAN a dozen commercial companies in the last 2-3 years where they're actively taking FOSS source and incorporating it into their products, because.. (and I quote) "..Its freeware, so we can use it however we wish.""

*shrug*

Works for me, now if you'll excuse me I have some MPAA/RIAA/Usenet to illegally download.

Re:Are we really making it better for us, or worse

[UbuntuDupe]

So wait, you're concerned about possible copyright violation due to internet distribution, and want to make sure that if the work is distributed on the internet, it has the appropriate precautions to make sure people don't violate the license? ... because that's exactly why the record industry supports DRM.

Re:Are we really making it better for us, or worse

[Kjella]

I've talked to NO LESS THAN a dozen commercial companies in the last 2-3 years where they're actively taking FOSS source and incorporating it into their products, because.. (and I quote) "..Its freeware, so we can use it however we wish." The licensing differences between "freeware" and "free software" seem to escape them.

A 150,000$ dollar lawsuit in RIAA style (multiply by number of source files as separate 'works' if you like) and a cease-and-desist forcing them to stop shipping their product, along with federal criminal copyright charges should be enough, don't you think? Not that I generally approve, but where are the ambulance chaser lawyers when you need them? I think the "I'm sure it's all a big mistake and you just release the source and we're happy" enforcement of the GPL is part of the problem - there's no penalty to being caught and companies in general have no ethics.

Also note that BSD code lets you incorporate it, LGPL too if you distribute chages to the library itself. If it's internal you're not distributing it so no requirements at all. But you said "incorporating it into their products", so I assume you ment just that.

Re:Are we really making it better for us, or worse

[Wildclaw]

Seeing commercial companies violating licenses like that sucks. It is not unexpected due to the fact that coorporatism is a breeding ground for greed and selfishness, and buisness laws often fail to protect the rights of those who aren't in the business of making big money themselves.

Categorizing "free software" under freeware however isn't really incorrect. The only license part of "free software" that could cast into doubt the validity of "free software" being freeware is the redistribution part. "Free software" has a couple of redistribution restrictions, but so does many programs considered freeware that are only availible from an official source. It could be argued that under a strict definition of freeware there may not be any redistribution restriction but that is question of how freeware is defined.

The source code modification clauses of "free software" provides extra opportunities for anyone wanting to create derivations of the program. These clauses aren't restrictions however, but an extra opportunity above and beyond that of ordinary freeware that don't allow you to modify the source code at all.

Re:Are we really making it better for us, or worse

[Temujin_12]

Koders.com lets you search by license.

Re:Are we really making it better for us, or worse

[Anonymous Brave Guy]

I've talked to NO LESS THAN a dozen commercial companies in the last 2-3 years where they're actively taking FOSS source and incorporating it into their products, because.. (and I quote) "..Its freeware, so we can use it however we wish."

<shrug> That's what you get for calling your software free when it's not. </shrug>

Note to mods about to mark me down because they don't agree: would you be thinking the same if I were objecting to the phrase "intellectual property theft"?

Re:Are we really making it better for us, or worse

[Doctor Memory]

I got the impression that the companies weren't using the code entirely, just lifting bits and pieces here and there (e.g., not a whole image library, but maybe the code to read/write JPEGs). Which is just as illegal, but might be a bit harder to prove...

Of course, it's Monday, so what do I know...

Re:Are we really making it better for us, or worse

[Raenex]

I'm curious about these companies because un my experience, companies take licenses very seriously.

In my experience, it's up to the developers to be license-aware. I've never been told about the importance of respecting licenses or to be careful about using third-party software.

Re:Are we really making it better for us, or worse

[mangobrain]

There are laws to protect the code.

There are laws there to protect a lot of things. To give a somewhat extreme example, there are laws that say it's the driver's fault if I get run over whilst cycling (in some countries/circumstances) - but those laws won't magically rush in and pull me out at the last second, or reinforce my cycle helmet such that it can withstand speeding lorries, or anything of the sort. You can't benefit from compensation if you're dead.

My point is this: given a compiled (i.e. non-human-readable) executable file, can you tell whether or not the source code from which it was compiled violates any licenses? Not realistically, no. Being illegal doesn't stop a lot of people from doing a lot of things, or we'd probably have achieved world peace already. Laws are also not automatically enforceable by merit of existence.

That said, I don't think this search engine will encourage anyone to commit licensing violations who isn't already perfectly happy doing so (or ignorant of what they're doing). I just don't think legality has anything to do with it, unfortunately.

Non-functional

[kahei]

'binary diff', 'levenshtein distance' -- no hits.

'morphological analyser' -- 1 hit (inappropriate)

It's completely useless. Am I jumping to conclusions? Mm... no, I don't think so, it really is utterly useless.

Still doesn't understand the code itself

[Sembiance]

One thing that I find disappointing with all the code search engines is they all treat them as regular text files, more or less.
None of them seem to make an effort at understanding the code syntax.

That's why a few years ago I wrote one for C/C++ code called http://csourcesearch.net/

I just did it as an experiment, and using all open source software and in my spare time, but I think it having the ability to syntactically know the difference between a comment, a function, a structure, etc. makes a big difference.

When Google launched their engine, I was disappointed they didn't take the extra time needed to make their parser/engine smart.

Re:Still doesn't understand the code itself

[mbadgett]

Sembiance is 100% right that treating code as text will not lead to relevant results. You need to understand the contents. With http://www.krugle.com/ we do parse the code understanding not only the differences between a function call or call definition, but also comments. We take this into consideration when determining ranking - along with the size of the project, amount the codes been used, etc.

Re:Still doesn't understand the code itself

[Simetrical]

When Google launched their engine, I was disappointed they didn't take the extra time needed to make their parser/engine smart.

Aw, c'mon, you can't blame them. It's a beta!

. . . wait a minute, this one actually isn't. Are there others?

Not as simple as it sounds

[Junks Jerzey]

Just because you find code doesn't mean that it works as advertised, doesn't have memory leaks, buffer overruns and so on.

Code also isn't tagged with environmental issues: how much memory is used, STL or Boost requirements, what system functions are needed, library dependencies, etc.

Code is often layered on top of custom libraries. Sure, here's code to render HTML, but it needs a dozen custom data structure modules from the Netscape code base, for example.

Adaptability and suckyness factor

[Programmer_In_Traini]

What code search engines don't tell us is sometimes the code you get just sucks ... or is poorly adapted to your need.

i usually search for a function to say, validate a string for phone ... or ... homemade browser or ... whatever, just function to help me do what im trying to do without reinventing the wheel because i know its been done already.

but most of the time when you do find it chances are it wont be adapted to your need, so much that you end up writing your own based on what you've seen. which explain why there are gazillion versions of (example) javascript calendars on the web.

One thing that happens too is that the code just sucks, its buggy, its buggy because of your own rules, or sometimes plain buggy.

I wish there was a checkbox on the page to specify "bug free" or "not sucky".

But that's impossible, if something could be done it would be to store only independent controls or function. things that work on their own. so that i don't need to end up rewriting the whole thing to make it work in my code.

Re:Adaptability and suckyness factor

[maxwell demon]

What code search engines don't tell us is sometimes the code you get just sucks ... or is poorly adapted to your need.

And what text search engines don't tell us is sometimes the text you get just sucks ... or is only remotely relevant to your need.

Which doesn't mean those search engines are not useful.

Re:Adaptability and suckyness factor

[Programmer_In_Traini]

Yes i know but the point of text search engine is information in general, not to allow you to plug what you read in your term paper as-is.

whereas the point of code search engine is to allow you to re-use code that's already been written so that you don't have to paraphrase what you've just seen.

I know its hard, or even unfeasible to build code that answer everybody's need, let alone, expect someone else to write a code that'll work for his project AND my project.

im just saying, if we're gonna talk about engines that allow you to find and re-use code, then the engine should be able to distinguish specific implementation from general implementation. Something that once its passed to someone else doesn't require rewriting it or at least, not all of it.

more relevant results?

[brechin]

So far, every search I have made, from names of people, companies, class names, etc. yielded results that didn't even contain the search term(s).

Fuzzy searching is one thing, but at least TRY to get an exact match and let me know when you're just taking the first 2 characters of a search term.

Learing code = time. Libraries = hell.

[slackmaster2000]

Two points.

First, in some cases taking a chunk of somebody else's code is a great time saver. It can be problematic, however, when the code doesn't do what you want and then you have to jump into it. Is there anything more painful than trying to understand somebody else's code? On rare occasions it's easy - good comments, good style, good structure. In most cases it's a pain in the ass, even if they've taken the effort to write good comments and nice code. In these cases one has to make the call on whether it would be easier to simply write their own code, using the 3rd party code as reference. Sometimes this is beneficial, especially if you're planning to maintain the code for a long time. In other cases it turns out to be a big waste of time. Magic eight ball, don't fail me!

Second, in the case of more "complete" code that you'll be linking to...this can lead to distribution hell. This is especially difficult for users of FOSS (and especially for non-Windows projects, simply because people are more apt to package an installer on Windows). Well, it's not difficult for the developer, it's difficult for the *user* when the developer says "oh, here's a library that I'd like to use...sure it's in alpha, and not part of most standard distributions, and relies on eighteen other packages that aren't part of most standard distributions, but it'll sure save me some time, so if the user wants to use my software, they can suffer like I have to suffer."

Choices choices choices. If you're going to insert code into your project, then it's up to you to decide if that code does what it should, and if you want to maintain it. If you're going to link to code, it's important to research first if your users have easy access to the libraries.

That said, there are no downsides to new ways of seeking out code. Just for research and learning it's always going to be a boon.

All Your Code...

[Anonymous Coward]

are belong to All The Code!

/ducks *sorry*

Not Invented Here syndrome

[mi]

I worked for a company, that wrote its own distributed computing system (in Java/XML). It sucked awfully by all measures (latency, CPU-load, memory requirements, bandwidth), but they would not dump it in favor of PVM or one of the MPI implementations because:

1. We don't know, who wrote that PVM thing and how to support it.
2. The guys, who wrote our own system are both really nice and dumping their work would offend them...

This is such a common problem, there is a term for it...

Not Incredibly Useful

[WED Fan]

Rather than use this as a product announcement, they should have quietly rolled it out. Gotten more than the Java repository going, then announced.

Rating: Meh

What I look for in using other peoples code

[simm1701]

Is the code easy to find? Will a quick search of sensible key words take me to a short list of results with high accuracy? No point in spending an hour wading through results that may or may not be useful when I can implement and test it myself in 2 hours.

Is the license clear? I may eventually want to release as open source or commercially use something I write. If I include someone else's code/library I have to make a note (hopefully in the LICENSE file provided with the code or in the top of the code comments) on what the license is. Is it BSD, GPL, public domain, not stated or some commercial license that lets me look at the code but not use it myself?

Is the code self contained? This generally means does it come as a library. I dont like copying and pasting code into my code - especially if its not the same coding practice as my own. (this comes abck to licenses above - if its self contained and with an incompatable license atleast I can rip and replace later if I need to)

Is the code well known? Is if the defacto standard for doing this type of thing (STL, Perl core, glibc)? Or is it one of several well known options for the same thing (gtk, qt, kde)? Or is it an unknown? This will help you know how well this code is field tested already - I don't like signing up to be someone else's beta tester for free!

Is the code still maintained? Is this an active company with a project? Or a group on source forge? Are the developers still around and the forums active? If I need a new feature further down the line is there chance of support? I don't usually want to pick up the whole dead weight of supporting unsupported code that I didn't write if I can avoid it.

Can I use it as is? It frequently takes longer to modify an "almost there" modules to do what you need than it would have done to reimplement the wheel as it were and write it yourself first time, and writing it yourself will atleast make future debugging easier assuming you have a good memory for design and good coding practices.

Is there documentation? The old comparison about documentation and sex, when its good its very very good and when its bad its better than nothing. I dont want to have to read someone elses uncommented un documented code just to evaluate if it might work for me. I want to be able to read a good overview of the library, its functions, methods, attributes, errors and exceptions - CPAN is an excellent (in most cases) example of what I mean.

Thats a pretty hard list of requirements to meet - true it shouldn't be, but this is the real world. If those requires are not met then odds are it will be less effort in the long run for more reward for me to implement it myself.

Re:What I look for in using other peoples code

[SineWave]

After searching the comments for "CPAN", I'm shocked that it isn't mentioned more (found 2 instances, at my level of filtering, anyway). It's such an excellent resource that, when I need to implement something at least a little out of the ordinary, it's the first place I turn. Modules are (generally) well documented, and the dependency-management aspect of the CPAN shell is excellent (can you imagine how wonderful it would be if RPM were that good?)

If there were C/C++ analogues of CPAN, the world would be a far better place. Why is CPAN a unique aspect of the Perl community?

Reusing random crap found on the internet..

[SpinyNorman]

... sounds to me like a thoroughly bad idea.

There's a reason for things like specifications, documentation, source control, testing, etc.

Maybe you'd rather google for popular home remedies rather than consult a professional doctor?

I'd google for *algorithms* if I was at a loss, but I'd certainly want to implement them myself.

What is useful in terms of code reuse is more controlled coherent collections of code that are highly tested, documented and generally controlled, such as the C++ Boost collection, but even there while I'd use it for a hobby project, I'd want to throroughly vet it for eror handling etc and run it through my own test suite before I'd consider using it professionally (assuming that it wasn't against company policy to do so).

There's also the whole issue of Copyright and patents. Nowadays there's no legal need to add a copyright notice to code - it's automatically copyright protected by default, so you could only reuse code that explicity grants a licence on terms that are suitable to your use (presumably including the separate right to modify as well as merely use). Patents are more tricky... At least if you unknowingly reimplement some patented algorithm yourself there's less liability than if you deliberately reused a patented algorithm, but grabbing stuff off the internet would seem to blow away any "clean room" type defense. Not that I support software patents, but unfortunately, in the US at least, they are a reality.

peer review

[Anonymous Coward]

Other than popularity ranking in the results, there is no karma system/peer review ranking associated with the snippets. Wouldn't that be also useful? Binaries at download places get a ranking system (4 cows for example, etc)

More to Come?

[WED Fan]

More languages Summer of 2007?

You product announce in Feb 2007, and expect people to remember to "check back with you" 6 months later? Let me know how that works for you. This crowd will have forgotten you by then because someone else will have done it properly.

Languages to get up RIGHT NOW and screw "Summer of 2007":

• JavaScript
• Perl
• Python
• SQL
• MDX
• C (++ and #)
• VB (.net and legacy)
• COBOL
• FORTRAN
• And many others where large libraries already exist...

The problem is, this outfit is already trying to be in the list of also-rans.

To spark submission, get backing, hold some competitions and rewards for code submissions. Winners are based upon level of abstraction (for max reuse) and level of usefulness.

Consider a "Howto" to create abstract code. Most of the submissions I see on other sites have wonderful examples, but it takes too long abstract the code for use in another project. A lot of young and not-so-young developers spend so much time with heads-down-over-the-code projects, they don't have time to learn how to really do it. They know the theories, but they are always too busy on the "current" project and will "refactor later when we have time to do the 2.0".

1. Dupe. 2. Mostly pointless.

[Qbertino]

1st of all it's a dupe. Aprox. a year old I suspect.
2dn: Frameworks, standardised open source Application stacks and comprehensive documentation are what speed up coding and developement. Copy/Pasting foreign code rarely helps.
3rd: It's only for Java.

The real reason is unrelated bloat

[Anonymous Coward]

It would help if developers would stop pulling logging frameworks and other superfluous crud into libraries. It's like the woman who swallowed the fly, developers are looking to solve a specific problem not introduce your entire friggin dependency tree.

Licensing is a separate issue, code that does one thing and does it well is widely reused (eg: libdl, libm, libz...).

Google Adwords?

[TheSoggyCow]

I find it somewhat ironic that there are Google Adwords Ads all over this "new open source coded" search engine... I mean don't they aim to take Google down?

Finding new code? That's easy...

[Anonymous Coward]

I didn't RTFA (since this is Slashdot), so here's how you find new code:

$ cat MainClass.java | grep "new"

Hope that helps!

Re:Finding new code? That's easy...

[chez69]

how about grep new MainClass.java?

overuse of cat is linked to excessive diarrhea. just saving you the trouble.

Why I rewrite code...

[pbhogan]

While I certainly would welcome anything that could help me find code, the reason I'd want it is to find reference code, not reusable code. I've been programming for, oh, two decades now and one thing I find myself doing constantly is finding a bunch of libraries or bits of code and coming to the conclusion that I should just write it myself because of one of the following:

1. The library/code is good, but doesn't quite work the way I want it to
2. The library/code is close, but getting it to work the way I want is painful
3. The library/code is bad
4. The documentation is bad/nonexistent
5. The license is prohibitive or annoying (i.e. it's not LGPL or BSD or the like)
6. I enjoy writing code and sometimes I feel I could do it more elegantly, or efficiently (I might just want a very specific and optimized part of it)

More often than than not though I just enjoy coding and I love learning to code by writing new code. The black box thing... eh... I like to tinker under the hood and find out how things work.

But my point is that finding code is not that hard. It's finding code that fits *exactly* what we want. Code is usually just not quite as modular as we'd like to believe and, if we're honest, as programmers we have a certain vanity about writing code so it does things My Way. :)

Software Tools

[bobs666]

I'm not a coder,

So your out of your domain.

IT was UNIX its self that came with the concept of "Software Tools".

It's the goal of any good programer to make simple tools that one can reuse over and over. But few can actuly do it, and do it well.

FOSS killer

[Anonymous Coward]

Consider that many people in FOSS mostly code for fun. If this gizmo is always going to find the exact piece of code they need.... then it is taking the fun out of coding! This will kill FOSS! And it is aimed at damaging Google as well! So it must be some Microsoft hideous machinery!! (You know how I know this? It does not work!!)

meh

[sharkboy9999]

this is the same thing, except this one is developed by students. http://sourcerer.ics.uci.edu/ it crawls sourceforge and lparses the code, the creator, etc. ITs in beta stages of course, but it has great potential.

wasting?

[stachu trawki]

Too much time is wasted re-implementing code that someone else has already done, for the sole reason it's faster than finding the other code.

I thought that if you take the *faster* route you are *saving*, not wasting time...

Context and Relvancy

[MagicMerl]

For those who have not yet used Krugle, check out a posting by Justin Royce a while back - his review is fairly complete. (http://webtekconcepts.com/2006/11/30/krugle-goes- grassroots/) Search engines ultimately are judged on the accuracy and relevancy of their results. Krugle parses the code to understand the context. The relevancy is based on whether the result is in a function call or class def vs comments etc, and Krugle leverages project meta data such as number of committers and frequency of project updates to accurately prioritize search results.

Re:Context and Relvancy

[Anonymous Coward]

To see a Java code search engine which does real code parsing and makes all Java language constructs parseable, check out jsourcery (http://jsourcery.com). You can search for classes, variables, methods etc. Also the search results link to source code which is hyperlinked so you can trace program execution.

Undocumented code is useless

[iamacat]

And documentation can be searched quite nicely on Google. What's the running time of this algorithm? What are the limits on the inputs? How to change traversal of the data structure to different order? Without a clear answer, maintaining the code may well be more difficult than writing it from scratch.

Java only?

[Anonymous Coward]

Java only? Please. Lack of broader language support makes this next to worthless. They're promising more languages in summer of 2007, but who knows whether they'll actually deliver. Krugle and GCS both support two dozen plus languages already, including C/C++ and all the popular scripting languages (Perl, PHP, Python, Ruby, etc.). Wake me in six months when ATC has made good on its promise to catch up.

Not exactly

[metamatic]

Here are some of the top reasons why I end up rejecting use of open source code:

1. No description on the web site of features & system requirements.

2. No documentation for how to install or use it.

3. No stable release and/or still alpha.

4. Written in PHP.

Finding the code is the least of the problems.

You're right, but there's more to it

[try_anything]

The origin of "hacking your own" starts with the most characteristic aspect of software development: unfamiliarity with the task and ignorance of the parts needed. Developers start with a small, limited idea of what they need and can't afford to sink a bunch of time into learning a large library or framework which might not suit their purposes. At this point, they're often accused of laziness. "You know you need to log stuff, so just download a logging library and use it." Yet it's too soon to do that because you don't know what features you need, or even whether you need a logging library at all. I've seen very sophisticated scientific programs that just use stdout and stderr. So in the beginning, it makes sense to just implement the bits of functionality they know they need, so they can put off the decision until they have a decent shot at getting it right.

After a bit of hacking around on a functional area, they have a rough idea what that part of the program needs to do. This is the point where things go awry. The sensible next step is to draw up a rough list of requirements, find and evaluate the existing libraries or frameworks, and choose one to replace the current code and fill future needs. Unfortunately, there are strong irrational factors that push against this.

• Failure to budget time (officially, in an organized project; emotionally, for an amateur effort) for this step makes it seem like a setback that must be avoided.
• Fallacious thinking about "sunk costs" creates an artificial incentive to continue developing one's own implementation.
• Bosses may see the abandoned implementation as a failure instead of a prototype.
• Switching from custom code to another library requires rewriting some existing code (a somewhat rational reason, but in practice usually overblown.)

And the ultimate reason:

• Ego attachment to one's own code makes it hard to throw it away.

People are proud of their code and proud of their unique style. They have a visceral aversion to replacing their unique conception of the problem with a standard approach. (It would be a shame if everyone threw away their unique approach; after all, we always need diversity and new approaches. But there's a difference between making a serious attempt to create something new and inventing excuses to cling to every bit of one's creative output.)

As a result of one or more of these pressures, the experimental implementation often hangs around, keeps growing, starts to look very much like previously existing libraries (as developers borrow ideas and come to convergent conclusions), and imposes high support overhead. Learning to use an existing, mature implementation of the functionality you want (or similar functionality) should be a prerequisite for making a serious attempt at creating a full-fledged alternative.

I know this guy

[Anonymous Coward]

He is just doing this as a joke to get on Slashdot. All it does is do an approximate substring search of the source text. He doesn't even parse it. Wow, it must be a slow news day today.

Re:Your GAY for code plagiarism

[Tim Browse]

Explain to me why a student, when caught plagiarising, is shamed for the act, but one who writes code is not?

Because a student is supposed to be learning how to do something, and possibly proving what they have learned to a 3rd party. Someone writing code (in other situations) is usually doing it to accomplish a specific task, minimising the time/money spent on doing that. If they can't find some existing code, they should be able to implement it themselves. If all they've ever done is copy code when they were a student, they are unlikely to have this ability.

Next question.

Why I Like Perl

[Zonk (troll)]

CPAN.

This is the #1 reason I work heavily in Perl. If I need to do something, 95% of the time there is a module on CPAN that will do the hard parts for me so I can just focus on getting what I need done.

I would love something like CPAN for Java. I prefer it's syntax and do a decent amount of webapps (w/ Echo2) with it. It's just when I work in Java takes me a while since I end up writing so much by myself.

Re:Why I Like Perl

[Skrynesaver]

I have to agree, CPAN means I can throw an interface over a module inside half an hour, it's not the way to develop final use code but for rapid solutions to one-off problems it's unbeatable.

I call bullshit on your bullshit

[WindBourne]

Obviously, you are not a coder. There are some coders who will reimplement everything. That is the classic NIH syndrome (back in the late 80's, early 90's, HP had it real bad).

Others will ask will it work for me? If it will not, then they will redo one (interestingly, many of them will "borrow" design and code from L?GPL version. I have seen it in 3 places now).

If it will work AND is not the core slowdown, then they will use.
If it works, but it is too slow due to being too general, then they will redevelop. And for a lot of core code to an application, that is the case. GTK/Gnome and KDE/qt have redeveloped a lot of code. I do not always agree with their choices, but ....

Overall, most coders do not redo.

One framework to rule them all...

[Anonymous Coward]

The .NET frameworks of were distributed to the developers of the land, but all of them were deceived. Deep in his fortress in Redmond, Gates developed his own framework. A master framework. One framework to rule them all. And soon, all of the developers were bound to Gates and forced to serve his bidding.

Re:One framework to rule them all...

[dasOp]

Yet another one whose descendants will hail RAW and Shea as the true prophets...

Actually this will NOT work for a number of ......

[3seas]

.... reasons.

source is done in various languages, modifying existing code to fit a new project can be more prone to creating bugs then doing it fresh, code might be useful in a way it was not original though of being used (creating another aspect of search...), a there may be licensing issues, etc..

Ultimately what we need is a higher level abstraction machine BUT disconnected from actual code. A level of abstraction that can be used to define the objective and constraints required and apply it to a code fragment/algorithm base and designed to go through a cycling of refinement regarding specifics. Once done, code is then generated in the language specified, or best fit.
And in time refinement of the code fragment/algorithm would be debugged to improve overall quality of resulting code.

Is this to far in the future in thinking about?

No! The base line abstraction mechanics is already identified and defined and even mostly coded.
Virtual Interaction Configuration
Also see: Abstraction Physics

I know this guy too

[Anonymous Coward]

Regardless of his motives, he analysed usage logs of his system for the period in which it was alpha, and from the search patterns it is clear that multiple people found it quite useful --- they found the sort of code they were looking for. not only that, but the search algorithm was optimised based on these trial usages.

Do you believe in this?

[rnd0110]

I do not. Most of the code is just not of that quality, uses wrong dependencies.... Is not generic enough, is not written in the language I like, etc, etc. Its one thing to have some quick sort or eigenvalue algorithm written right ONCE. It is quite different with the 99% or everything else.

Ready-set-go!

[kid-noodle]

First one to compile and run an Eliza program purely by (semantic) googling of blogs and sections of code, wins.

People don't reuse code because it's broken

[achurch]

Ignoring the slashvertisement, I think the real reason people tend not to reuse code is because any code they find will be either (1) broken, or (2) not up to the specific task and also broken. With rare exceptions, all code is broken to some degree, including yours (and including mine). Newer code tends to be slightly less broken about older code, as more people find new ways to break things; but however much some CS professors like to go on about OOP or whatever the latest fad is, the art of software engineering simply hasn't advanced to the point where people can reliably build non-broken software, in the sense that civil engineers build non-broken bridges or architects build non-collapsing buildings. Until we find the silver bullet that reduces software engineering to a reliably solvable problem, there's really not much we can do.

What, there's no silver bullet? Well, then I guess we're just SOL(*), aren't we?

(*) SOL: Secure in Our Ljobs (the L is silent)

Already done ...?

[smahesh]

How is this different from existing code search engines like http://www.koders.com/?
What is the USP of this new code search engine?

Regards,
Mahesh

re-use overhead can be worse than re-inventing

[pebrmatt]

Most of the time when I'm looking for source code, it's just to get some examples of how other people have dealt with a particular problem, or when figuring out how to use a new API. Only rarely have I been able to just drop in someone else's code and have it work as-is. There are usually dependencies/conflicts to work out, or incompatible parameters (arrays vs delimited buffer, etc). The sample code may also be tied to a sample GUI, which would need to be merged with my own. Or hacked severely to work as a console/server app. Many times, the time spent understanding the code well enough to modify it would have been better spent re-inventing the wheel.

This isn't to say that re-use is always bad. I've used things like pkzip and zlib libraries and saved a _lot_ of time (and some of my now-scarce sanity).

On the other hand, some libraries pack so many features (e.g. Stingray) that implementation can be rather involved. Nothing against Stringray -- they make some very difficult stuff fairly easy to do. But for the first project I used it on, we were only using a small subset of the capabilities and the overhead of learning the API, adapting to their updates/upgrades and rebuilding/distributing the library modules outweighed the advantages. Especially over the life of the project, which evolved from Visual Studio 6 on Win98 to VS.Net on XP. There's also been significant turnover in the development team over the years, adding to the learning curve.

Reuse verses Write

[NicknamesAreStupid]

There are 270,000 words in the English language. Most speakers know only a fraction and use an even smaller fraction. When they need to express themselves, they reuse the closest word that easily comes to mind or they make up 'slang'. Why? It may be a complexity issue related to the capacity of human memory, or it might be an emotional issue, or it is most likely is something that I don't even understand.

Coding is like that, but to a greater degree. Determining the 'code fit' may be harder than finding the right word, even if you have the library in your lap. It may take a machine to fix the coding problem, just like it took a book to catalogue all the words. Someday, if we can find the right words to describe the code we need, a machine will probably find it for us.