Slashdot Mirror
No More Coding From Scratch?
Susan Elliott Sim asks: "In the science fiction novel, 'A Deepness in the Sky,' Vernor Vinge described a future where software is created by 'programmer archaeologists' who search archives for existing pieces of code, contextualize them, and combine them into new applications. So much complexity and automation has been built into code that it is simply infeasible to build from scratch. While this seems like the ultimate code reuse fantasy (or nightmare), we think it's starting to happen with the availability of Open Source software. We have observed a number of projects where software development is driven by the identification, selection, and combination of working software systems. More often than not, these constituent parts are Open Source software systems and typically not designed to be used as components. These parts are then made to interoperate through wrappers and glue code. We think this trend is a harbinger of things to come. What do you think? How prevalent is this approach to new software development? What do software developers think about their projects being used in such a way?"
The immortal blue screen of death, following windows code down through the ages.
Are we in hell?
It's the UNIX philosophy - take tools, glue them together with a little shell scripting, and viola (sic)
microsoft will never go open source, the only code they reused in vista was the bugs!
Code recycling, and that word was chosen specifically for this case, is a time-honored and proven concept. One just has to look at Microsoft's products to see that.
If you think debugging is a pain NOW...
..."Objects"! Perhaps have a framework to enable the interconnections of these objects, perhaps entitled the Common Object Model. I think you're on to something! Quick - base an operating system around it! It would probably rapidly become the worlds most popular one!
I predict that this will happen shortly after all composers simply re-arrange themes and phrases from previous musical pieces in order to create any new compositions and authors simply splice paragraphs from existing books to form new ones.
It's always a long day... 86400 doesn't fit into a short.
More often than not, these constituent parts are Open Source software systems and typically not designed to be used as components. These parts are then made to interoperate through wrappers and glue code.
Isn't that the point of Free software? That it is flexible, that we have the source code so we *can* make it do just what we want, rather than be limited by the original authors?
What's the point of coding from scratch if you don't have to?
Microsoft is to software what Budweiser is to beer.
Why do geeks idolize authors who have no idea about how the real world works?
Machine language maybe. But if you are using the for loops, arrays, stacks, conditions, or whatever of your high-level language, you are already reusing code at that point. We just keep taking the theft to higher and higher levels. All good art starts with theft.
I certainly do not want to be developing the same common hardware implementation for a FPGA over and over and over again. Give me something the code for something that works and let me get to the stuff that is unique to the project.
Ooo man the floppy drive is broken. No wait. The computer is just upside down.
Love sees no species.
code reuse works best in languages like lisp and not so much in C/C++ family? I'm saying this mostly because in lisp I can drop down to almost any section of code instantly (most of the time, not always) while C/C++ IDEs tracking down a bug on large codebases can be a day long job frequently.
In the other languages, I often feel like treading in a minefield when reusing anything less than proven APIs or functions, let alone entire codebases.
If a piece of open source software is used as a component often enough, people will turn it into a component (through an API, plugins, software packages, modules etc). This improves the overall design of the software and allows better code reuse, but more importantly reduces maintainance issues by allowing developers to upgrade components with relative ease. I see this as a "Good Thing(tm)".
This sig is intentionally left blank
think Visual Studio!
This idea is being pushed by suits who think it will cut costs and increase profits. And it probably will, in the short term... as long as your customers lower their expectations instead of jumping ship. Welcome to the future of unmaintainable garbage software. (Then again, if you've been running Windows you probably won't notice the difference.)
Nice piece of science fiction but the reality is there are orders of magnitude more things that can be done writing new pieces of code from scratch than just combining existing ones. People paying for the software tend to make sometimes ridiculous demands on customisation of everything from look and feel to algorithms for doing the actual work for everything from transaction processing, science or graphics. Therefore new snippets will continually need to be coded.
What we're talking about here is the reuse of libraries. It's actually good practice to find a library and use a tested piece of code if you're doing something, than to go out and re-write it from scratch. My experience is that most developers are bad at searching for and reusing code from their own projects let alone from some vast archive (aka the Internet). The tools and processes in place for finding code that you can trust and that does what you want are so poor that at present we're far from even this level of reuse. Hell coders keep re-writing string handling routines that exist aplenty. But where do I find test cases and results for the existing libraries that I can examine. The quality of the libraries varies widely by language and area of functionality. If I trust the libraries I end up writing code that doesn't work and cop flak for it.
For example nothing wrong with Java String handling, but have you ever tried to use the standard date parsing routines. Hint: Even with lenient=false it does try to guess what you've meant to enter if you enter something that's close to being a real date. If you want to reject incorrect input, you've got to write it yourself. This isn't an excercise in Java bashing - every language has areas of weakness in the libraries - but it does explain why programmers often prefer to write it themselves. Even if they do decide to reuse, if they're not careful to retest the algorithms, they are taking big risks in terms of getting working functionality at the end of the day.
These posts express my own personal views, not those of my employer
I'm pretty sure that the novel of Vernor Ving that talks about this issue is "A Fire Upon the Deep" and not the "Deepness in the Sky". The later is a prequel of the former. "A Fire Upon the Deep" was also one of the first science fiction novels to portrait Usenet.
The promise of SOA says that you won't have to do this. If you believe in that promise then anyone that develops projects in the future will create them in discrete elements that are accessible as a web service. If you want to start a new development project, just utilize those services that you need and ignore the ones you don't. Because the functionality is encapsulated (and therefore, written, debugged and tested) within the service you're good to go.
I see application projects in the future acting like glue that holds many services like this together and makes them more than the sum of their parts.
I'm a big tall mofo.
...though basically well written, is full of dumb ideas. This particular dumb idea comes from the notion that the development tools we use now are pretty much as sophisticated as they're ever going to get. That's right up there with the famous urban legend about the proposal to close the patent office.
Vinge's problem is that he makes too much of the famous failures of AI, and has fallen in with the camp that believes that software will never be able to compete with wetware. That has yet to be proven, but even if its true, simply replicating human intelligence (so-called "hard AI") is not the only strategy.
This type of coding had been going on in the Windows world for many years (10?), using glue languages like Visual Basic and off-the-shelf "components". The hackers in the past seemed to deride the folks using this technology. The real code reuse will really take off under Linux only after a unified component architecture is established, similar to much-maligned, but very useful ActiveX on Windows. This plumbing should belong to the OS, at the very least to enforce just one architecture (Linus seems to keep good lid on the option proliferation). Please don't point to many component architectures already available in the open source world - the very fact of multiple choices makes component pretty much useless as anyone who tried to port a non-trivial program from Gnome to KDE knows well.
As the developer of freely available software, I find the prospect of people using my code a mixed bag. Partially, I feel an ownership of the code I write and am somewhat offended by the idea of people using it . However, as a paid engineer, I go through this at regular intervals; older projects get handed to others for support, as I work on new components.
On the other hand, I welcome the idea that my free code would be used by others - it is a flattering prospect, I suppose.
Others profit from this sort of re-use: COM, CORBA, Jars, etc...
A key barrier to code reuse is that technology marches on. Many advances render older code either obsolete or, at best, cumbersome. Recent examples might, for example, include adding templates to .NET, which rendered vast tracts of our .NET 1.1 internal class libraries obsolete.
Never, ever lose a file again. Ever.
Isn't this why we have layers of abstraction, such as high-level programming languages and external libraries?
Sure, if someone who only knew assembler from 20 years ago saw the GUIs present today, they might agree with the article. Imagine if you had to write a Windows application in pure assembler. It would be ridiculous! But we have languages like C# or the Windows API for C/C++ to aid with these kinds of things.
As we progress through time, programmers will simply use higher and higher level languages.
Unless we move out of 64-bit architecture, right? I mean, we'll have to by Sunday, December 4, 292,277,026,596 or else we'll have something slightly similiar to Y2K (or Y2038). But really, aren't a lot of virus' and spybots program this way? I wouldn't know; the only thing I could do related to reprogramming a virus or spybot would be renaming the exe file that the dumb@$$ has to open to get his system infected.
Ginga no Rekshiya Mata Each page.
Few people write their own C compilers, libc, math libraries, HTTP protocol handlers and so on. If other/bigger components are available, of course they will be reused.
I've been working on a MediaWiki modification that details every portion of the function/object/application creation process. The idea was that I could put a function or object together using pre-existing chunks in the wiki and then the end result will automatically be fully commented and based on code that has already been tested and proven.
No more buffer overflow exploits, no more null pointer dereferences, and no more need for 30+ libraries that have duplicate or similar code in them... or so the thought goes.
So far, I have a few entries targetted for the 6502 processor and the script that generates the end result code is mostly working... sort of. heh. Eventually, I'll fill it out with more useful stuff and then make it public, but until then, it's just a hobby to pass the time.
various "experts" have been talking this nonsense for years. it's never going to happen.
If you mod me down, I will become more powerful than you can imagine....
Idea about using decades old logic doesn't work. Someone has to mantain all that code: kernel, basic libraries, platform API's etc.
Problem is, software gets old, obsoleted by new ideas and when next-gen of hardware takes over, so updates are needed all the time. If you don't pay attention, you can end up with cruft like X Windows protocol which people are now trying to fix and improve to compete with other popular OS's.
Unlike in this novel, actual (best) code search won't need digging, it will be advertised, categorised, documented, just ready to be used.
Some basic tasks like file io or token processing and such minor things might be reused. But even then porting something so simple like a string tokenizer written for a full fledged virtual memory OS like Unix/WinXP to a fixed memory handheld device is highly non trivial, especially if you want to handle multi-byte i18n char streams
If the author sells what he was smoking while coming up with the article, he stands to make tons of money.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
Your code will be assimilated.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
The article is a bit of nonsense really. It ignores the fact that building software is a vertical problem anyway where most often you pick the highest level tool to get the job done. You now have
Transistors
Digital Logic
Machine Code
Compilers
C Code Family
Dynamic Languages / Visual Languages
What next...
In 20 years time nobody will be pissing around with C code or Java or or Lisp ( ok maybe lisp) except for historical/maintaince reasons. There will be new higher level constructs leveraging streamlined minimal lower level constructs. Many of the problems defined today by large code bases will be rewritten using less effort and more sophisticated expressive tools. In 20 years time there will be 2^20 cores on a chip, perhapps. I doubt code bits from today will solve those sorts of problems.
--
My slashdot captcha was "pervert" huh!
The bikini - security through obscurity since 1943
This new concept fascinates me. We can take these individual blocks of code. Let's call them "libraries", and join them together with some kind of programmatic glue and potentially reuse them to make some kind of application. This is definitely cutting edge thought. I'm not sure it will catch on though.
This is a sig
Unless you're working in R&D or making things that don't exist (a lot of game engines that are from scratch, a lot of open source projects, new protocole implementation, etc), thats probably what you do 40 hours a week. I know a lot of people for example who have to work with web services, and for a reason or another have to do contract first (because they do things their toolkit, if they even use one, doesn't handle), they often copy paste a WSDL template and modify it. A lot of database management tools have a librairy of template stored procedures. Googling for code snippets and samples is always good...
Especialy with environments like Java or
I was in a training class for Rational Software Architecture, I was 'told' that some company's have set it up and augmented the rational suite so that they can get about 85% of generated code before a developer has to touch it. There claim was the architect will be the main driver behind future development and the key cog in the enterprise.
isnt code reuse the cause of that major flaw?
no one questioned the code for years
back in the day we didnt have no old school
I tried to get RCA to move toward a system like this back in 1985 or 1986 but there was too much protection of each group's work to overcome; managers did not want to release "their" code to others even for use inside the one company.
What?!? Why is this news? We have libraries - those are the big chunks of useful code - packaged and (hopefully) documented for re-use and widely distributed.
When you design new applications you look for libraries that do the bulk of the work - and the application becomes mostly 'glue' to hold the libraries together. Scripting languages are the epitome of this - where large piles of carefully written and optimised libraries make up for the overhead of interpreting the actual library code.
Dunno why anyone finds this surprising - it's what we've been doing almost since the dawn of programming.
The tricky part is noticing when you can't find a suitable library - so rather than dumping a bunch of code into the application, design it as a reusable library. There is a small overhead to doing this - but rarely more than (say) 5% of the development cost. So when writing OpenSourced code this is generally done well. The hard part is in commercial code where you have someone with a chequebook and a severe lack of foresight breathing down your neck who would rather save 5% now than save 30% on the next project.
www.sjbaker.org
The Neverwinter Nights 2 Toolset was redone in .NET, but I checked the various components, and it at least used these:
- RAD Game Tools' Bink library
- RAD Game Tools' Granny library
- RAD Game Tools' Miles Sound System
- Crownwood Software's DotNetMagic 2005
- D. Turner, R. Wilhelm, W. Lemberg -- FreeType font rasterizer
- Glacial Components Software's GlacialTreeList
- Mike Krueger's (of ICSharpCode) SharpZLib library
- Divelements Limited's SandBar
- Various libraries done by Sano
- Various libraries done by Quantum Whale
- Davide Icardi's (of DevAge.com) Source / SourceGrid libraries
- Matthew Hall's XPExplorerBar
- Zlib
These were third party work used in the toolset application, and could be more as well (just checked standalone libraries, not any statically linked stuff).
Beware: In C++, your friends can see your privates!
I was speaking to someone at work last week and this exact topic came up. We have 5000+ batch jobs written in JCL and whenever someone needs to write a new program, they just mine through what's already out there and use it for the new stuff.
...
It's not that uncommon at my workplace
With companies like EA forcing insanely short deadlines to get Generic Sports Game 2007 out by the holidays, it pretty much forces their code slaves to reuse mostly everything in its' horrific state. If end product quality matters then "from scratch" programming will have a larger priority.
----------
Check out my blackbox styles
The notion of complete code-reuse and never coding anything from scratch is a very good idea in theory. The problem is that in order for it to work the components from which things are built have to be reliable and trustworthy, and such things are not always easy to find.
I've been studying errors and defects in engineering (both computers and otherwise) for many years. I was raised by an accident investigator, so I have an innate understanding of why things fail and what people can do to avoid it. The core reason for anything to fail is the assumption of correctness. In all accidents and failures, one can always find somebody somewhere that made an assumption about a component's behavior. The fact they did not know its true behavior caused them to employ the component when they should not have. Thus, the other elements of the system that depended on its assumed behavior also fail, and it just cascades from there.
In software, such assumptions of behavior often take the form of (to name a few) failing to detect null-pointers, failing to enforce buffer limits, or failing to enforce proper protocol operation. Opening the source to the world can help to detect such assumptions, but someone still has to check. As overkill as it may seem, I've been known to write unit tests for third-party libraries to ensure they behave as their documentation says. Many times I've found them to be faulty. Had I not tested them and just assumed their correctness, my product would not have worked. Even though my code may have been correct and the author of the libraries were at fault for their code not working right, I too would have been in error for making a false assumption.
Until such time as we can trust the components upon which we base our work, we cannot employ 100% code reuse. To reach such a stage, we need a radical shift in our approach to engineering. We need to ensure that our products do as they're supposed to and not just assume. Unfortunately, the rush to market often prevents us from doing what we should, and that's a shame. First to market is not always best to market.
You may not realize it, but as of today, virtually all new applications being developed re-use existing working software systems. For example, when you develop a new web application, the components that you re-use are: the OS, HTTP server, database server, scripting language (PHP...), etc. Of course nobody is re-developing everything from scratch. Very often only the higher level software layer (ie. your application) is developed partially or mostly from scratch, but everything else (ie. the internal or lower level components) are re-used from existing projects.
This has been a useful and common method of software development for a long time - at least as long as I've been programming professional, and we were taught it in school, of course, though we rarely did it, since the point was to learn how to do it ourselves before using other's code. Probably 85% of my new development work is tying components that either I wrote or someone else did into new and interesting configurations. The stuff I use (typically compiled code written by other programmers) was in itself designed from custom libraries of things other programmers wrote, and so on and back. There are pieces of legacy code that date back at least 15 years in our custom libraries. Code reuse makes designing and making useful software faster, more standardized and a lot easier to understand. Sure, sometimes you want to do it yourself and that's good, but really? The chances that your print routine or sort are going to look substantially different from anyone else's are low. Of course, none of that applies to research programming,as far as I can tell.
One of the problems is the necessity for glue code. There needs to standards for software libraries. You can have different libraries, but certain functions should always get you something that will work in the context. One should be able to take out any library from any piece of software, and replace it with another library (assuming there is more than one for the same purpose).
Stupidity is like nuclear power, it can be used for good or evil. And you don't want to get any on you.
While its easy to grab code and quickly put it together to make something that works, it is usually the licences and the difficult to interpret legalese that can limit this approach. This becomes relevant if distribution of the resulting software is required, and is especially relevant in a commercial context. Clean-room implementations are going to be around for a while to come, IMHO.
-- Mike
This has been the PHBs wet-dream since programming began. They see writing programs as assembly. It isn't assembly. It's design. You can't automate design.
They always talk about making generic components that can just be "glued" together to create a functioning application. You can't. That "glue" is the business logic, and it's what your program does. It's what you are paying the developers to do.
If you have good developers (skilled, not monkeys) they will create (or use) libraries to do most of the heavy lifting, but they still need to put those libraries together in a way that solves a business problem. If you could just automatically glue pieces together, how can you provide a useful product for your customers? Sure, you can take an "email system" and a "social networking" system and "glue" them together, but to do what exactly? Solve some business problem perhaps? How will you "glue" them? By developing business logic.
It's like taking two different ideas / products, placing them in a room together, and expecting "synergy" to create something new and great. Even if you have some great idea about how those two things can be combined to create something, you still need someone to do the design work of actually combining them.
You wouldn't expect an air conditioner and a storage locker to just magically combine together to create a modern refrigerator, would you? You need someone smart to integrate them. That is what a developer does.
We've heard it all before. Next story please.
It's truly disturbing to me how someone who put so much effort into writing a response could have completely failed to understand simple words like "all" and "any".
It's always a long day... 86400 doesn't fit into a short.
Look at recently developed programming environments that have made it big: Java and C#, running on a virtual machine, using a large class library.
My first impression of Java was of "a small language with a big class library" - I don't mean that in a bad way, the relatively small number of syntactic elements in the actual language and handing of things like say, threading to classes is a good thing.
But coding from scratch? What's that supposed to mean - typing ones and zeros at a OS-less motherboard? Working with all this support is the present and future of coding. Working without it is becoming a niche - for people who make the VMS, Oss, device drivers and class libraries that the rest of us depend upon. And yes, this trend for richer class libraries will continue. Duh.
My Karma: ran over your Dogma
StrawberryFrog
We're all leveraged when we can build on the good ideas and good code of others. The danger, in some organizations, is when they believe they can hire low-skill or non-programmers to put the pre-designed components together. As an old-timer, I've watched software development over the years, and there is a continual tendancy of enterprise managers to conceptualize programmers as low-cost, commodity labor. One company reorganized their development group out of existence, conceiving of this work to be done by vendors and rare consultants. The result is that they are frequently in a state of chrisis from their systems failing and from having nobody on staff with the skill to diagnose or fix them. Managers need to know that they need highly skilled developers, and they need to pay them what they are worth. These people can intelligently build on the work of others in a way that works for the unique problems of their business.
You mean A Fire Upon the Deep (http://en.wikipedia.org/wiki/A_Fire_Upon_the_Deep ). Deepness is a great book, but information complexity (and AI) is much more in Fire's area.
Makes me nostalgic for the days when I used to start with an idea and create the design from there. But even then, it was almost always the case that everything I built was built on top of, or in the context of, another software system. (well, there was this one time when I wrote my own assembler so I could write my own boot ROM for a machine you've never heard of... but that's not exactly normal)
Am I part of the core demographic for Swedish Fish?
Ever since the first person reused existing code.
This is just another step along the same continuum.
I wrote a small perl program the other day. If you count all the code that executed when I ran it, "my" code was only an itsy-bitsy-teen-weeny part of the whole.
"Move along now, nothing for you to see here."
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
okay.. this might work for your standard issue make-monkey-press-button-to-do-repetative-task
business drone software.. but I don't think you'll see any dynamically resized 3D model of a tank
or something as complicated hooking into a com.buzzword.WebServiceAPI over http for each calculation..
we moved beyond vax hardware speeds for a reason.. and sometimes that reason is not so you
can bloat your network with spurious XML-over-HTTP traffic
Whilst this is an uber fantasy for minimizing cost and development time, it will never exist. You just have to look at the amount of bugs that exist in software, they will never really vanish and if we're not writing bespoke software they are likely to double. The text talks about architecture within the project being what will replace the hard work, however software architects exist only in an a project at the start. It's rare to bring them in at the end and can you imagine the hell you'd go through having to debug some alien code if it's not working? The people who make up architects in my University are just the people who are trying to make money for Computer Science, I pretty much see them as a lower form of a professional. The real Computer Scientists are those who bring new concepts, not those who spend all their time planning. I accept we need people from other disciplines to enhance our working practices but it should never be viewed that these people will take over the core subject. For example we all know Computer Science was started from a bunch of crazy Mathematicians (Just look at Babbage and how he was publically ridiculed because he didn't complete his thinking engine), surely someone could also abstract from that time that we won't eventually need Mathematicians because Computer Scientists will still be needed. But we still have the crazy Mathematicians as they are still required for the absolute core stuff. From those principles let's honestly look at how businesses work, You'd never get such a repository put together of lots of clever code. A 100% bug free module for a specific task would cost more than the moon. (About £8.8x10^22 in Cheese Pounds)
Code re-use is a great idea. Free software is a great idea. Taken together, and to an extreme, they can cause problems, particularly where security is concerned. What happens when someone finds a security flaw? How can you contact the people who are reusing your code if you have no idea who they are?
To take a personal example, my delta compression code, which I wrote for FreeBSD, is now being used by Apple and Mozilla to distribute updates; I've talked to their developers, and if I find a security flaw in this code (very unlikely, considering how simple it is), I'll be able to inform them and make sure they get the fix. On the other hand, I know developers from several Linux distributions have been looking at using my code, but I'm not sure if they're actually using it; and searching on Google reveals dozens of other people who are using (or at very least talking about using) this code.
Putting together software by scavenging code from all over the Internet is like eating out of a garbage dump: Some of what you get is good and some of what you get is bad; but when there's a nation-wide recall of contaminated spinach, you'll have no idea if what you're eating is going to kill you.
Tarsnap: Online backups for the truly paranoid
economics...That is, until the hardware demands recoding, reuse is both financially and programatically a generally sound philosophy. Of course GIGO still applies :)
errr....umm...*whooosh* *whoosh* Is this thing on ?
It won't happen that way until there is a coherent established framework for functional testing of the "building block" code. It is GREAT to reuse code, and I've worked on a few projects that became GPL due to my incorporation of GPL code. However, most professional projects I've worked on require more on the reliablity front so no time (time == $$money$$) is lost. Any reused/mined code incorporated into these projects has had to be so rigorously inspected and tested for function that when combined with the time spent on banging together glue code, we may as well have started from scratch for those code chunks.
The exception to this is tiny code chunks that either have a simple testable API (eg. no functions that return values variable with time), libraries/code that have been so widely used that you can "mostly" assume that it's valid (eg. zlib), or occasionaly some banged together prototype "proof of concept" app.
Until some standard framework is created that the code can be tested for validity by those incorporating the code, it likely won't happen fast (other than for the exceptions listed above).
For the record, I don't know what that standard framework may be. Personally, i'm in the habit of whenever i write a class, i include a static test function in the class that can be called to put it through its paces and check for validity on all functions and expected data alignments. (eg. on some arm processors i've worked on, a struct is packed based on a 4-byte boundary, and sometimes extra "surprise" filler bytes are inserted. Lots of fun to track those down when you're not expecting them.)
Demonstrant's Open Source Tools
The problem with this concept is deceptively simple:
:)
1. The most difficult part of programming is developing software that does a good job of unanticipated uses
2. Most programmers are fine with something that just does the job they need it to do, and no more
3. Relatively few useful software components can operate in isolation, the rest place requirements on the environment
The consequence of this is that contrary to the theory that writing something once should be sufficient, the truth of the matter is that in the vast majority of practical cases, it is simpler to write the code once again, bespoke for your particular needs and environment, than it is to pick up the best available pre-written component to do it.
The essence of the problem is one of assumptions. Any component has an environment, this is the sum of state and information it requires in order to be able to operate. The component also (probably), has an API. This is the formal environment, the sum o state and information that it makes explicitly available to the programmer.
The problem that you have is that there is no "right" place for the percentage of the environment to be formalised into an API. For all those of you who are thinking "Well, the answer is obvious, everything!", imagine a component that manipulates dates.
Think about this for a moment, I use it because it's an example which almost every language I have encountered has done "poorly" (where poorly is the situation in which I find myself implementing something I felt should have been there by default, like adding or subtracting a month).
The environment for a date component, the true environment, is larger than any date component I have ever seen has exposed. For a start, which calendar are you working in? what timezone? what are the DST considerations for that timezone? is there some mechanism for updating these DST settings for those crazy timezones that change DST times all the time? is the begining of the week Sunday or Monday?, what is the current locale? If I add a month to the 31st of October, is it now the 30th of November, or the 1st of December? what if I subtract a month? is month addition associative?
Keep in mind that every question there has been solved at some point or another, and the majority of it is solved by the few really good date libraries out there (perl has a particularly fantastic one the name of which escapes me), but I've never seen one that did a good job of updating its daylight savings information, the majority of the *nix style ones simply assume that the operating system has good information, normally the catalog that came with your most recently install of glibc.
Anyway, back to the point, the issue is simply one of appropriateness. There are some components that make a lot of assumptions, but their interface is simple. There are others that make very few assumptions, but working with them is mind bogglingly complicated. Anyone who has ever had to work with mapserver or equivalent mapping problems will understand the gradient. But it is not at all uncommon to find yourself in a situation, even with something as well covered as dates, where the conventional components don't cover the part of the environment you need covered. Interfaces are either too big and complex, too small and simple, or just focused in a different area (how many components have been dropped because they were perfect, except that they didn't handle multithreading?).
In the end, rewriting this stuff is going to be a fact of life for a long time yet (unless we get really smart AIs or a revolution in programming I can't foresee), although hopefully better coverage of more common problems will continue to be a feature (it's much easier to do things today than it was 10 years ago basically because of this, regardless of what the lisp guys say
You can't win a fight.
and it's called CPAN.
"For a successful technology, reality must take precedence over public relations, for Nature cannot be fooled"
I think that we are not going to be "mining" for code so much as using smarter and smarter tools to develop the code in the first place. Historically, we used to code by flipping bits on the bare metal. Later we developed smarter tools called assemblers and loaders to flip the bits for us. Then we developed "high" level languages like Fortran, Pascal and C, and developed tools to compile our programs. Our tools became smarter. We don't write the code for GUIs anymore. We drag and drop widgets in a design software, click a button, and reams of code get written for us. Object oriented programming could take us farther still, as the GUI elements could already be pre-defined. (Take a card game for example. All the methods for manipulating and displaying cards, and stacks of cards, etc, are already all in place All the programmer has to do is code the rules of the game.)
As we progress in abstracting the data we deal with, we remove ourselves from raw coding. Once we start abstracting and manipulating programs as data, we will enter a realm where we will be able to do stuff like:
Programmer:"Give me a klondike solitaire program, but extend it to a crown and anchor deck".
Computer:Klondike Solitaire found.
Computer: Unknown term 'crown and anchor deck' in context Klondike Solitaire.
Computer: Unknown term 'crown and anchor deck' in context card games.
Computer: There is a dice game with crown and anchor symbols.
Programmer: "Create card deck variant crown and anchor deck by adding a suit with symbol crown, and by adding a suit with symbol anchor."
Computer: Card deck variant crown and anchor deck created. Shall I make this card deck variant publicly available?
Programmer: "Yes"
Computer: Klondike Solitaire Crown and Anchor variant ready.
When our name is on the back of your car, we're behind you all the way!
Often it is desirable to reuse someone else's library. But often it is still very valuable to provide the code yourself, to prevent another dependency on someone else's code, and also avoid something that might limit portability.
I think it is the same cost/performance threshold that people use in everyday life, in making a decision to purchase or "do it yourself." Is using a third-party library worth the hassle of integrating and packaging it?
For example, one developer wanted to add a default dependency on the enormous Boost C++ library to our project in order to provide a chart-drawing ability. Was it worth it to us to add such a burden? No. Would it be beneficial to an app whose purpose is to draw charts? Probably.
We will add your binary and technological distinctiveness to our own. --the Borg
We're all hypocrites. We all have hidden parts, it's the contrast between them that make us more a hypocrite than others
This is pathetic. Let's look at two examples of software systems which builds on _lots_ of existing systems: PHP. This builds on _old_ wellknown software (png, sql, etc etc) to fully extend the language. People hate it, but don't have anywhere to run (unless you wanna go with the java monster). Now there's Ruby. Not necessarily better, but people run to it like crazy, just to get the hell away from php! -> No hit. Gnome. This builds on completely new software written from scratch just months or very few years ago such as: avahi, cairo, dbus, poppler etc... -> Hit. This article is baloney, and we'll continue to have these moronic future-visions of a world without [insert a moron's view here]. Such as "paper", "software written from scratch", "war of terror", "republicans"... These might all be heart-breaking positive flower-power ideas, but they ain't gonna happen. I bet this "Susan" has never ever read one line of source code, but she's the "management" kind of person. You know, those who just doesn't know anything, but they're paid more than you, to screw with you, making your job a nightmare, not doing any good themselves, except "managing" things... *shrugs*
There was another SciFi story which was similar, where code became an evolving creature, and that apocrethary like individuals would visit the realm they inhabit (the Wildnet : a multiworld internet of sorts) and bring various specimens back. Problem was if what they brought back had evolved things not intended... as the realm which code which was used day to day needed to be protected from the wildnet evolved code which could have defenses or offensive capabilities designed to take them out... I wonder if that would be the next step... The Book was Night Sky Mine by Melissa Scott.
due to massive code reuse we now suffer from the "mindrot" disease that disables us from re-inventing anything, even after we have long forgotten it.
That's why there are many times more Microsoft developers than there are for any other platform. MS has made programming insanely simple. DLL's and other forms of COM objects make code reuse very, very simple. I can make a application with a functional web browser embedded in it in under 30 seconds. Can't do that on any other platform that I'm aware of.
That's what is currently happening....for instance, take VB. Each line of code you write isn't an individual assembly instruction--it's basically a prewritten piece of code....
All this guy is doing is taking it to a higher level. It sounds exciting, but is really obvious :-P.
Ride the skies
We already have reusable components, they're called libraries and they keep getting better. But to think that we can ever easily take some piece of software that isn't written to act like a library and reuse it for something else is pure fantasy. If you have a library with a design specification, you have a bug. If you have a piece of software, it might work quite fine until suddenly somewhere in a mangled mess of code you have some implicit assumption somewhere (which may have been perfectly reasonable) that you just broke. You can't retroactively redefine the code to adhere to a specification, which you obvioulsy need to use it as a component. Sure, you can build some test cases and pray really hard that this black box won't fail on you but that is rarely satisfactory. Or maybe you have so stringent control on your code you can drop it in anywhere, but in that case you'd be the first such application I see.
Live today, because you never know what tomorrow brings
In certain situations this is useful and is already being used to an extent. But rabid use of it is generally bad. Remindes me of hammer factory factory
What about web services. Aren't they pretty cross-platform and language independent?
Isn't that where we want to be eventually?
More often than not, these constituent parts are Open Source software systems and typically not designed to be used as components. These parts are then made to interoperate through wrappers and glue code. We think this trend is a harbinger of things to come. What do you think?
It is a harbinger all right. It is a harbinger of loads of work to come, straightening out that rotten stinking spaghetti. Hint: all you need to do to establish your reputation as a top open source coder is to dig into one of those piles of poo and fix it. Oh, and don't bother asking anyone for permission, just do it. That is the way it works.
Why not start with anything a Ximian coder has touched, since all that code needs to be forked anyway now that Ximian has jumped into bed with the enemy. Suggestion: throw away the bonobo and orbit interfaces, they are just bloatware. Reroll using dbus. Throw away Gnome VFS. Reroll using our real VFS. Lather, rinse, repeat.
Have you got your LWN subscription yet?
That's where having a global database of excellent code comes in.
Software engineering should be a much more tightly regulated. Right now, anybody with a C++ for Dummies can call themself a software engineer. Software engineers should be as regulated as engineers in other disciplines. There should be best practices, peer reviews, etc.
When our name is on the back of your car, we're behind you all the way!
The way I learned how to program in pretty much any language was by looking at other people's code, seeing how they did things, and duplicating those methods in my own. Eventually, I get good enough at a thing that I can come up with my own ideas, or that I remember how things are done without having to go look.
Libraries
I personally enjoy not reusing things. It makes me feel dirty.
This just illustrates how little people understand software development. If blind managers were designing houses I'm sure we would end up with the boss saying "here is a functional bathroom - can you patch it to this functional kitchen?". Well maybe yes - and the toilet would be under the table.
Finding analogies is difficult.
Why not walk into a roomfull of stories written by a creative writing class and try to cob them together into a best seller?
Many years ago IBM was touting the line that with the new faster hardware companies could get away with cheaper programmers. The latest incarnation of that idea is outsourcing to 3rd world countries. Now I'm not trying to run down programmers who are born and live in the 3rd world. What I am saying is that trying to cut costs by gutting programming talent is foolishness and it illustrates a complete lack of understanding on the part of people who coger up such notions... regardless of what guise they do it in.
The only thing that comes from patching together gobs of code is a mess... and undocumented and unreliable and unsupportable mess.
The whole idea is just pure bull.
I'm pretty sure that when you collect pieces of code and glue them together from a higher level of abstraction, you have a programming language. I mean, a simple addition operator in a high level language is several opcodes in assembly for moving stack pointers and such. What this does demand is that the open source community would make the best impact on the world they could by arranging a very detailed archive of code that would cover a wide range of functions, and all be designed with reuse in mind.
I hate to be so blunt, but anyone debating this is an idiot. Code reuse has been with us a long time, and it's the only way that exponentially more complex systems will be built without exponentially larger teams. If you are thinking "but I never cut and paste code" then ask yourself if you ever use a library, for example (standard template library, OpenGL, even stdio.h functions used to be coded by hand 30 years ago). Or if you code in a higher level language where useful functions are wrapped up for you (an entire web browser as an ActiveX control, anyone?). Yes 90% of code is crap. But it is ridiculous to think that we shouldn't build on the 10% of good stuff that is, or can be, neatly encapsulated.
I forsee a future where author-archaeologists compose books out of fragments of old books. I suppose "cribbin" the archaologist who never left the library from Asimov's novel, Foundation is poetic justice.
It's a fact.
This is exactly how DNA works. Programmers have so far not had much success at this approach, one because our computers are puny, and two because our programming practices have been tailored for engineered code. But as hardware gets fast enough that most common tasks can be run at a one-millionth slowdown and still run fine, we will get to a point where we can write glue code that runs some other piece of code, throws away 99.9% of its computation, and only uses the rest, simply because of the value of human time vs. computer time. I have written code like this. It was just research projects, not "production systems", but nevertheless it is possible that the future will have less and less coding from scratch.
Some context for people who didn't read the book... or didn't read it carefully enough.
First, Vernor Vinge has a PhD in Computer Science. This obviously doesn't guarantee he can't be wrong, but to those commenters who said something like "these ideas are idiotic"... you've got an uphill battle to convince me that you're that much smarter than Vernor Vinge, especially as most of you saying that don't show me you understood what he was saying in the first place.
Second, A Deepness In The Sky is set in his "zones of thought" universe. In this universe, the fundamental limits of computation vary depending on where you are physically in the galaxy. This is only faintly hinted at in A Deepness In The Sky, it is explicitly spelled out in A Fire Upon the Deep. This limit on computation may or may not be real. One of the effects of this limit on computation is that you can build a system larger than you can really handle, and eventually all such systems come apart in one way or another. This story is set thousands of years in the future and it is explicitly (albeit subtly) pointed out that the software running the ramscoop ships has direct continuity with modern software. (Qeng Ho computers use the UNIX epoch as the most fundamental form of timekeeping; apparently even the relativistic compensation is layered on top of that.) We are at the very, very beginning, where it is still feasible to burn an OS down entirely and start from scratch.... or is that really still feasible? (Perhaps Microsoft will soon find out.)
Those of you posting that "we can always wrap it up in an API or whatever", I'd say two things: First, you get the Law of Leaky Abstractions working against you. The higher up the abstractions you go, the more problems you have. (Look it up if you don't know what that is.) The more sub-systems you make, the larger the state space of the total system becomes (exponentially), and the harder it is to know what's going on. It is entirely plausible that you eventually hit a wall beyond which you can't pass without being smarter, which, per the previous paragraph, you can't be.
In other places in the galaxy, you can be smarter, and Vernor Vinge postulates the existence of stable societies on the order of thousands or tens of thousands of years or beyond, where the society actually outlasts the component species, because the software base that makes up the species does not exceed the ability of the relevant intelligence to understand it.
Both cases (software might exceed intelligence, intelligence might grow with software) are extremely arguable, and I do not think he is advocating either one per se. (Leave that for his Singularity writings.) But you do him a disservice if you think he is not aware of the issues; he's extremely aware of the issues, to the point that he is the reason some of us are aware of the issues.
(Even this is a summary. In isolation, probably the best argument is that it is always possible to create a software system one can not understand or control, but one person can be wise enough to avoid that situation. However, in A Deepness in the Sky Vernor Vinge explicitly talks about how in a societal situation, one can be forced by competitive pressures to over-integrate your system and make it vulnerable. "OK, but the government can be smart enough to realize that's going to happen and step in to stop it." First... smart government? But even if you overcome that objection, now your society faces death-by-surveillance and other automated law enforcement mechanisms, which since they can't be human-intelligent will fail. If you avoid that (and it is a big temptation), then you face the problem of anarchy. And remember that "governance" is anything that governs; even if the "formal government" doesn't regulate you to soceital death, private corporations may do it. Anyhow, upshot, Vernor Vinge has done a lot of thinking on this topic, it shows in his books, it is not showing in the criticisms I've seen posted, and when it gets down to it he really has more questions than answers.)
Hey it was more stable and easier to write for and you didn't need 10+ gig to install it.
Hardly anyone programs in assembly. Why not call them "fortrans" because they're translations of a formula for a component?
Unfortunately, this already happens. The standard for a "substantial" portion under US copyright case law is so short that most melodies are likely already taken. Spider Robinson wrote a short story about such a situation.
It is also arguable that some people are afraid to write music due to the fear of being sued for subconscious infringement, losing the case, and possibly having to sell the house to pay the damages. For examples from US copyright case law see Bright Tunes Music v. Harrisongs Music and Three Boys Music v. Michael Bolton.
Perl has had this for 8 years already. It's called the CPAN.
All the components come with tests integrated into them. When you try to install the component, if the tests don't all pass, then baby it don't install.
Which is why the slogan "90% percent of every Perl program already exists on the CPAN" is so apt.
Same here - when I was learning VS.NET, there were so many common routines that used a simple but annoying-to-type algorithm that were I using a language like C/C++, would just provide more code to put typos in or forget a keyword. Once I unlearned doing things the long way, I would just type in the first few letters of the function I wanted, pick the right suggestion from the list, and watch as one line does something like connect to an SQL server over a network.
I'd say we're at the point where code reuse is not only ubiquitous, but transparent - at least in some languages. There will always be fundamental building block languages like assembler for those who NEED to get down and dirty with the hardware. Personally, I'd like to do more of that, but I can't really justify it since whatever I'm doing, I can afford to sacrifice a little CPU or RAM in exchange for saving 30+ hours of coding.
Comment removed based on user account deletion
A lot of this may come from software license incompatibilities. Publishers of proprietary software don't want some copylefted library such as libiberty or readline "infecting" the entire project, which may contain valuable trade secrets or patented processes, with the requirements of the GNU GPL. Even in free software, it's impossible to use GPL and Apache license software together.
Whenever I glued code by a wrapper i shot myself in the foot very efficiently. Wrappers are the last resort. Wrappers are only acceptable if the coder of the code to be wrapped planned his code to be wrapped. If you use a wrapper adding semantics to a component included by the wrapper is a pain in the ass-you have to change three parts instead of two. If needed or if deployment time is near and there is no time to adopt the code in a good way, then a writing a wrapper has to be carried out with a lot of thinking.
My paradigm would be: only use things as components which were meant to be used as components.
Well you have to admit that would be convenient... no need to get up after a big meal.
I don't care if it's 90,000 hectares. That lake was not my doing.
If anyone cares, this sounds an awful lot like a variation on a theme by Poul Anderson, from an early novel of his called "Brainwave". Premise: the earth has been inside a region that physically supresses clarity of thought throughout all of human evolution. The region shifts, and suddenly everyone become orders of magnitude smarter than they have been.
I recently read an article in ACM Communications about code reuse and which demographic is more likely to reuse software artifacts. (This may have been an issue of Communications from last year; I'm catching up). The study found that new software developers are much more likely to reuse components than old hands. The major reason was pretty simple: old hands are more likely to have a bit of ego and fall prey to the NIH syndrome. Newbies, on the other hand, are worried about their job (or standing, in a community), and so will tend to reuse existing code rather than write anew. This study was conducted among software development companies and enterprises, not among open source communities. But code reuse in communities seems seriously lacking. Why is that?
I don't think we're close to the end of coding from scratch, at least not in the sense that coding from
scratch will be replaced by grabbing existing F/OSS parts and combining them. And the reason is
exactly because
More often than not, these constituent parts are Open Source software systems and typically not designed to be used as components.
Trying to glue things together that weren't meant to be used that way can sometimes be done, but it's rare that it works well and it's usually a major kludge. Until people start building "pieces" that *are* meant to be used as components of a bigger system, then no.
That said, the recent trend towards using coarse grained services / SOA, is a step in that direction. When people start writing apps
that are designed to interoperate as part of an SOA, then I think we'll see a lot more reuse. But right now that's just not
the case with the major open-source apps.
If someone wants to try an experiment, try taking, ah, let's say SugarCRM and SQLLedger; and combine them into a cohesive system. Unless
you're masochistic you'll probably decide it would be easier to write an integrated system from scratch. This is assuming that you start
with a policy of NOT modifying the base code of either system (after all, who wants to be on the hook for maintaining a private fork of something like that, for basically forever?)
// TODO: Insert Cool Sig
Let's be honest here. How many people call themselves programmers and have no idea what a zero flag is or what the difference between Big and Little Endian is? The current level of "being programmer" is to be able to hack together a few lines of code with pre-set functions that pretty much do the work for you. It's actually frightening to see graphics programmers who don't know how to normalize a vector because "there's a function for that".
Fine, of course there is one, and most likely it is by heaps more optimized than what you could hack together, but, heck, at least KNOW what you're doing!
Current schools squeeze out codetappers, but no programmers. They know a few concepts, they've learned the essential functions of one or two languages (most of the time Java and VB, or now C#), but put them in front of another imperative language and they're lost because their functions ain't there. And no resource kit, teh horrorz!
Of course, those people are cheaper. I get about 2k more than the guy sitting next to me, and probably only because I can see the difference between a CF and a ZF. And that I can accomplish in 2 hours what he does in a day, simply because as soon as there's no function for it and he'd have to figure it out himself, he's lost.
A lot of people who are squeezed out of school today know their tools, but they do not know the theory behind it. They don't know why they do what they do. They only know "Write code A and computer does B", but they don't have a touch of an idea why this happens.
And that's where buffer overflows come from. Because, hey, what the hell is a stack? Is there a function for that?
To get the rambling back on track, I could well see that "program-archaeology" is the next logical step in this. Actually, it has already started.
Or is anyone here that didn't resort to example code or open source code to find an algo that does what you wanted to do but couldn't figure out?
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Truely ... a sad day when a "Hello World" application is 1.5Meg ....
Moving to high-level programming languages doesn't seem significantly different than what Vinge is (accurately, I think) predicting. We already reuse code when we type our code into a computer instead of writing hex by hand on a legal pad. Just like human language increased in the complexity of the ideas it could communicate with simpler particles, the language with which we talk to processors continues to grow and pack more implied "meaning" into smaller and smaller informational bits. Functions, subroutines, classes, OOP...we've been moving in this direction for a long time. I'm not smarter than Vinge, but I would say that as a trained computer scientist, he might have tunnel vision when it comes to what programming actually "is", simply because he was so thoroughly schooled in it!
A piece of advice: saying "only real [insert any profession/hobby] do [something]" sounds a bit like you feel the need to prove yourself. And believe me, that is not good here on /.
You know, there is a difference between trolling and pointing out the flaws in your reasoning. Just saying.
Before one could reuse sourcecode on a global scale I think we first need to switch to a more advanced programming language, probally one not even invented yet. Today a hell of a lot of code does nothing useful, its just there to duct tape together different APIs or to make the compiler happy, while doing nothing by itself. And this isn't just a small portion of the code, but often a very large one, like half of the application or even more. Before that 'useless code' isn't reduced drastically I don't think reuse on a larger scale can work, since it would simply end in even more wrappering and duct taping.
If a system fails any of these three criteria, the pressure to reimplement will be irresistable.
Vernor Vinge is an adequate science fiction writer, but his ideas about software engineering are purely fanciful. The very concept that you could replace actual programming with a purely investigative procedure has clearly never actually done any programming. Even if this were possible, it wouldn't be profitable: for any reasonably complex program, the amount of effort needed to understand the program well enough to repurpose it is, at least, equal to the amount of effort required to write a completely new implementation. Very few software systems, and almost no individual components, meet all three cirteria to resist re-implementation.
Device interfaces change. Communication formats change. Platforms change. Programming languages change. Worst of all, requirements change. Any one of these changes is enough to cause a complete rewrite of a component, even if that component has completely open source code. If a proram's source code is unavailable, or is written in an obscure language, the cost of re-implementation in the face of changing requirements can be a tiny fraction of dealing with the existing program.
just a ghost in the machine.
"A Deepness in the Sky" is a excellent novel. If you like SF, it's a must read.
Those few Slashdot readers for whom buying a paper book would be impractical can get an electronic text on a pirate channel. <pirate> Launch an IRC chat client (Opera browser has one built in) and join the room bookz on Undernet, Europe. Once there, enter the line
!EBrarian Hugo 2000 Winner Novel - Vinge, Vernor - A Deepness in the Sky.rar
For simplicity I have skipped the search stage here, listing a file that would probably be accessible for some time. You could get a list of all files by typing e.g.
@seek Vinge Deepness Sky
</pirate>
17779 eligible voters in a district, 17779 'vote' as one. This is Russia.
Likewise, how many glue codes have been dropped because they were perfect, except that their developers didn't handle execution models with multiple processes and one thread per process? Conventional wisdom is that multiple processes communicating through message passing (e.g. local sockets) are easier to get correct than multiple threads communicating through shared memory.
I use libraries when I write code. Sometimes I use those libraries to write other libraries (which I later use). My boss can write directly in assembler when it's needed, but he's still calling existing system functions which are just inaccesable from C. If you really want to get down to it, the people who made those system functions were just using the instructions already built into the hardware, which I hear is increasingly being created using higher-level tools.
-- 'The' Lord and Master Bitman On High, Master Of All
SOA is a nice concept (and buzzword). At the local-machine level, this is exactly what unix has always been. You have a bunch of programs with a standard interface (text through pipes/stdio instead of xml through http) and you tie them togather with "glue code" shell scripts (or perl).
More often than not, these constituent parts are Open Source software systems and typically not designed to be used as components. These parts are then made to interoperate through wrappers and glue code. We think this trend is a harbinger of things to come. What do you think?
Three words : Not gonna scale.
since the advent of assemblers, I doubt that anybody really codes from scratch anymore.
Alastair Reynolds : Century rain. " It's all right,' Niagara said. 'I won't be the least bit offended if you call me a Slasher. You probably regard the term as an insult.' 'Isn't it?' Auger asked, surpriced. 'Only if you want it to be.' Niagara made carefull gesture, like some religious benediction: a diagonal slice across his chest and a stab to the heart. 'A slash and a dot,' he said. 'I doubt it means anything to you, but this was once the mark of an alliance of progressive thinkers linked together by one of the very first computer networks. The Federation of Polities can trace its existence right back to that fragile collective, in the early decades of the Void Century. ,,
Slashdotters are murderous genocidal maniacs in that book ;-)
Oh no!!! A science fiction book talks about something - fud! I program from scratch everyday at work and it hasn't slowed down.
One of the problems that is occuring now is installing some of these programs. They may not list all the other programs you need first. This causes installs to fail for various reasons, missing header files, missing libraries, incompatable compiler options, incorrect versions of things.
It can be a nightmare with some programs tracking down all the dependencies. I've been there with several flavors of *nix and also with windows.
As reuse of code and/or code fragments becomes more and more common the problems with getting all the right stuff that is presumed to be installed located and installed to make life happy.
I recently put together a browser like tool in about an hour that would do a task in a matter of minutes that would have taken a week to do manually.
Of couse I could have coded it from C, built my own booting OS and something that would have done the tcp/ip then parsed the results into something recognizable as english text and written in hundreds of potential responses to whatever I got back and it wouldn't have taken more than a decade. Yeah, that would have been "code reuse" too since I didn't write C but I'm not sure that even with a decade I could have done it in binary.
Are we there yet?
B) Eliminate all the stupid users. This is frowned upon by society.
I've been using some open source components for a recent project, so this seems pretty relevant. I needed to upload a file from Java, so used the jakarta HttpClient. Works pretty well! Likewise, I needed to handle uploads, and javamail was stripping unescaped backslashes sent by IE/WIN. Jakarta to the rescue again, they've got a handy project just for parsing multipart form data.
Both of these were things I could have implemented myself, but certainly ended up saving a lot of time and probably bugs. I was impressed with the level of documentation on the apache jakarta site, it gave me the information I needed to start using the things right away. Plus, having the jakarta project name attached to them gave me some confidence that they weren't utter garbage.
But most importantly, each of the libs is small and single-purpose, and doesn't try to be a swiss army knife. It was very nice to find two existing frameworks that did what I want, and not much more.
These are projects I would happily contribute to.
You drank my drink, you drunk!
If this is what the future holds, then the average project will be so bloated that Vista will look like a "hello world" program by comparison.
You don't win friends with salad.
And you won't make friends with Simpson's misquotes. ;-)
MS has made programming insanely simple. DLL's and other forms of COM objects make code reuse very, very simple.
It's hard to beat the pipe | for insane simplicity.
By definition, command line unix reuses the output of one small program as the input of the next on all text based code. It's simpler than any dll linking and has been around since long before DLLs in Windows, Window itself or the MS DOS Windows initially sat upon (and then spent years trying to pretend it didn't).
This has always been Windows' strength and weakness. Unless an outright coder, you have no way of chaining one program to another. Power users resent having to code to do something that's just right there in CLI systems. On the flip side, with it being so much less accessible, the average user doesn't feel forced to learn the more complex process (sorry linux users but the average grandmother doesn't want to hear "Oh, you just need to grep the output of your find for..." and isn't going to adopt your platform while she assumes she has to do so to get anything done).
I remember creating subroutines to draw a circle on the screen, or fill that circle with a colour. I remember writing lots of different code that is now considered part of the standard C libraries. I think the key to proper code reuse is the same as it has always been: discrete functions that have properly documented uses and predictable results. I think the reason why some people have issues with code reuse, is because they aren't reusing the code properly. Instead, a program or process that sort of fits the parameters is used and then it breaks because something happened the programmer didn't originally anticipate. There will still be a lot of coding to do even with reusable code, since existing code isn't going to take into account every situation. The bottom line is, code reuse already happens today, and will continue. In order to do it properly, it will just come at a slower (but more effective) pace.
--
Luck is just skill you didn't know you had.
I can definitely agree that we are already there in some areas, like large systems. At least in the open source world in my opinion. Think about any task you'd want to accomplish... Does it involve networking? Almost certainly you'll want to reuse some networking library. GUI? There are plenty of GUI frameworks to choose from. An application you use lacks feature X; would you rather write a new application from scratch with that feature or add the feature to the app you use?
As a matter of fact, I find it lamentable that so many programmers are fumbling around trying to solve problems that have already been solved. I once had a book (part of it consisted of microfiche sheets, so that tells you how long it's been) from the ACM containing all the published fundamental algorithms. True, some of them were in assembly, COBOL, FORTRAN, and other languages that aren't popular now, but the theory was that programmers should be working on solving new problems, not reinventing solutions to old problems. Translating an algorithm from a different language is preferable (in productive mesurements) than reinventing it yourself.
Vinge has a point, and although I agree that it makes for a good story (because every story needs a problem), it is not good science: Surely those of us working on automated programming systems and generators will also try to invent a method of description that succinctly defines the outcomes of our program segments, and then we will use some type of genetic algorithm or other method of making the program most efficient within the constraints of the outcomes. In other words, software will be "grown", not coded, but the description of what is desired will include a constraint that makes it efficient. Furthermore, most of the tasks and workflow will be described in a similar language, so if a task has already been described, the method of solving it may have been cataloged and "component-ized" so the archeologists' job will be as simple as following a circuit diagram. However, IMO, that part of the problem will be done by robots...
I think the current code repositories and model repositories are precursors to huge "code warehouses" or "model warehouses". The program designer will define the task, the program generator will decompose the task description semantically and use some type of pattern matching to find a suitable "component" in the repository, generate the program or system according to some type of design or artistic standards, and then grind the system 'til it becomes highly efficient.
I believe we are closing in on a time when program development will be more akin to sculpture than logic.
"The mind works quicker than you think!"
The joke was that first there was reinventing the wheel, then there was copy and paste coding, then there was COM, then there was .NET with assemblies, and the next generation after that will have what came after assemblies.
This is stupid. We already use multipurpose tools for generating the code we want - we call them compilers. What they're talking about is the simple process of building up higher and higher level code. We're no longer coding in punch cards or buttons on the front of the computer, are we? If you had made the same prediction back in 1960, you'd have "code archaeologists" going back and finding old punch cards to translate into programs today or maybe people reusing the same bits of assembler instead of coding it from hand.
Did you ever notice that *nix doesn't even cover Linux?
Wouldn't that be a step back? I mean, someone has to write those blocks of code that you reuse... and if no-one does, 1980 will be calling soon, saying they want their state-of-the-art sources back...
Pentaho, the Open Source Business Intelligence Suite is a very prominent example of this new approach on development. It consists of well-known projects like JFreeReport, JFreeChart, Jasper, Kettle, Mondrian and others.
What the Pentaho guys do beyond marketing is writing glue code and user-friendly development tools. They also usually hire the project owners after incorporating a new project.
fiction becomes reality? Have a look at http://merobase.com/ for example. You can start to recycle code right now.
You've just explained the major economic reason for the general lack of code reuse over the last couple of decades.
Deleted
Actually, i think Ariane 5 screwed up because of a conversion of a 64-bit floating point to a 16 bit integer. I believe that this actually has more to do with lack of sufficient testing (particularly on boundary values), rather than the use of 'glue code'
But in a different sort of way - It isnt necessary to actually reuse the code the way he describes. He simply have to look at software engineering ideas of patterns. Patterns ( http://www.amazon.com/Pattern-Oriented-Architectur e-Patterns-Concurrent-Networked/dp/0471606952 ) Have entered the main stream and are finnaly being taught in many undergrad programs - taught and used in graduate programs, and a serious way in which distributed object computing can be done. Patterns provide a tested and verified method for implementing common paradigms of programming, yet can be easily contextualized.
This is a wonderful idea that I've been hearing variations of for many, many years. I don't think it any more likely for software than I do for books, however. It is conceivable that someday authors would create a "new" book by selecting sentences, paragraphs and maybe even chapters from existing books and just adding a bit of "glue" to connect them. In fact, this has probably already been done. But was it any good? I doubt it.
As for object re-use, I can never forget one of my programming instructors saying, "Don't reinvent the wheel." But despite using that phrase all the time, he was never able to explain why we have so many different types of wheels, with more being added all the time. How can anyone truly believe that something as complex as software will ever be so perfectly encapsulated? Heck, we aren't even doing it with simple objects like houses or cars, though, maybe, someday, there will only be one standard set of car (or house) parts that are used. Total uniformity would certainly ease maintenance, although the aethstetic aspects may suffer.
For a computer example, how about something fairly fundamental, like a vehicle guidance system... Changing a few parameters and adding a little glue code should be the most that is needed for the softwear to drive a car, fly a plane, navigate a submarine, roll a segway, or travel to Mars, right? After all, all of those things deal with the same inputs (mass, gravity, inertia, direction, velocity, etc.) don't they? I expect that if such a chunk of software is written it will be massively outweighed by the "glue" code required to support it on each particular device.
But I'm a pessimist and a Doubting Thomas who probably should have been born in Missouri...
Does anyone code from scratch these days? perhaps some embedded devices developers...the rest of us use the APIs provided by the operating system and programming language we use, or use alternatives found at code repositories.
I can't think of a single project I've worked on that didn't include at least one major piece of FOSS.
Retired from software... maybe. Sort of.
Giving a new meaning to trash-bin, eh?
There you are, staring at me again.
#include
void main()
{
printf("What? It isn't already happening?\n");
exit(0);
}
If all you have is a hammer, everything looks like a thumb.
Another SF writer, http://www.amazon.com/Newtons-Wake-Space-Ken-MacLe od/dp/B000C4SFRE/sr=8-6/qid=1162740889/ref=pd_bbs_ 6/104-0751852-8763166?ie=UTF8&s=books/Ken McLeod, has had ideas similar to this in his books. In the last one I read, Newtons Wake, the characters often have to engage in combat archaeology, which mainly involves the aggressive hacking of ancient technology. He introduces the concepts with a sense of humour -- in one segment of the book, the characters' electronics are attacked by a virus beamed from a an advanced technological artifact. They are able to tell the virus is based on human-created code because once they go deep enough with their combat archaeology, it's east to spot the Microsoft patches in the virus code.
Reuse of sotware (as suggested by the original posting) is (in my opinion) greatly hampered by the use of C/C++ and systems that are not modular based. By "module" I mean what Niklaus Wirth created, first with Modula and later with Oberon and Component Pascal. Only by having strick information hiding (which is the default in Oberon) can one hope to be able to paste and glue (reuse) software. The C/C++ default is the export everything. It is a white box. The Oberon default is to export nothing. It is a black box. Blackboxes are much easier and safer to use.
Actually, the worst software engineer I ever worked with had a PhD in computer science. It tells you nothing about someones understanding about software engineering. For all practical purposes the two are different fields.
There are two different fields within CS, is probably a better way of looking at it. I don't know what Vinge's exact area of study is, but given his writing I'd guess he probably focusses on computational theory and/or AI. This means he's likely to have a very solid foundation in terms of software engineering. This guess fits him in a category along with Donald Knuth.
Other computer scientists study stuff like language semantics, concurrent architectures, mathematical approaches, stuff like that. I'd guess the guy you worked with is one of those.
There really isn't a good class of programmers in academia. Computer theory and AI people spend all their programming time developing small apps in Scheme and Java (look at recent AI literature if you don't believe this). These apps tend to demonstrate simple concepts and are usually simple enough that they were written from the ground up with minimal reuse.
As a former professional software engineer currently finishing up a CS PhD , I can say from experience that people with a CS PhD are the least qualified to discuss software engineering. They generally have no experience working on projects that require more than one person and learn architecture from reading rather than doing (also a probem in industry). On the other hand, experience developing real software makes getting a PhD pretty easy. Most CS PhD dissertations are really just medium sized software projects with a few performace studies. If you already know how to write software, you don't have to spend the extra time 'discovering' software engineering while you're working on your disseration.
-Anonymous PhD Candiate
...application programming will consist mostly of using collections (lets call them "libraries") of code other people have written in the past, along with constructing bits of glue code to connect them together.
Er, isn't that what is done now? I mean, sure, there are some situations where you can't rely on either actual libraries or the equivalent in the built in functionality of an HLL (like writing certain basic code either to support new hardware or to tweak existing low-level code for better performance or extended functionality), but its not like most programming today is "programming from scratch".
There's always, I think, going to be an irreducible amount of programming from scratch as long as new hardware is created which presents a new interface to software, but, yes, using code that other people have written avoid having to reinvent the wheel is important now, and will probably be increasingly important as there are more and more libraries of code out there that are available to use.
Seems to me that, while there is some colorful language in the description, there's not much new predicted here.
They didn't build it from scratch. They used all sorts of screws and nails that had been engineered by other people.
This is just a natural progression. A few years ago we stopped having to write quicksort routines because they were already done, now we dont have to write webservers or rich text editors because they are done for us.
The end result will be that it becomes cheaper to make complex applications and so the customer will demand even more complex ones.
There's tons of Open Source out there that you /could/ use, and there's tons of commercial libraries out there after your dollars.
/I/ think is worth the money, let alone can sell to a PHB. ... and for Open Source, you need your employer to OK its use. Now this may be OK in smaller companies, but in the large companies I've worked for it's a hurdle. Mainstream stuff like OpenSSL, Perl, Hibernate etc. are fine (because someone else has gone through the process before you), but anything else has to go through a heavyweight approvals process that's often more work than writing the damn thing yourself.
But for commercial software, you've got to convince your PHB that it's worth the money (and I rarely see a commercial library that
There are two main reasons for this:
- people are concerned about using code of unknown quality
- people are absolutely terrified of IP law: accidentally incorporating GPL code in something they don't want to GPL; relying on software that apparently has an acceptable licence only for it to turn out to be someone else's original work.
That, to me, is the big barrier to re-use on a massive scale.
Thanks! I have long been wondering whether SOA is a joke I am not clever enough to get, or whether it is a technical solution I am not clever enough to understand. That site made it all a lot clearer.
I always tend to create the technology behind companies I start, from scratch; but using libraries which are very rich, and becoming better all the time. Modularization and improved libraries, tends to blur the lines of what "from scratch" means. Even a raw C program that uses the standard C library, isn't technically "from scratch," since it leverages the work of others in the libs.
Love many, trust a few, do harm to none.
It is a great idea to take different components and glue them together using languages like python, perl, (or for the web) php. However once those components are glued together they only fit one specific purpose. No one can take advantage of all of your glue-work, which may be a large project in itself.
We need better ways to glue and then encapsulate. Take a number of components possibly of different languages, or even possibly architectures, glue them together, wrap them up in a single shared object binary, and expose only the needed portions.
With a language like python you can get 90% there, but its reliance on the python runtime is a major bummer. Compiled python may be a valid choice, but I haven't seen it used too much in practice.
Fast Federal Court and I.T.C. updates
Not "code re-use", re-re-re-discovered again?
One day a group of scientists got together and decided that man had come a long way and no longer needed God. So they picked one scientist to go and tell Him that they were done with Him. The scientist walked up to God and said, "God, we've decided that we no longer need you. We're to the point that we can clone people and do many miraculous things, so why don't you just go on and get lost." God listened very patiently and kindly to the man and after the scientist was done talking, God said, "Very well, how about this, let's say we have a man making contest." To which the scientist replied, "OK, great!" But God added, "Now, we're going to do this just like I did back in the old days with Adam." The scientist said, "Sure, no problem" and bent down and grabbed himself a handful of dirt. God just looked at him and said no..no...no , You Get Your Own Dirt!
There is nothing new under the sun. Everything is just a rehash of Ada Loveless' work.
Program in Perl and pattern matching comes built-in. Program in Python and you get dictionaries -- key-value pairs. Delphi gives you its VCL. Program in Java and you get instant memory management.
Making a robust module that people are going to want to reuse is hard, and the people developing OS's and computer languages and associated frameworks are willing to do that hard work. Letting a thousand flowers bloom of a miriad of 3rd party suppliers hasn't developed to the same extent, although there was some hope that component frameworks like ActiveX, .NET, or Java would make that possible.
The idea was that the right framework would facilitate reuse -- in practice, the right framework is in itself the reuse because it has the features you need built in.
You're quoting the British version of the show. The American version said "You don't MAKE friends with salad." Just Google it you stupid f|_|ck:
You don't make friends with salad
First, that's what libraries are all about. But then, that's also why compilers have become gigantic they're trying to write every possible function that will ever be needed into the libraries.
Of course, companies have their own functions, but almost *never* have a company librarian, or standards that hand you, as a programmer, these functions - it's always "go see if it's out there, if anyone knows".
But then code, properly written, is easily usable like this. The ultimate example is, of course, *Nix (grep blah filename | awk 'NR > 5 && NF > 0 && ! ($1 in myarray) { myarray[$1] = $2;}END{ for (i in myarray) { print i, myarray[1]}' | sort |etc |etc.
mark
IIRC, the problem was something like "my ship might end up in a zone where only a 386 would work", so software archeology was one way to write code for archaic systems that would be poorly understood by someone used to computers thousands of years advanced from that. This is NOT only science fiction. Imagine trying to get a room full of Apple IIs, Ataris, 286s, C-64s, and Amigas all up and running. Anyone remember all the custom autoexec.bat and config.sys files needed to get certain games to run under DOS? BTW, One of the coolest things in Vinge's book was the galaxy-wide network of Usenet servers :)
I'm the source of the original story. (Thanks, Clif, for choosing my post!) What I submitted was much longer and what I was really interested in unintended resuse of Open Source code and the emergence of tools to support such practices. You can see the unabridged version on my lab's web site.
I'd like to respond to a few of the comments.
I'm not trying to float an idea, I'm just making some observations of what people are currently doing.
Yes, reuse is old. Reuse is sometimes bad and sometimes good.
But what is new, I think, is the wholesale use Open Source systems as components, and the way in which people trawl the web for source code. We've had libraries, DLLs, Perl modules, classes, and the like for ages, but these were designed to be reused. But these Open Source systems were built as stand-alone systems, that people are turning into components ad hoc. When people start designing a system, they start with the idea, see what's available out there that they can reuse, and continue the design process from there. I was trying to ask some speculative questions about what software tools are needed in these situations.
In particular, there has been a proliferation of search engines for code on the web, such as koders.com, krugle.com, Google Code Search, and Sourcerer, here at UCI. (I'm just a university researcher and not affiliated with these sites in any way.) I have a suspicion that people need features other than the ones built into these tools and that people are being very creative with existing web resources to locate the code that they want. I am a conducting a web survey to find out whether this is the case.
I had been told that Vinge's "Deepness in the Sky," contains more detail about zones of thought and complexity, and it's sitting on my shelf. But I didn't want to insert a gratuitous reference to a source that I have not read. Look elsewhere on this topic for a more detailed discussion of Vinge's ideas.
The comments have been really interesting. Some went in directions that I didn't anticipate and others provided insights that were new to me.
You have at least a few different levels of abstraction in the industry (you could probably think of more if you tried): applied physicists who understand transistors and atoms and all that stuff, hardware people who understand NAND gates and RTL, Systems people who understand Linux and kernel modules, applications developers who understand software engineering (hopefully), and scripting/web people who understand integrating programs.
I hate to say this, but people who are further up the hierarchy tend to look down on those below. Physicists tend to think that hardware is just a kludge. Hardware people tend to think that software is easy. And everyone thinks web development and scripting is easy (except the people doing it).
The people further down on the hierarchy can usually be more laid-back and less mathematically oriented. Ideas from academia also tend to be more relevant to people further up the hierarchy, and less relevant to those below.
None of these levels are really "fixed"-- new advances are being made all the time by the physicists, and it ripples down the chain.
This is bad news for people who want things to stay the same, but good news for people who enjoy action and adventure (and being considered obsolete old farts at 35). Also, look for more layers of abstraction to emerge in the future.
"No more coding from scratch" would only really be true if the operating system and software levels froze up, and new development ceased. But as long as those crazy physicists keep pushing more transistors on to a chip, it's a good bet that today's software will look like a bottle rocket compared to the Space Shuttle of tomorrow. Hold on to your seats, folks, and don't sell that MSFT or GOOG just yet.
"Any connection between your reality and mine is purely coincidental." -Slashdot
Maybe one day geeks, nerds (or whatever you want to call them or call themselves) will get this in the part of the brain that is necessary: programming is not art.
There, I don't care if it hurts. I have made my good deal of programming, saved the world once or twice, was very satisfied about it, and then moved on. I am also an artist on my free time, play piano to good standards, compose a bit, write novels, etc. So I think I know the differences.
Programming is an engineering-mathematical activity that can be analyzed formally in terms of mathematical equations and theorems and that can (and should) be sujected to constraints, costs and project planning for it to be of any general use. This is more so in the context of the original posting.
Now, if writing a perl script makes you feel all artsy-farsy, all the power to you, but the pleasure derived from it is not aesthtic in nature and will not convey any aesthetic reaction from anybody normal.
IANAL but write like a drunk one.
Many composers do exactly what you are saying.
Beethoven "Prometheus" theme is heard in many of his compositions, as is its theme from the 5th Simphony.
As a matter of fact, Wagner advocates this approach (leif motifs) as part of composing an opera.
And there are uncountable examples of composers that borrowed material from others to make their own compositions (the lazy bastards).
IANAL but write like a drunk one.
The subject of the ICFP 2006 Programming Contest was about exploring a dead system of an old civilization. The challenge is still available. Try it! It was really fun.