Slashdot Mirror
No More Coding From Scratch?
Susan Elliott Sim asks: "In the science fiction novel, 'A Deepness in the Sky,' Vernor Vinge described a future where software is created by 'programmer archaeologists' who search archives for existing pieces of code, contextualize them, and combine them into new applications. So much complexity and automation has been built into code that it is simply infeasible to build from scratch. While this seems like the ultimate code reuse fantasy (or nightmare), we think it's starting to happen with the availability of Open Source software. We have observed a number of projects where software development is driven by the identification, selection, and combination of working software systems. More often than not, these constituent parts are Open Source software systems and typically not designed to be used as components. These parts are then made to interoperate through wrappers and glue code. We think this trend is a harbinger of things to come. What do you think? How prevalent is this approach to new software development? What do software developers think about their projects being used in such a way?"
Code recycling, and that word was chosen specifically for this case, is a time-honored and proven concept. One just has to look at Microsoft's products to see that.
If you think debugging is a pain NOW...
..."Objects"! Perhaps have a framework to enable the interconnections of these objects, perhaps entitled the Common Object Model. I think you're on to something! Quick - base an operating system around it! It would probably rapidly become the worlds most popular one!
I predict that this will happen shortly after all composers simply re-arrange themes and phrases from previous musical pieces in order to create any new compositions and authors simply splice paragraphs from existing books to form new ones.
It's always a long day... 86400 doesn't fit into a short.
More often than not, these constituent parts are Open Source software systems and typically not designed to be used as components. These parts are then made to interoperate through wrappers and glue code.
Isn't that the point of Free software? That it is flexible, that we have the source code so we *can* make it do just what we want, rather than be limited by the original authors?
What's the point of coding from scratch if you don't have to?
Microsoft is to software what Budweiser is to beer.
Machine language maybe. But if you are using the for loops, arrays, stacks, conditions, or whatever of your high-level language, you are already reusing code at that point. We just keep taking the theft to higher and higher levels. All good art starts with theft.
I certainly do not want to be developing the same common hardware implementation for a FPGA over and over and over again. Give me something the code for something that works and let me get to the stuff that is unique to the project.
Ooo man the floppy drive is broken. No wait. The computer is just upside down.
Love sees no species.
code reuse works best in languages like lisp and not so much in C/C++ family? I'm saying this mostly because in lisp I can drop down to almost any section of code instantly (most of the time, not always) while C/C++ IDEs tracking down a bug on large codebases can be a day long job frequently.
In the other languages, I often feel like treading in a minefield when reusing anything less than proven APIs or functions, let alone entire codebases.
If a piece of open source software is used as a component often enough, people will turn it into a component (through an API, plugins, software packages, modules etc). This improves the overall design of the software and allows better code reuse, but more importantly reduces maintainance issues by allowing developers to upgrade components with relative ease. I see this as a "Good Thing(tm)".
This sig is intentionally left blank
Nice piece of science fiction but the reality is there are orders of magnitude more things that can be done writing new pieces of code from scratch than just combining existing ones. People paying for the software tend to make sometimes ridiculous demands on customisation of everything from look and feel to algorithms for doing the actual work for everything from transaction processing, science or graphics. Therefore new snippets will continually need to be coded.
What we're talking about here is the reuse of libraries. It's actually good practice to find a library and use a tested piece of code if you're doing something, than to go out and re-write it from scratch. My experience is that most developers are bad at searching for and reusing code from their own projects let alone from some vast archive (aka the Internet). The tools and processes in place for finding code that you can trust and that does what you want are so poor that at present we're far from even this level of reuse. Hell coders keep re-writing string handling routines that exist aplenty. But where do I find test cases and results for the existing libraries that I can examine. The quality of the libraries varies widely by language and area of functionality. If I trust the libraries I end up writing code that doesn't work and cop flak for it.
For example nothing wrong with Java String handling, but have you ever tried to use the standard date parsing routines. Hint: Even with lenient=false it does try to guess what you've meant to enter if you enter something that's close to being a real date. If you want to reject incorrect input, you've got to write it yourself. This isn't an excercise in Java bashing - every language has areas of weakness in the libraries - but it does explain why programmers often prefer to write it themselves. Even if they do decide to reuse, if they're not careful to retest the algorithms, they are taking big risks in terms of getting working functionality at the end of the day.
These posts express my own personal views, not those of my employer
I'm pretty sure that the novel of Vernor Ving that talks about this issue is "A Fire Upon the Deep" and not the "Deepness in the Sky". The later is a prequel of the former. "A Fire Upon the Deep" was also one of the first science fiction novels to portrait Usenet.
The promise of SOA says that you won't have to do this. If you believe in that promise then anyone that develops projects in the future will create them in discrete elements that are accessible as a web service. If you want to start a new development project, just utilize those services that you need and ignore the ones you don't. Because the functionality is encapsulated (and therefore, written, debugged and tested) within the service you're good to go.
I see application projects in the future acting like glue that holds many services like this together and makes them more than the sum of their parts.
I'm a big tall mofo.
...though basically well written, is full of dumb ideas. This particular dumb idea comes from the notion that the development tools we use now are pretty much as sophisticated as they're ever going to get. That's right up there with the famous urban legend about the proposal to close the patent office.
Vinge's problem is that he makes too much of the famous failures of AI, and has fallen in with the camp that believes that software will never be able to compete with wetware. That has yet to be proven, but even if its true, simply replicating human intelligence (so-called "hard AI") is not the only strategy.
As the developer of freely available software, I find the prospect of people using my code a mixed bag. Partially, I feel an ownership of the code I write and am somewhat offended by the idea of people using it . However, as a paid engineer, I go through this at regular intervals; older projects get handed to others for support, as I work on new components.
On the other hand, I welcome the idea that my free code would be used by others - it is a flattering prospect, I suppose.
Others profit from this sort of re-use: COM, CORBA, Jars, etc...
A key barrier to code reuse is that technology marches on. Many advances render older code either obsolete or, at best, cumbersome. Recent examples might, for example, include adding templates to .NET, which rendered vast tracts of our .NET 1.1 internal class libraries obsolete.
Never, ever lose a file again. Ever.
Isn't this why we have layers of abstraction, such as high-level programming languages and external libraries?
Sure, if someone who only knew assembler from 20 years ago saw the GUIs present today, they might agree with the article. Imagine if you had to write a Windows application in pure assembler. It would be ridiculous! But we have languages like C# or the Windows API for C/C++ to aid with these kinds of things.
As we progress through time, programmers will simply use higher and higher level languages.
Unless we move out of 64-bit architecture, right? I mean, we'll have to by Sunday, December 4, 292,277,026,596 or else we'll have something slightly similiar to Y2K (or Y2038). But really, aren't a lot of virus' and spybots program this way? I wouldn't know; the only thing I could do related to reprogramming a virus or spybot would be renaming the exe file that the dumb@$$ has to open to get his system infected.
Ginga no Rekshiya Mata Each page.
Few people write their own C compilers, libc, math libraries, HTTP protocol handlers and so on. If other/bigger components are available, of course they will be reused.
I've been working on a MediaWiki modification that details every portion of the function/object/application creation process. The idea was that I could put a function or object together using pre-existing chunks in the wiki and then the end result will automatically be fully commented and based on code that has already been tested and proven.
No more buffer overflow exploits, no more null pointer dereferences, and no more need for 30+ libraries that have duplicate or similar code in them... or so the thought goes.
So far, I have a few entries targetted for the 6502 processor and the script that generates the end result code is mostly working... sort of. heh. Eventually, I'll fill it out with more useful stuff and then make it public, but until then, it's just a hobby to pass the time.
various "experts" have been talking this nonsense for years. it's never going to happen.
If you mod me down, I will become more powerful than you can imagine....
Idea about using decades old logic doesn't work. Someone has to mantain all that code: kernel, basic libraries, platform API's etc.
Problem is, software gets old, obsoleted by new ideas and when next-gen of hardware takes over, so updates are needed all the time. If you don't pay attention, you can end up with cruft like X Windows protocol which people are now trying to fix and improve to compete with other popular OS's.
Unlike in this novel, actual (best) code search won't need digging, it will be advertised, categorised, documented, just ready to be used.
Some basic tasks like file io or token processing and such minor things might be reused. But even then porting something so simple like a string tokenizer written for a full fledged virtual memory OS like Unix/WinXP to a fixed memory handheld device is highly non trivial, especially if you want to handle multi-byte i18n char streams
If the author sells what he was smoking while coming up with the article, he stands to make tons of money.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
Your code will be assimilated.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
Because he has some cool ideas. Sure, they might have zero practicality or association with the real world...but they're still cool. ;)
The article is a bit of nonsense really. It ignores the fact that building software is a vertical problem anyway where most often you pick the highest level tool to get the job done. You now have
Transistors
Digital Logic
Machine Code
Compilers
C Code Family
Dynamic Languages / Visual Languages
What next...
In 20 years time nobody will be pissing around with C code or Java or or Lisp ( ok maybe lisp) except for historical/maintaince reasons. There will be new higher level constructs leveraging streamlined minimal lower level constructs. Many of the problems defined today by large code bases will be rewritten using less effort and more sophisticated expressive tools. In 20 years time there will be 2^20 cores on a chip, perhapps. I doubt code bits from today will solve those sorts of problems.
--
My slashdot captcha was "pervert" huh!
The bikini - security through obscurity since 1943
This new concept fascinates me. We can take these individual blocks of code. Let's call them "libraries", and join them together with some kind of programmatic glue and potentially reuse them to make some kind of application. This is definitely cutting edge thought. I'm not sure it will catch on though.
This is a sig
Unless you're working in R&D or making things that don't exist (a lot of game engines that are from scratch, a lot of open source projects, new protocole implementation, etc), thats probably what you do 40 hours a week. I know a lot of people for example who have to work with web services, and for a reason or another have to do contract first (because they do things their toolkit, if they even use one, doesn't handle), they often copy paste a WSDL template and modify it. A lot of database management tools have a librairy of template stored procedures. Googling for code snippets and samples is always good...
Especialy with environments like Java or
I was in a training class for Rational Software Architecture, I was 'told' that some company's have set it up and augmented the rational suite so that they can get about 85% of generated code before a developer has to touch it. There claim was the architect will be the main driver behind future development and the key cog in the enterprise.
isnt code reuse the cause of that major flaw?
no one questioned the code for years
back in the day we didnt have no old school
No, this is common practice, because it often just makes sense. Maintaining a set of common components means that when you do reuse a piece, you can maintain it once, and release the updated component to all of the affected software. So if you have 30 different customer applications that all use the same iText based report generators, you only have to fix a bug in one place, or add a new feature (like a new type of chart) in one place, and you can release the update for all 30 applications (and satisfy your 30 maintenance contracts). I have a set of my own components and FOSS components (like iText), that I reuse for various common features, and I'm sure the big companies (CA for example) do the same.
--That's the point of being root, you can do anything you want, even if it's stupid.
What?!? Why is this news? We have libraries - those are the big chunks of useful code - packaged and (hopefully) documented for re-use and widely distributed.
When you design new applications you look for libraries that do the bulk of the work - and the application becomes mostly 'glue' to hold the libraries together. Scripting languages are the epitome of this - where large piles of carefully written and optimised libraries make up for the overhead of interpreting the actual library code.
Dunno why anyone finds this surprising - it's what we've been doing almost since the dawn of programming.
The tricky part is noticing when you can't find a suitable library - so rather than dumping a bunch of code into the application, design it as a reusable library. There is a small overhead to doing this - but rarely more than (say) 5% of the development cost. So when writing OpenSourced code this is generally done well. The hard part is in commercial code where you have someone with a chequebook and a severe lack of foresight breathing down your neck who would rather save 5% now than save 30% on the next project.
www.sjbaker.org
"Welcome to the future of unmaintainable garbage software."
That's a strong statement. I disagree - I think a strong component library will actually improve maintainability of software by an incredible amount. Software Engineering is the last major engineering field to realize a useful and standardized component model. Electronics would be nowhere without IC's. Mechanical engineering would be nowhere if people had to design motors and engines seperately for each application.
All this increases maintanilbity by decreasing debugging time when something goes wrong.
The Neverwinter Nights 2 Toolset was redone in .NET, but I checked the various components, and it at least used these:
- RAD Game Tools' Bink library
- RAD Game Tools' Granny library
- RAD Game Tools' Miles Sound System
- Crownwood Software's DotNetMagic 2005
- D. Turner, R. Wilhelm, W. Lemberg -- FreeType font rasterizer
- Glacial Components Software's GlacialTreeList
- Mike Krueger's (of ICSharpCode) SharpZLib library
- Divelements Limited's SandBar
- Various libraries done by Sano
- Various libraries done by Quantum Whale
- Davide Icardi's (of DevAge.com) Source / SourceGrid libraries
- Matthew Hall's XPExplorerBar
- Zlib
These were third party work used in the toolset application, and could be more as well (just checked standalone libraries, not any statically linked stuff).
Beware: In C++, your friends can see your privates!
I was speaking to someone at work last week and this exact topic came up. We have 5000+ batch jobs written in JCL and whenever someone needs to write a new program, they just mine through what's already out there and use it for the new stuff.
...
It's not that uncommon at my workplace
With companies like EA forcing insanely short deadlines to get Generic Sports Game 2007 out by the holidays, it pretty much forces their code slaves to reuse mostly everything in its' horrific state. If end product quality matters then "from scratch" programming will have a larger priority.
----------
Check out my blackbox styles
The notion of complete code-reuse and never coding anything from scratch is a very good idea in theory. The problem is that in order for it to work the components from which things are built have to be reliable and trustworthy, and such things are not always easy to find.
I've been studying errors and defects in engineering (both computers and otherwise) for many years. I was raised by an accident investigator, so I have an innate understanding of why things fail and what people can do to avoid it. The core reason for anything to fail is the assumption of correctness. In all accidents and failures, one can always find somebody somewhere that made an assumption about a component's behavior. The fact they did not know its true behavior caused them to employ the component when they should not have. Thus, the other elements of the system that depended on its assumed behavior also fail, and it just cascades from there.
In software, such assumptions of behavior often take the form of (to name a few) failing to detect null-pointers, failing to enforce buffer limits, or failing to enforce proper protocol operation. Opening the source to the world can help to detect such assumptions, but someone still has to check. As overkill as it may seem, I've been known to write unit tests for third-party libraries to ensure they behave as their documentation says. Many times I've found them to be faulty. Had I not tested them and just assumed their correctness, my product would not have worked. Even though my code may have been correct and the author of the libraries were at fault for their code not working right, I too would have been in error for making a false assumption.
Until such time as we can trust the components upon which we base our work, we cannot employ 100% code reuse. To reach such a stage, we need a radical shift in our approach to engineering. We need to ensure that our products do as they're supposed to and not just assume. Unfortunately, the rush to market often prevents us from doing what we should, and that's a shame. First to market is not always best to market.
You may not realize it, but as of today, virtually all new applications being developed re-use existing working software systems. For example, when you develop a new web application, the components that you re-use are: the OS, HTTP server, database server, scripting language (PHP...), etc. Of course nobody is re-developing everything from scratch. Very often only the higher level software layer (ie. your application) is developed partially or mostly from scratch, but everything else (ie. the internal or lower level components) are re-used from existing projects.
This has been a useful and common method of software development for a long time - at least as long as I've been programming professional, and we were taught it in school, of course, though we rarely did it, since the point was to learn how to do it ourselves before using other's code. Probably 85% of my new development work is tying components that either I wrote or someone else did into new and interesting configurations. The stuff I use (typically compiled code written by other programmers) was in itself designed from custom libraries of things other programmers wrote, and so on and back. There are pieces of legacy code that date back at least 15 years in our custom libraries. Code reuse makes designing and making useful software faster, more standardized and a lot easier to understand. Sure, sometimes you want to do it yourself and that's good, but really? The chances that your print routine or sort are going to look substantially different from anyone else's are low. Of course, none of that applies to research programming,as far as I can tell.
One of the problems is the necessity for glue code. There needs to standards for software libraries. You can have different libraries, but certain functions should always get you something that will work in the context. One should be able to take out any library from any piece of software, and replace it with another library (assuming there is more than one for the same purpose).
Stupidity is like nuclear power, it can be used for good or evil. And you don't want to get any on you.
While its easy to grab code and quickly put it together to make something that works, it is usually the licences and the difficult to interpret legalese that can limit this approach. This becomes relevant if distribution of the resulting software is required, and is especially relevant in a commercial context. Clean-room implementations are going to be around for a while to come, IMHO.
-- Mike
This has been the PHBs wet-dream since programming began. They see writing programs as assembly. It isn't assembly. It's design. You can't automate design.
They always talk about making generic components that can just be "glued" together to create a functioning application. You can't. That "glue" is the business logic, and it's what your program does. It's what you are paying the developers to do.
If you have good developers (skilled, not monkeys) they will create (or use) libraries to do most of the heavy lifting, but they still need to put those libraries together in a way that solves a business problem. If you could just automatically glue pieces together, how can you provide a useful product for your customers? Sure, you can take an "email system" and a "social networking" system and "glue" them together, but to do what exactly? Solve some business problem perhaps? How will you "glue" them? By developing business logic.
It's like taking two different ideas / products, placing them in a room together, and expecting "synergy" to create something new and great. Even if you have some great idea about how those two things can be combined to create something, you still need someone to do the design work of actually combining them.
You wouldn't expect an air conditioner and a storage locker to just magically combine together to create a modern refrigerator, would you? You need someone smart to integrate them. That is what a developer does.
Sounds great, in theory. But in the real world, most code (other people's code... always other people's) isn't that well modularized and can't be reused without introducing lots of bugs to an application. Every hour saved writing code will cost you three in debugging and re-writing.
We've heard it all before. Next story please.
It's truly disturbing to me how someone who put so much effort into writing a response could have completely failed to understand simple words like "all" and "any".
It's always a long day... 86400 doesn't fit into a short.
http://www.compositiontoday.com/sound_bank/viola/v iola.asp
Look at recently developed programming environments that have made it big: Java and C#, running on a virtual machine, using a large class library.
My first impression of Java was of "a small language with a big class library" - I don't mean that in a bad way, the relatively small number of syntactic elements in the actual language and handing of things like say, threading to classes is a good thing.
But coding from scratch? What's that supposed to mean - typing ones and zeros at a OS-less motherboard? Working with all this support is the present and future of coding. Working without it is becoming a niche - for people who make the VMS, Oss, device drivers and class libraries that the rest of us depend upon. And yes, this trend for richer class libraries will continue. Duh.
My Karma: ran over your Dogma
StrawberryFrog
We're all leveraged when we can build on the good ideas and good code of others. The danger, in some organizations, is when they believe they can hire low-skill or non-programmers to put the pre-designed components together. As an old-timer, I've watched software development over the years, and there is a continual tendancy of enterprise managers to conceptualize programmers as low-cost, commodity labor. One company reorganized their development group out of existence, conceiving of this work to be done by vendors and rare consultants. The result is that they are frequently in a state of chrisis from their systems failing and from having nobody on staff with the skill to diagnose or fix them. Managers need to know that they need highly skilled developers, and they need to pay them what they are worth. These people can intelligently build on the work of others in a way that works for the unique problems of their business.
Makes me nostalgic for the days when I used to start with an idea and create the design from there. But even then, it was almost always the case that everything I built was built on top of, or in the context of, another software system. (well, there was this one time when I wrote my own assembler so I could write my own boot ROM for a machine you've never heard of... but that's not exactly normal)
Am I part of the core demographic for Swedish Fish?
Ever since the first person reused existing code.
This is just another step along the same continuum.
I wrote a small perl program the other day. If you count all the code that executed when I ran it, "my" code was only an itsy-bitsy-teen-weeny part of the whole.
"Move along now, nothing for you to see here."
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Whilst this is an uber fantasy for minimizing cost and development time, it will never exist. You just have to look at the amount of bugs that exist in software, they will never really vanish and if we're not writing bespoke software they are likely to double. The text talks about architecture within the project being what will replace the hard work, however software architects exist only in an a project at the start. It's rare to bring them in at the end and can you imagine the hell you'd go through having to debug some alien code if it's not working? The people who make up architects in my University are just the people who are trying to make money for Computer Science, I pretty much see them as a lower form of a professional. The real Computer Scientists are those who bring new concepts, not those who spend all their time planning. I accept we need people from other disciplines to enhance our working practices but it should never be viewed that these people will take over the core subject. For example we all know Computer Science was started from a bunch of crazy Mathematicians (Just look at Babbage and how he was publically ridiculed because he didn't complete his thinking engine), surely someone could also abstract from that time that we won't eventually need Mathematicians because Computer Scientists will still be needed. But we still have the crazy Mathematicians as they are still required for the absolute core stuff. From those principles let's honestly look at how businesses work, You'd never get such a repository put together of lots of clever code. A 100% bug free module for a specific task would cost more than the moon. (About £8.8x10^22 in Cheese Pounds)
Code re-use is a great idea. Free software is a great idea. Taken together, and to an extreme, they can cause problems, particularly where security is concerned. What happens when someone finds a security flaw? How can you contact the people who are reusing your code if you have no idea who they are?
To take a personal example, my delta compression code, which I wrote for FreeBSD, is now being used by Apple and Mozilla to distribute updates; I've talked to their developers, and if I find a security flaw in this code (very unlikely, considering how simple it is), I'll be able to inform them and make sure they get the fix. On the other hand, I know developers from several Linux distributions have been looking at using my code, but I'm not sure if they're actually using it; and searching on Google reveals dozens of other people who are using (or at very least talking about using) this code.
Putting together software by scavenging code from all over the Internet is like eating out of a garbage dump: Some of what you get is good and some of what you get is bad; but when there's a nation-wide recall of contaminated spinach, you'll have no idea if what you're eating is going to kill you.
Tarsnap: Online backups for the truly paranoid
economics...That is, until the hardware demands recoding, reuse is both financially and programatically a generally sound philosophy. Of course GIGO still applies :)
errr....umm...*whooosh* *whoosh* Is this thing on ?
It won't happen that way until there is a coherent established framework for functional testing of the "building block" code. It is GREAT to reuse code, and I've worked on a few projects that became GPL due to my incorporation of GPL code. However, most professional projects I've worked on require more on the reliablity front so no time (time == $$money$$) is lost. Any reused/mined code incorporated into these projects has had to be so rigorously inspected and tested for function that when combined with the time spent on banging together glue code, we may as well have started from scratch for those code chunks.
The exception to this is tiny code chunks that either have a simple testable API (eg. no functions that return values variable with time), libraries/code that have been so widely used that you can "mostly" assume that it's valid (eg. zlib), or occasionaly some banged together prototype "proof of concept" app.
Until some standard framework is created that the code can be tested for validity by those incorporating the code, it likely won't happen fast (other than for the exceptions listed above).
For the record, I don't know what that standard framework may be. Personally, i'm in the habit of whenever i write a class, i include a static test function in the class that can be called to put it through its paces and check for validity on all functions and expected data alignments. (eg. on some arm processors i've worked on, a struct is packed based on a 4-byte boundary, and sometimes extra "surprise" filler bytes are inserted. Lots of fun to track those down when you're not expecting them.)
Demonstrant's Open Source Tools
The problem with this concept is deceptively simple:
:)
1. The most difficult part of programming is developing software that does a good job of unanticipated uses
2. Most programmers are fine with something that just does the job they need it to do, and no more
3. Relatively few useful software components can operate in isolation, the rest place requirements on the environment
The consequence of this is that contrary to the theory that writing something once should be sufficient, the truth of the matter is that in the vast majority of practical cases, it is simpler to write the code once again, bespoke for your particular needs and environment, than it is to pick up the best available pre-written component to do it.
The essence of the problem is one of assumptions. Any component has an environment, this is the sum of state and information it requires in order to be able to operate. The component also (probably), has an API. This is the formal environment, the sum o state and information that it makes explicitly available to the programmer.
The problem that you have is that there is no "right" place for the percentage of the environment to be formalised into an API. For all those of you who are thinking "Well, the answer is obvious, everything!", imagine a component that manipulates dates.
Think about this for a moment, I use it because it's an example which almost every language I have encountered has done "poorly" (where poorly is the situation in which I find myself implementing something I felt should have been there by default, like adding or subtracting a month).
The environment for a date component, the true environment, is larger than any date component I have ever seen has exposed. For a start, which calendar are you working in? what timezone? what are the DST considerations for that timezone? is there some mechanism for updating these DST settings for those crazy timezones that change DST times all the time? is the begining of the week Sunday or Monday?, what is the current locale? If I add a month to the 31st of October, is it now the 30th of November, or the 1st of December? what if I subtract a month? is month addition associative?
Keep in mind that every question there has been solved at some point or another, and the majority of it is solved by the few really good date libraries out there (perl has a particularly fantastic one the name of which escapes me), but I've never seen one that did a good job of updating its daylight savings information, the majority of the *nix style ones simply assume that the operating system has good information, normally the catalog that came with your most recently install of glibc.
Anyway, back to the point, the issue is simply one of appropriateness. There are some components that make a lot of assumptions, but their interface is simple. There are others that make very few assumptions, but working with them is mind bogglingly complicated. Anyone who has ever had to work with mapserver or equivalent mapping problems will understand the gradient. But it is not at all uncommon to find yourself in a situation, even with something as well covered as dates, where the conventional components don't cover the part of the environment you need covered. Interfaces are either too big and complex, too small and simple, or just focused in a different area (how many components have been dropped because they were perfect, except that they didn't handle multithreading?).
In the end, rewriting this stuff is going to be a fact of life for a long time yet (unless we get really smart AIs or a revolution in programming I can't foresee), although hopefully better coverage of more common problems will continue to be a feature (it's much easier to do things today than it was 10 years ago basically because of this, regardless of what the lisp guys say
You can't win a fight.
and it's called CPAN.
"For a successful technology, reality must take precedence over public relations, for Nature cannot be fooled"
I think that we are not going to be "mining" for code so much as using smarter and smarter tools to develop the code in the first place. Historically, we used to code by flipping bits on the bare metal. Later we developed smarter tools called assemblers and loaders to flip the bits for us. Then we developed "high" level languages like Fortran, Pascal and C, and developed tools to compile our programs. Our tools became smarter. We don't write the code for GUIs anymore. We drag and drop widgets in a design software, click a button, and reams of code get written for us. Object oriented programming could take us farther still, as the GUI elements could already be pre-defined. (Take a card game for example. All the methods for manipulating and displaying cards, and stacks of cards, etc, are already all in place All the programmer has to do is code the rules of the game.)
As we progress in abstracting the data we deal with, we remove ourselves from raw coding. Once we start abstracting and manipulating programs as data, we will enter a realm where we will be able to do stuff like:
Programmer:"Give me a klondike solitaire program, but extend it to a crown and anchor deck".
Computer:Klondike Solitaire found.
Computer: Unknown term 'crown and anchor deck' in context Klondike Solitaire.
Computer: Unknown term 'crown and anchor deck' in context card games.
Computer: There is a dice game with crown and anchor symbols.
Programmer: "Create card deck variant crown and anchor deck by adding a suit with symbol crown, and by adding a suit with symbol anchor."
Computer: Card deck variant crown and anchor deck created. Shall I make this card deck variant publicly available?
Programmer: "Yes"
Computer: Klondike Solitaire Crown and Anchor variant ready.
When our name is on the back of your car, we're behind you all the way!
Often it is desirable to reuse someone else's library. But often it is still very valuable to provide the code yourself, to prevent another dependency on someone else's code, and also avoid something that might limit portability.
I think it is the same cost/performance threshold that people use in everyday life, in making a decision to purchase or "do it yourself." Is using a third-party library worth the hassle of integrating and packaging it?
For example, one developer wanted to add a default dependency on the enormous Boost C++ library to our project in order to provide a chart-drawing ability. Was it worth it to us to add such a burden? No. Would it be beneficial to an app whose purpose is to draw charts? Probably.
We will add your binary and technological distinctiveness to our own. --the Borg
We're all hypocrites. We all have hidden parts, it's the contrast between them that make us more a hypocrite than others
due to massive code reuse we now suffer from the "mindrot" disease that disables us from re-inventing anything, even after we have long forgotten it.
That's why there are many times more Microsoft developers than there are for any other platform. MS has made programming insanely simple. DLL's and other forms of COM objects make code reuse very, very simple. I can make a application with a functional web browser embedded in it in under 30 seconds. Can't do that on any other platform that I'm aware of.
That's what is currently happening....for instance, take VB. Each line of code you write isn't an individual assembly instruction--it's basically a prewritten piece of code....
All this guy is doing is taking it to a higher level. It sounds exciting, but is really obvious :-P.
Ride the skies
We already have reusable components, they're called libraries and they keep getting better. But to think that we can ever easily take some piece of software that isn't written to act like a library and reuse it for something else is pure fantasy. If you have a library with a design specification, you have a bug. If you have a piece of software, it might work quite fine until suddenly somewhere in a mangled mess of code you have some implicit assumption somewhere (which may have been perfectly reasonable) that you just broke. You can't retroactively redefine the code to adhere to a specification, which you obvioulsy need to use it as a component. Sure, you can build some test cases and pray really hard that this black box won't fail on you but that is rarely satisfactory. Or maybe you have so stringent control on your code you can drop it in anywhere, but in that case you'd be the first such application I see.
Live today, because you never know what tomorrow brings
I, for one, was very pleased to see, among other things, the use of Unix time in A Deepness in the Sky.
The friend who'd lent me the book was also amused when I explained it to him... (not every SF lover is a computer geek)
Ignore this signature. By order.
In certain situations this is useful and is already being used to an extent. But rabid use of it is generally bad. Remindes me of hammer factory factory
What about web services. Aren't they pretty cross-platform and language independent?
Isn't that where we want to be eventually?
Arguably better than 'Walla'. Yes, I've seen it spelled that way.
Lump lingered last in line for brains, and the ones she got were sorta rotten and insane.
More often than not, these constituent parts are Open Source software systems and typically not designed to be used as components. These parts are then made to interoperate through wrappers and glue code. We think this trend is a harbinger of things to come. What do you think?
It is a harbinger all right. It is a harbinger of loads of work to come, straightening out that rotten stinking spaghetti. Hint: all you need to do to establish your reputation as a top open source coder is to dig into one of those piles of poo and fix it. Oh, and don't bother asking anyone for permission, just do it. That is the way it works.
Why not start with anything a Ximian coder has touched, since all that code needs to be forked anyway now that Ximian has jumped into bed with the enemy. Suggestion: throw away the bonobo and orbit interfaces, they are just bloatware. Reroll using dbus. Throw away Gnome VFS. Reroll using our real VFS. Lather, rinse, repeat.
Have you got your LWN subscription yet?
That's where having a global database of excellent code comes in.
Software engineering should be a much more tightly regulated. Right now, anybody with a C++ for Dummies can call themself a software engineer. Software engineers should be as regulated as engineers in other disciplines. There should be best practices, peer reviews, etc.
When our name is on the back of your car, we're behind you all the way!
The way I learned how to program in pretty much any language was by looking at other people's code, seeing how they did things, and duplicating those methods in my own. Eventually, I get good enough at a thing that I can come up with my own ideas, or that I remember how things are done without having to go look.
Libraries
This just illustrates how little people understand software development. If blind managers were designing houses I'm sure we would end up with the boss saying "here is a functional bathroom - can you patch it to this functional kitchen?". Well maybe yes - and the toilet would be under the table.
Finding analogies is difficult.
Why not walk into a roomfull of stories written by a creative writing class and try to cob them together into a best seller?
Many years ago IBM was touting the line that with the new faster hardware companies could get away with cheaper programmers. The latest incarnation of that idea is outsourcing to 3rd world countries. Now I'm not trying to run down programmers who are born and live in the 3rd world. What I am saying is that trying to cut costs by gutting programming talent is foolishness and it illustrates a complete lack of understanding on the part of people who coger up such notions... regardless of what guise they do it in.
The only thing that comes from patching together gobs of code is a mess... and undocumented and unreliable and unsupportable mess.
The whole idea is just pure bull.
I'm pretty sure that when you collect pieces of code and glue them together from a higher level of abstraction, you have a programming language. I mean, a simple addition operator in a high level language is several opcodes in assembly for moving stack pointers and such. What this does demand is that the open source community would make the best impact on the world they could by arranging a very detailed archive of code that would cover a wide range of functions, and all be designed with reuse in mind.
I hate to be so blunt, but anyone debating this is an idiot. Code reuse has been with us a long time, and it's the only way that exponentially more complex systems will be built without exponentially larger teams. If you are thinking "but I never cut and paste code" then ask yourself if you ever use a library, for example (standard template library, OpenGL, even stdio.h functions used to be coded by hand 30 years ago). Or if you code in a higher level language where useful functions are wrapped up for you (an entire web browser as an ActiveX control, anyone?). Yes 90% of code is crap. But it is ridiculous to think that we shouldn't build on the 10% of good stuff that is, or can be, neatly encapsulated.
It's a fact.
This is exactly how DNA works. Programmers have so far not had much success at this approach, one because our computers are puny, and two because our programming practices have been tailored for engineered code. But as hardware gets fast enough that most common tasks can be run at a one-millionth slowdown and still run fine, we will get to a point where we can write glue code that runs some other piece of code, throws away 99.9% of its computation, and only uses the rest, simply because of the value of human time vs. computer time. I have written code like this. It was just research projects, not "production systems", but nevertheless it is possible that the future will have less and less coding from scratch.
Some context for people who didn't read the book... or didn't read it carefully enough.
First, Vernor Vinge has a PhD in Computer Science. This obviously doesn't guarantee he can't be wrong, but to those commenters who said something like "these ideas are idiotic"... you've got an uphill battle to convince me that you're that much smarter than Vernor Vinge, especially as most of you saying that don't show me you understood what he was saying in the first place.
Second, A Deepness In The Sky is set in his "zones of thought" universe. In this universe, the fundamental limits of computation vary depending on where you are physically in the galaxy. This is only faintly hinted at in A Deepness In The Sky, it is explicitly spelled out in A Fire Upon the Deep. This limit on computation may or may not be real. One of the effects of this limit on computation is that you can build a system larger than you can really handle, and eventually all such systems come apart in one way or another. This story is set thousands of years in the future and it is explicitly (albeit subtly) pointed out that the software running the ramscoop ships has direct continuity with modern software. (Qeng Ho computers use the UNIX epoch as the most fundamental form of timekeeping; apparently even the relativistic compensation is layered on top of that.) We are at the very, very beginning, where it is still feasible to burn an OS down entirely and start from scratch.... or is that really still feasible? (Perhaps Microsoft will soon find out.)
Those of you posting that "we can always wrap it up in an API or whatever", I'd say two things: First, you get the Law of Leaky Abstractions working against you. The higher up the abstractions you go, the more problems you have. (Look it up if you don't know what that is.) The more sub-systems you make, the larger the state space of the total system becomes (exponentially), and the harder it is to know what's going on. It is entirely plausible that you eventually hit a wall beyond which you can't pass without being smarter, which, per the previous paragraph, you can't be.
In other places in the galaxy, you can be smarter, and Vernor Vinge postulates the existence of stable societies on the order of thousands or tens of thousands of years or beyond, where the society actually outlasts the component species, because the software base that makes up the species does not exceed the ability of the relevant intelligence to understand it.
Both cases (software might exceed intelligence, intelligence might grow with software) are extremely arguable, and I do not think he is advocating either one per se. (Leave that for his Singularity writings.) But you do him a disservice if you think he is not aware of the issues; he's extremely aware of the issues, to the point that he is the reason some of us are aware of the issues.
(Even this is a summary. In isolation, probably the best argument is that it is always possible to create a software system one can not understand or control, but one person can be wise enough to avoid that situation. However, in A Deepness in the Sky Vernor Vinge explicitly talks about how in a societal situation, one can be forced by competitive pressures to over-integrate your system and make it vulnerable. "OK, but the government can be smart enough to realize that's going to happen and step in to stop it." First... smart government? But even if you overcome that objection, now your society faces death-by-surveillance and other automated law enforcement mechanisms, which since they can't be human-intelligent will fail. If you avoid that (and it is a big temptation), then you face the problem of anarchy. And remember that "governance" is anything that governs; even if the "formal government" doesn't regulate you to soceital death, private corporations may do it. Anyhow, upshot, Vernor Vinge has done a lot of thinking on this topic, it shows in his books, it is not showing in the criticisms I've seen posted, and when it gets down to it he really has more questions than answers.)
Hey it was more stable and easier to write for and you didn't need 10+ gig to install it.
Component based programming for Linux turned out very different. Since Linux stuff tends to be developed from the bottom up, starting with a library, then a console app, then a GUI, most "components" ended up just being stuff lower on the stack, like libpng, gzip or whatever. That, and file formats tend to be standardised and read with a library, rather than file access being implemented in any specific application that needs to be loaded every time the file needs to be accessed.
.NET, seems to be more of a developer friendly way of providing and using library components as a developer does in Linux. .NET components remind me of how libraries are used in Linux. IMO this is a cleaner way of working and it's nice that Microsoft are starting to think this way.
On DOS and Windows, applications tended to be all-in-one blobs that use proprietary file formats and only depend on the operating system itself. Thus, OLE made sense. There was OLE-type technologies on Linux (such as Bonobo and KParts) but they never took off in any big way due to the general lack of need. In most cases, they were just used for automation, which better technologies such as DBus now exist.
As for "real code reuse" on Linux... try ldd-ing any application and seeing how many other libraries it uses to do it's bidding. Many application's I've written are just sticking together bits and pieces of other libraries, and that's in C++. I'd go as far as saying that code reuse on Linux is currently better than that in Windows. Generating quick applications with Linux tends to be pretty easy using bash, Python and even C, C++ or C#, where libraries have directly translated into "components" and "modules". The thing on Linux slowing down development isn't having to reinvent the wheel every time you write code, but rather, deciding what libraries you want to depend on and the difficulty of the languages favoured in Linux development, like C.
Since you bring it up, OLE seems to be taking a backseat in Windows now, where ActiveX is rarely used by anything other than IE plugins. Now clipboard/d'n'd/DDE and Microsoft Office are the only times I actively see it being used. COM is still heavily in use by libraries such as DirectX, however. It was never really used to it's full potential, such as to open Photoshop documents in Microsoft's preview application. Windows, with
Hardly anyone programs in assembly. Why not call them "fortrans" because they're translations of a formula for a component?
Unfortunately, this already happens. The standard for a "substantial" portion under US copyright case law is so short that most melodies are likely already taken. Spider Robinson wrote a short story about such a situation.
It is also arguable that some people are afraid to write music due to the fear of being sued for subconscious infringement, losing the case, and possibly having to sell the house to pay the damages. For examples from US copyright case law see Bright Tunes Music v. Harrisongs Music and Three Boys Music v. Michael Bolton.
Same here - when I was learning VS.NET, there were so many common routines that used a simple but annoying-to-type algorithm that were I using a language like C/C++, would just provide more code to put typos in or forget a keyword. Once I unlearned doing things the long way, I would just type in the first few letters of the function I wanted, pick the right suggestion from the list, and watch as one line does something like connect to an SQL server over a network.
I'd say we're at the point where code reuse is not only ubiquitous, but transparent - at least in some languages. There will always be fundamental building block languages like assembler for those who NEED to get down and dirty with the hardware. Personally, I'd like to do more of that, but I can't really justify it since whatever I'm doing, I can afford to sacrifice a little CPU or RAM in exchange for saving 30+ hours of coding.
A lot of this may come from software license incompatibilities. Publishers of proprietary software don't want some copylefted library such as libiberty or readline "infecting" the entire project, which may contain valuable trade secrets or patented processes, with the requirements of the GNU GPL. Even in free software, it's impossible to use GPL and Apache license software together.
Whenever I glued code by a wrapper i shot myself in the foot very efficiently. Wrappers are the last resort. Wrappers are only acceptable if the coder of the code to be wrapped planned his code to be wrapped. If you use a wrapper adding semantics to a component included by the wrapper is a pain in the ass-you have to change three parts instead of two. If needed or if deployment time is near and there is no time to adopt the code in a good way, then a writing a wrapper has to be carried out with a lot of thinking.
My paradigm would be: only use things as components which were meant to be used as components.
Well you have to admit that would be convenient... no need to get up after a big meal.
I don't care if it's 90,000 hectares. That lake was not my doing.
If anyone cares, this sounds an awful lot like a variation on a theme by Poul Anderson, from an early novel of his called "Brainwave". Premise: the earth has been inside a region that physically supresses clarity of thought throughout all of human evolution. The region shifts, and suddenly everyone become orders of magnitude smarter than they have been.
Very true... that's why it is your job as a software engineer to think long and hard about what code libraries you will use as part of your project. Choose well, and your problem is already partially solved.... choose poorly, and your project can be doomed to failure before it even begins.
Every hour saved writing code will cost you three in debugging and re-writing.
If you adopted buggy code, yes. But adopting poor/inappropriate code was your mistake.
I don't care if it's 90,000 hectares. That lake was not my doing.
I don't think we're close to the end of coding from scratch, at least not in the sense that coding from
scratch will be replaced by grabbing existing F/OSS parts and combining them. And the reason is
exactly because
More often than not, these constituent parts are Open Source software systems and typically not designed to be used as components.
Trying to glue things together that weren't meant to be used that way can sometimes be done, but it's rare that it works well and it's usually a major kludge. Until people start building "pieces" that *are* meant to be used as components of a bigger system, then no.
That said, the recent trend towards using coarse grained services / SOA, is a step in that direction. When people start writing apps
that are designed to interoperate as part of an SOA, then I think we'll see a lot more reuse. But right now that's just not
the case with the major open-source apps.
If someone wants to try an experiment, try taking, ah, let's say SugarCRM and SQLLedger; and combine them into a cohesive system. Unless
you're masochistic you'll probably decide it would be easier to write an integrated system from scratch. This is assuming that you start
with a policy of NOT modifying the base code of either system (after all, who wants to be on the hook for maintaining a private fork of something like that, for basically forever?)
// TODO: Insert Cool Sig
Let's be honest here. How many people call themselves programmers and have no idea what a zero flag is or what the difference between Big and Little Endian is? The current level of "being programmer" is to be able to hack together a few lines of code with pre-set functions that pretty much do the work for you. It's actually frightening to see graphics programmers who don't know how to normalize a vector because "there's a function for that".
Fine, of course there is one, and most likely it is by heaps more optimized than what you could hack together, but, heck, at least KNOW what you're doing!
Current schools squeeze out codetappers, but no programmers. They know a few concepts, they've learned the essential functions of one or two languages (most of the time Java and VB, or now C#), but put them in front of another imperative language and they're lost because their functions ain't there. And no resource kit, teh horrorz!
Of course, those people are cheaper. I get about 2k more than the guy sitting next to me, and probably only because I can see the difference between a CF and a ZF. And that I can accomplish in 2 hours what he does in a day, simply because as soon as there's no function for it and he'd have to figure it out himself, he's lost.
A lot of people who are squeezed out of school today know their tools, but they do not know the theory behind it. They don't know why they do what they do. They only know "Write code A and computer does B", but they don't have a touch of an idea why this happens.
And that's where buffer overflows come from. Because, hey, what the hell is a stack? Is there a function for that?
To get the rambling back on track, I could well see that "program-archaeology" is the next logical step in this. Actually, it has already started.
Or is anyone here that didn't resort to example code or open source code to find an algo that does what you wanted to do but couldn't figure out?
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Truely ... a sad day when a "Hello World" application is 1.5Meg ....
A piece of advice: saying "only real [insert any profession/hobby] do [something]" sounds a bit like you feel the need to prove yourself. And believe me, that is not good here on /.
You know, there is a difference between trolling and pointing out the flaws in your reasoning. Just saying.
I my experience, people who refer to the 'real world' and claim to know 'how the real world works' are in fact completely incapable of defining what 'real' or 'real world' means, and incapable of explaining why we should be talking about this 'real world' rather than some other opposing world.
Mostly we are admonished to talk about the 'real world' when someone feels that we are talking too much about ideas or abstracts. But much of what we know about the 'real world' was discovered through abstract, foundational or theoretical ideas, by people who were no doubt often told that their ideas had no significance in the 'real world'. I have this creeping suspicion that in this usage, 'the real world' often means 'my limited and narrow experience'.
Before one could reuse sourcecode on a global scale I think we first need to switch to a more advanced programming language, probally one not even invented yet. Today a hell of a lot of code does nothing useful, its just there to duct tape together different APIs or to make the compiler happy, while doing nothing by itself. And this isn't just a small portion of the code, but often a very large one, like half of the application or even more. Before that 'useless code' isn't reduced drastically I don't think reuse on a larger scale can work, since it would simply end in even more wrappering and duct taping.
Reuse in various forms keeps popping up as a "new idea" every 5 years or so. However, it has proven elusive. The reason to me appears to be that a truly generic unit would have a gazillion features in order to anticipate every need, and thus be a huge hunk of confusion. I used to try to strive for genericness in the past, and unanticipatable features always pop out of nowhere and gum up what I thought was an elegant design. The solution requires solving a combinatorial explosion, not to mention a complicated interface to handle all the potential options.
Table-ized A.I.
If a system fails any of these three criteria, the pressure to reimplement will be irresistable.
Vernor Vinge is an adequate science fiction writer, but his ideas about software engineering are purely fanciful. The very concept that you could replace actual programming with a purely investigative procedure has clearly never actually done any programming. Even if this were possible, it wouldn't be profitable: for any reasonably complex program, the amount of effort needed to understand the program well enough to repurpose it is, at least, equal to the amount of effort required to write a completely new implementation. Very few software systems, and almost no individual components, meet all three cirteria to resist re-implementation.
Device interfaces change. Communication formats change. Platforms change. Programming languages change. Worst of all, requirements change. Any one of these changes is enough to cause a complete rewrite of a component, even if that component has completely open source code. If a proram's source code is unavailable, or is written in an obscure language, the cost of re-implementation in the face of changing requirements can be a tiny fraction of dealing with the existing program.
just a ghost in the machine.
"A Deepness in the Sky" is a excellent novel. If you like SF, it's a must read.
Those few Slashdot readers for whom buying a paper book would be impractical can get an electronic text on a pirate channel. <pirate> Launch an IRC chat client (Opera browser has one built in) and join the room bookz on Undernet, Europe. Once there, enter the line
!EBrarian Hugo 2000 Winner Novel - Vinge, Vernor - A Deepness in the Sky.rar
For simplicity I have skipped the search stage here, listing a file that would probably be accessible for some time. You could get a list of all files by typing e.g.
@seek Vinge Deepness Sky
</pirate>
17779 eligible voters in a district, 17779 'vote' as one. This is Russia.
Likewise, how many glue codes have been dropped because they were perfect, except that their developers didn't handle execution models with multiple processes and one thread per process? Conventional wisdom is that multiple processes communicating through message passing (e.g. local sockets) are easier to get correct than multiple threads communicating through shared memory.
I use libraries when I write code. Sometimes I use those libraries to write other libraries (which I later use). My boss can write directly in assembler when it's needed, but he's still calling existing system functions which are just inaccesable from C. If you really want to get down to it, the people who made those system functions were just using the instructions already built into the hardware, which I hear is increasingly being created using higher-level tools.
-- 'The' Lord and Master Bitman On High, Master Of All
SOA is a nice concept (and buzzword). At the local-machine level, this is exactly what unix has always been. You have a bunch of programs with a standard interface (text through pipes/stdio instead of xml through http) and you tie them togather with "glue code" shell scripts (or perl).
> So if you have 30 different customer applications that all use the same iText based report generators, you only have to fix a bug in one place, or add a new feature (like a new type of chart) in one place, and you can release the update for all 30 applications ...and break half of them
More often than not, these constituent parts are Open Source software systems and typically not designed to be used as components. These parts are then made to interoperate through wrappers and glue code. We think this trend is a harbinger of things to come. What do you think?
Three words : Not gonna scale.
since the advent of assemblers, I doubt that anybody really codes from scratch anymore.
Alastair Reynolds : Century rain. " It's all right,' Niagara said. 'I won't be the least bit offended if you call me a Slasher. You probably regard the term as an insult.' 'Isn't it?' Auger asked, surpriced. 'Only if you want it to be.' Niagara made carefull gesture, like some religious benediction: a diagonal slice across his chest and a stab to the heart. 'A slash and a dot,' he said. 'I doubt it means anything to you, but this was once the mark of an alliance of progressive thinkers linked together by one of the very first computer networks. The Federation of Polities can trace its existence right back to that fragile collective, in the early decades of the Void Century. ,,
Slashdotters are murderous genocidal maniacs in that book ;-)
One of the problems that is occuring now is installing some of these programs. They may not list all the other programs you need first. This causes installs to fail for various reasons, missing header files, missing libraries, incompatable compiler options, incorrect versions of things.
It can be a nightmare with some programs tracking down all the dependencies. I've been there with several flavors of *nix and also with windows.
As reuse of code and/or code fragments becomes more and more common the problems with getting all the right stuff that is presumed to be installed located and installed to make life happy.
I recently put together a browser like tool in about an hour that would do a task in a matter of minutes that would have taken a week to do manually.
Of couse I could have coded it from C, built my own booting OS and something that would have done the tcp/ip then parsed the results into something recognizable as english text and written in hundreds of potential responses to whatever I got back and it wouldn't have taken more than a decade. Yeah, that would have been "code reuse" too since I didn't write C but I'm not sure that even with a decade I could have done it in binary.
Are we there yet?
B) Eliminate all the stupid users. This is frowned upon by society.
And some other guy will try to argue that there's some logical flaw in the idea that a CS professor doesn't understand real-world software development just because this idea is expressed on a product of computer science.
Of course, most of the technology that made the Internet possible was created by individuals who were not trained in computer science and were not academics, but that's another issue.
I've been using some open source components for a recent project, so this seems pretty relevant. I needed to upload a file from Java, so used the jakarta HttpClient. Works pretty well! Likewise, I needed to handle uploads, and javamail was stripping unescaped backslashes sent by IE/WIN. Jakarta to the rescue again, they've got a handy project just for parsing multipart form data.
Both of these were things I could have implemented myself, but certainly ended up saving a lot of time and probably bugs. I was impressed with the level of documentation on the apache jakarta site, it gave me the information I needed to start using the things right away. Plus, having the jakarta project name attached to them gave me some confidence that they weren't utter garbage.
But most importantly, each of the libs is small and single-purpose, and doesn't try to be a swiss army knife. It was very nice to find two existing frameworks that did what I want, and not much more.
These are projects I would happily contribute to.
You drank my drink, you drunk!
If this is what the future holds, then the average project will be so bloated that Vista will look like a "hello world" program by comparison.
MS has made programming insanely simple. DLL's and other forms of COM objects make code reuse very, very simple.
It's hard to beat the pipe | for insane simplicity.
By definition, command line unix reuses the output of one small program as the input of the next on all text based code. It's simpler than any dll linking and has been around since long before DLLs in Windows, Window itself or the MS DOS Windows initially sat upon (and then spent years trying to pretend it didn't).
This has always been Windows' strength and weakness. Unless an outright coder, you have no way of chaining one program to another. Power users resent having to code to do something that's just right there in CLI systems. On the flip side, with it being so much less accessible, the average user doesn't feel forced to learn the more complex process (sorry linux users but the average grandmother doesn't want to hear "Oh, you just need to grep the output of your find for..." and isn't going to adopt your platform while she assumes she has to do so to get anything done).
I remember creating subroutines to draw a circle on the screen, or fill that circle with a colour. I remember writing lots of different code that is now considered part of the standard C libraries. I think the key to proper code reuse is the same as it has always been: discrete functions that have properly documented uses and predictable results. I think the reason why some people have issues with code reuse, is because they aren't reusing the code properly. Instead, a program or process that sort of fits the parameters is used and then it breaks because something happened the programmer didn't originally anticipate. There will still be a lot of coding to do even with reusable code, since existing code isn't going to take into account every situation. The bottom line is, code reuse already happens today, and will continue. In order to do it properly, it will just come at a slower (but more effective) pace.
--
Luck is just skill you didn't know you had.
As a matter of fact, I find it lamentable that so many programmers are fumbling around trying to solve problems that have already been solved. I once had a book (part of it consisted of microfiche sheets, so that tells you how long it's been) from the ACM containing all the published fundamental algorithms. True, some of them were in assembly, COBOL, FORTRAN, and other languages that aren't popular now, but the theory was that programmers should be working on solving new problems, not reinventing solutions to old problems. Translating an algorithm from a different language is preferable (in productive mesurements) than reinventing it yourself.
Vinge has a point, and although I agree that it makes for a good story (because every story needs a problem), it is not good science: Surely those of us working on automated programming systems and generators will also try to invent a method of description that succinctly defines the outcomes of our program segments, and then we will use some type of genetic algorithm or other method of making the program most efficient within the constraints of the outcomes. In other words, software will be "grown", not coded, but the description of what is desired will include a constraint that makes it efficient. Furthermore, most of the tasks and workflow will be described in a similar language, so if a task has already been described, the method of solving it may have been cataloged and "component-ized" so the archeologists' job will be as simple as following a circuit diagram. However, IMO, that part of the problem will be done by robots...
I think the current code repositories and model repositories are precursors to huge "code warehouses" or "model warehouses". The program designer will define the task, the program generator will decompose the task description semantically and use some type of pattern matching to find a suitable "component" in the repository, generate the program or system according to some type of design or artistic standards, and then grind the system 'til it becomes highly efficient.
I believe we are closing in on a time when program development will be more akin to sculpture than logic.
"The mind works quicker than you think!"
This is stupid. We already use multipurpose tools for generating the code we want - we call them compilers. What they're talking about is the simple process of building up higher and higher level code. We're no longer coding in punch cards or buttons on the front of the computer, are we? If you had made the same prediction back in 1960, you'd have "code archaeologists" going back and finding old punch cards to translate into programs today or maybe people reusing the same bits of assembler instead of coding it from hand.
Did you ever notice that *nix doesn't even cover Linux?
Wouldn't that be a step back? I mean, someone has to write those blocks of code that you reuse... and if no-one does, 1980 will be calling soon, saying they want their state-of-the-art sources back...
Pentaho, the Open Source Business Intelligence Suite is a very prominent example of this new approach on development. It consists of well-known projects like JFreeReport, JFreeChart, Jasper, Kettle, Mondrian and others.
What the Pentaho guys do beyond marketing is writing glue code and user-friendly development tools. They also usually hire the project owners after incorporating a new project.
fiction becomes reality? Have a look at http://merobase.com/ for example. You can start to recycle code right now.
Mod the parent up. It's the best example of code reuse and component programming in history.
The big problem with software components is the interface. Not the software interface, the other one: the GUI. No matter if it's browser-based or window-based, it remains a fact that a small change in interface requires a similar change on many levels of depth along the chain of components.
At the very least, you have to separate the software in Interface Software and Do Stuff Software, and to put them together is considerable effort. This is pretty much the multi-tier philosophy of web applications. It's good enough, but it's far from "cut and paste" component programming.
That's partly why components work very well in non-UI environments (libraries, unix shell). If anyone finds a way to go around this problem, lots of programmers will be out of jobs
You've just explained the major economic reason for the general lack of code reuse over the last couple of decades.
Deleted
Actually, i think Ariane 5 screwed up because of a conversion of a 64-bit floating point to a 16 bit integer. I believe that this actually has more to do with lack of sufficient testing (particularly on boundary values), rather than the use of 'glue code'
This is a wonderful idea that I've been hearing variations of for many, many years. I don't think it any more likely for software than I do for books, however. It is conceivable that someday authors would create a "new" book by selecting sentences, paragraphs and maybe even chapters from existing books and just adding a bit of "glue" to connect them. In fact, this has probably already been done. But was it any good? I doubt it.
As for object re-use, I can never forget one of my programming instructors saying, "Don't reinvent the wheel." But despite using that phrase all the time, he was never able to explain why we have so many different types of wheels, with more being added all the time. How can anyone truly believe that something as complex as software will ever be so perfectly encapsulated? Heck, we aren't even doing it with simple objects like houses or cars, though, maybe, someday, there will only be one standard set of car (or house) parts that are used. Total uniformity would certainly ease maintenance, although the aethstetic aspects may suffer.
For a computer example, how about something fairly fundamental, like a vehicle guidance system... Changing a few parameters and adding a little glue code should be the most that is needed for the softwear to drive a car, fly a plane, navigate a submarine, roll a segway, or travel to Mars, right? After all, all of those things deal with the same inputs (mass, gravity, inertia, direction, velocity, etc.) don't they? I expect that if such a chunk of software is written it will be massively outweighed by the "glue" code required to support it on each particular device.
But I'm a pessimist and a Doubting Thomas who probably should have been born in Missouri...
Does anyone code from scratch these days? perhaps some embedded devices developers...the rest of us use the APIs provided by the operating system and programming language we use, or use alternatives found at code repositories.
I can't think of a single project I've worked on that didn't include at least one major piece of FOSS.
Retired from software... maybe. Sort of.
Giving a new meaning to trash-bin, eh?
There you are, staring at me again.
#include
void main()
{
printf("What? It isn't already happening?\n");
exit(0);
}
If all you have is a hammer, everything looks like a thumb.
Another SF writer, http://www.amazon.com/Newtons-Wake-Space-Ken-MacLe od/dp/B000C4SFRE/sr=8-6/qid=1162740889/ref=pd_bbs_ 6/104-0751852-8763166?ie=UTF8&s=books/Ken McLeod, has had ideas similar to this in his books. In the last one I read, Newtons Wake, the characters often have to engage in combat archaeology, which mainly involves the aggressive hacking of ancient technology. He introduces the concepts with a sense of humour -- in one segment of the book, the characters' electronics are attacked by a virus beamed from a an advanced technological artifact. They are able to tell the virus is based on human-created code because once they go deep enough with their combat archaeology, it's east to spot the Microsoft patches in the virus code.
Reuse of sotware (as suggested by the original posting) is (in my opinion) greatly hampered by the use of C/C++ and systems that are not modular based. By "module" I mean what Niklaus Wirth created, first with Modula and later with Oberon and Component Pascal. Only by having strick information hiding (which is the default in Oberon) can one hope to be able to paste and glue (reuse) software. The C/C++ default is the export everything. It is a white box. The Oberon default is to export nothing. It is a black box. Blackboxes are much easier and safer to use.
Actually, the worst software engineer I ever worked with had a PhD in computer science. It tells you nothing about someones understanding about software engineering. For all practical purposes the two are different fields.
There are two different fields within CS, is probably a better way of looking at it. I don't know what Vinge's exact area of study is, but given his writing I'd guess he probably focusses on computational theory and/or AI. This means he's likely to have a very solid foundation in terms of software engineering. This guess fits him in a category along with Donald Knuth.
Other computer scientists study stuff like language semantics, concurrent architectures, mathematical approaches, stuff like that. I'd guess the guy you worked with is one of those.
...application programming will consist mostly of using collections (lets call them "libraries") of code other people have written in the past, along with constructing bits of glue code to connect them together.
Er, isn't that what is done now? I mean, sure, there are some situations where you can't rely on either actual libraries or the equivalent in the built in functionality of an HLL (like writing certain basic code either to support new hardware or to tweak existing low-level code for better performance or extended functionality), but its not like most programming today is "programming from scratch".
There's always, I think, going to be an irreducible amount of programming from scratch as long as new hardware is created which presents a new interface to software, but, yes, using code that other people have written avoid having to reinvent the wheel is important now, and will probably be increasingly important as there are more and more libraries of code out there that are available to use.
Seems to me that, while there is some colorful language in the description, there's not much new predicted here.
They didn't build it from scratch. They used all sorts of screws and nails that had been engineered by other people.
This is just a natural progression. A few years ago we stopped having to write quicksort routines because they were already done, now we dont have to write webservers or rich text editors because they are done for us.
The end result will be that it becomes cheaper to make complex applications and so the customer will demand even more complex ones.
There's tons of Open Source out there that you /could/ use, and there's tons of commercial libraries out there after your dollars.
/I/ think is worth the money, let alone can sell to a PHB. ... and for Open Source, you need your employer to OK its use. Now this may be OK in smaller companies, but in the large companies I've worked for it's a hurdle. Mainstream stuff like OpenSSL, Perl, Hibernate etc. are fine (because someone else has gone through the process before you), but anything else has to go through a heavyweight approvals process that's often more work than writing the damn thing yourself.
But for commercial software, you've got to convince your PHB that it's worth the money (and I rarely see a commercial library that
There are two main reasons for this:
- people are concerned about using code of unknown quality
- people are absolutely terrified of IP law: accidentally incorporating GPL code in something they don't want to GPL; relying on software that apparently has an acceptable licence only for it to turn out to be someone else's original work.
That, to me, is the big barrier to re-use on a massive scale.
I'm trying to solve that very problem as my PhD. My solution would involve using "semantic web" formats (which are kind of superior forms of XML, or restricted forms of logic) to represent data, and tags (a la del.icio.us) allowing users to build meaningful data collections without programming a backend.
The yet unsolved part is how to give users the power to automate common tasks. I plan to use end-user programming techniques for this.
You've expressed the problem in a very concise way. There are already systems trying to bring the Unix philosophy to the User environment, like Archy and the Haystack project, which use different but related approaches.
Singularity: a belief in the "God" idea with the "demiurge" relation inverted.
I always tend to create the technology behind companies I start, from scratch; but using libraries which are very rich, and becoming better all the time. Modularization and improved libraries, tends to blur the lines of what "from scratch" means. Even a raw C program that uses the standard C library, isn't technically "from scratch," since it leverages the work of others in the libs.
Love many, trust a few, do harm to none.
It is a great idea to take different components and glue them together using languages like python, perl, (or for the web) php. However once those components are glued together they only fit one specific purpose. No one can take advantage of all of your glue-work, which may be a large project in itself.
We need better ways to glue and then encapsulate. Take a number of components possibly of different languages, or even possibly architectures, glue them together, wrap them up in a single shared object binary, and expose only the needed portions.
With a language like python you can get 90% there, but its reliance on the python runtime is a major bummer. Compiled python may be a valid choice, but I haven't seen it used too much in practice.
Fast Federal Court and I.T.C. updates
Not "code re-use", re-re-re-discovered again?
One day a group of scientists got together and decided that man had come a long way and no longer needed God. So they picked one scientist to go and tell Him that they were done with Him. The scientist walked up to God and said, "God, we've decided that we no longer need you. We're to the point that we can clone people and do many miraculous things, so why don't you just go on and get lost." God listened very patiently and kindly to the man and after the scientist was done talking, God said, "Very well, how about this, let's say we have a man making contest." To which the scientist replied, "OK, great!" But God added, "Now, we're going to do this just like I did back in the old days with Adam." The scientist said, "Sure, no problem" and bent down and grabbed himself a handful of dirt. God just looked at him and said no..no...no , You Get Your Own Dirt!
Thanks for the links. I took a peek and they're interesting but don't look like they're ready for the real world just yet.
... parameterN). The rest can be as problem-specific as needed.
I discovered Lisp last year. Unfortunately I don't have time to learn beyond simple programs (though i still intend to) but i loved the concept of tackling the problem from the other end: instead of having an environment simple enough for everybody to use, have one so powefull that skilled programers can turn it into specific languages for specific problems.
I don't know how this applies to the interface problem, but reading this in your links my mind automaticaly went to lisp. Helps a lot that the basic syntax can be learned in a minute: (function parameter1 parameter2
There is nothing new under the sun. Everything is just a rehash of Ada Loveless' work.
Program in Perl and pattern matching comes built-in. Program in Python and you get dictionaries -- key-value pairs. Delphi gives you its VCL. Program in Java and you get instant memory management.
Making a robust module that people are going to want to reuse is hard, and the people developing OS's and computer languages and associated frameworks are willing to do that hard work. Letting a thousand flowers bloom of a miriad of 3rd party suppliers hasn't developed to the same extent, although there was some hope that component frameworks like ActiveX, .NET, or Java would make that possible.
The idea was that the right framework would facilitate reuse -- in practice, the right framework is in itself the reuse because it has the features you need built in.
First, that's what libraries are all about. But then, that's also why compilers have become gigantic they're trying to write every possible function that will ever be needed into the libraries.
Of course, companies have their own functions, but almost *never* have a company librarian, or standards that hand you, as a programmer, these functions - it's always "go see if it's out there, if anyone knows".
But then code, properly written, is easily usable like this. The ultimate example is, of course, *Nix (grep blah filename | awk 'NR > 5 && NF > 0 && ! ($1 in myarray) { myarray[$1] = $2;}END{ for (i in myarray) { print i, myarray[1]}' | sort |etc |etc.
mark
IIRC, the problem was something like "my ship might end up in a zone where only a 386 would work", so software archeology was one way to write code for archaic systems that would be poorly understood by someone used to computers thousands of years advanced from that. This is NOT only science fiction. Imagine trying to get a room full of Apple IIs, Ataris, 286s, C-64s, and Amigas all up and running. Anyone remember all the custom autoexec.bat and config.sys files needed to get certain games to run under DOS? BTW, One of the coolest things in Vinge's book was the galaxy-wide network of Usenet servers :)
I'm the source of the original story. (Thanks, Clif, for choosing my post!) What I submitted was much longer and what I was really interested in unintended resuse of Open Source code and the emergence of tools to support such practices. You can see the unabridged version on my lab's web site.
I'd like to respond to a few of the comments.
I'm not trying to float an idea, I'm just making some observations of what people are currently doing.
Yes, reuse is old. Reuse is sometimes bad and sometimes good.
But what is new, I think, is the wholesale use Open Source systems as components, and the way in which people trawl the web for source code. We've had libraries, DLLs, Perl modules, classes, and the like for ages, but these were designed to be reused. But these Open Source systems were built as stand-alone systems, that people are turning into components ad hoc. When people start designing a system, they start with the idea, see what's available out there that they can reuse, and continue the design process from there. I was trying to ask some speculative questions about what software tools are needed in these situations.
In particular, there has been a proliferation of search engines for code on the web, such as koders.com, krugle.com, Google Code Search, and Sourcerer, here at UCI. (I'm just a university researcher and not affiliated with these sites in any way.) I have a suspicion that people need features other than the ones built into these tools and that people are being very creative with existing web resources to locate the code that they want. I am a conducting a web survey to find out whether this is the case.
I had been told that Vinge's "Deepness in the Sky," contains more detail about zones of thought and complexity, and it's sitting on my shelf. But I didn't want to insert a gratuitous reference to a source that I have not read. Look elsewhere on this topic for a more detailed discussion of Vinge's ideas.
The comments have been really interesting. Some went in directions that I didn't anticipate and others provided insights that were new to me.
Lisp is the mother of all languages :-)
:-)
Once you learn it, you achieve the zen of programming - lisp is the closest possible language to the mathematical definition of computing. Have you read The Little Lisper? (republished later as The Little Schemer). You can read it in a few evenings, learning some deep programming paradigms and the basics of several complex applications, including compilers.
End-user programming has already had one big success in its history: the Spreadsheet. Programmers of visual languages consider that spreadsheets are the most versatile (and most widespread) modelling language to date; problems of many different domains can be expressed with cells and functions, and non-programmers can use it efficiently to that end.
This flexibility is grounded on the same principles as Lisp: a simple syntax without special cases, which can be used to build any grammar on it. Mi goal is to find a similar language wich can model automatic tasks the same way as the spreadsheet can model data. I'm still in a phase where I accept suggestions
Singularity: a belief in the "God" idea with the "demiurge" relation inverted.
You have at least a few different levels of abstraction in the industry (you could probably think of more if you tried): applied physicists who understand transistors and atoms and all that stuff, hardware people who understand NAND gates and RTL, Systems people who understand Linux and kernel modules, applications developers who understand software engineering (hopefully), and scripting/web people who understand integrating programs.
I hate to say this, but people who are further up the hierarchy tend to look down on those below. Physicists tend to think that hardware is just a kludge. Hardware people tend to think that software is easy. And everyone thinks web development and scripting is easy (except the people doing it).
The people further down on the hierarchy can usually be more laid-back and less mathematically oriented. Ideas from academia also tend to be more relevant to people further up the hierarchy, and less relevant to those below.
None of these levels are really "fixed"-- new advances are being made all the time by the physicists, and it ripples down the chain.
This is bad news for people who want things to stay the same, but good news for people who enjoy action and adventure (and being considered obsolete old farts at 35). Also, look for more layers of abstraction to emerge in the future.
"No more coding from scratch" would only really be true if the operating system and software levels froze up, and new development ceased. But as long as those crazy physicists keep pushing more transistors on to a chip, it's a good bet that today's software will look like a bottle rocket compared to the Space Shuttle of tomorrow. Hold on to your seats, folks, and don't sell that MSFT or GOOG just yet.
"Any connection between your reality and mine is purely coincidental." -Slashdot
Maybe one day geeks, nerds (or whatever you want to call them or call themselves) will get this in the part of the brain that is necessary: programming is not art.
There, I don't care if it hurts. I have made my good deal of programming, saved the world once or twice, was very satisfied about it, and then moved on. I am also an artist on my free time, play piano to good standards, compose a bit, write novels, etc. So I think I know the differences.
Programming is an engineering-mathematical activity that can be analyzed formally in terms of mathematical equations and theorems and that can (and should) be sujected to constraints, costs and project planning for it to be of any general use. This is more so in the context of the original posting.
Now, if writing a perl script makes you feel all artsy-farsy, all the power to you, but the pleasure derived from it is not aesthtic in nature and will not convey any aesthetic reaction from anybody normal.
IANAL but write like a drunk one.
Many composers do exactly what you are saying.
Beethoven "Prometheus" theme is heard in many of his compositions, as is its theme from the 5th Simphony.
As a matter of fact, Wagner advocates this approach (leif motifs) as part of composing an opera.
And there are uncountable examples of composers that borrowed material from others to make their own compositions (the lazy bastards).
IANAL but write like a drunk one.
The subject of the ICFP 2006 Programming Contest was about exploring a dead system of an old civilization. The challenge is still available. Try it! It was really fun.