I have been involved in software reuse since the mid-1980s and possibly even earlier. There has been lots of energy expended on the problem of making existing implementations extensible, one of the strengths of OO technology, though not requiring OO. The big piece that has always been missing has been a major concerted effort focused on facilitation matching a developer's needs with existing software.
There are many mechanisms that can assist such as:
1 - technical reviews. When these happen, you get a number of co-workers together to review your work. Not only does this assist in ensuring that direct work (architecture, design, code) is correct, but it also provides an opportunity for all those involved in the review to search their knowledge of pre-existing "parts", be they architectural, design, or actual code, and to suggest you investigate them. Of course, if you're like me, then actual review meetings where a number of people sit down and examine your work just do not happen any more. Thus this form of identifying existing work that can be reused no longer functions.
2 - CASE tools... I have long felt that CASE tools, yup those tools that are totally out of vogue right now, would be of greatest value if they had a dual function. Their primary function would continue to be as a means of describing architecture, design, or code, but a secondary function would be to, in the background, perform a continuous search of existing work looking for matches. I have never seen a tool that does this, yet this seems a tremendously valuable function.
3 - personal memory - only works for those items you already are familiar with, which frequently gets voided when changing jobs.
4 - institutional memory - this is similar to the technical review mechanism, yet is less well defined. The real question here is HOW does an individual tap into an institutional memory? Documentation search? This is far less than perfect even if all work was well documented. Code search? Even worse at turning up matches to needs.
So... the bottom line is that it truly is VERY difficult to match up needs of a software development effort with the existing software that is available.
Once case in point... I worked on a very large project for an FAA (Federal Aviation Administration) contract. One mechanism I needed was a circular buffer/queue. These seem very straight forward to implement, and an obvious place to use an existing piece of design/code. Well, even after extensive search and review I could not find such a part and built my own. Later, I discovered there were at least six independent implementations of a circular buffer/queue in this single project team. All of them were general enough to meet the other implementation's needs, yet somehow none of us knew of the others' overlapping work. If we couldn't coordinate the reuse of these six independent efforts (and that means we all built the same basic algorithm, found the same set of bugs... and yes, using our code management tool I was able to see the same bugs being fixed in each implementation... and thus a total and unnecessary duplication of effort), how in the world will we ever solve the problem of reusing work outside the single project team, or outside a company?
There are some examples of wild success with reuse... though they seem to me to be more success though definition. All of those shell scripts that are built from individual command line tools are examples of reuse, where each command line tool represents a unit of software available for reuse. But, I think we all think of reuse more at the code module level... a function, or class, or small library. And it is at this level that I think we fail miserably, and it is my contention that we fail because we can't easily find the candidates for reuse.
I think the larger problem is that this administration tends to take a "damn the torpedoes, full speed ahead" approach. If they think something is the right thing to do, they ignore any voices to the contrary and just forge ahead. This is exactly how you end up with problems like Chernobyl (read the book, its in there) and the two space shuttle crashes. As tragic as the shuttle crashes are, they do not have a direct long term impact on world health and safety. That is not the case when it comes to Nuclear power. It is scary indeed to think of what these folks might do to pursue their ideological views without letting a few inconvenient facts get in their way.
To tie into another posting... we may just get that operational test where they, with a fully operating core they dump the coolant. If things work as they are engineered and as the design intends, sure, things will likely be pretty much safe. BUT, what if things go wrong. We've just created our own disaster without any means of undoing it.
Why don't we throw a few live ICBMs at, say, a couple of our own cities to test SDI? Who says you can't test it?
I suspect a lot of people were not around in the pre-FSF/GPL days. The various Universities with computers would share their software with each other, and pretty much anyone else who had a computer.
As soon as two machines could exchange files via UUCP, people and organizations were sharing software for no charge. There were even cases before any kind of communications based transfer were possible where, if you supplied the tape and shipping, you could get a vast array of software for that day.
Of course, back then even the software you paid the big-bucks for was distributed in source code. I was involved with one of the big players of the time when everyone realized there was a workable alternative to shipping source code to every customer and asking them to compile and install every software product. Customers would write their own patches and utilities based on the insights into commercial product code offered by the source distributions. I should add, though, that given the huge burden of building products (a large percentage of even a large mainframe's disk space was consumed for many hours building just one of the major products), customers were right at the head of the line insisting on binary software distribution. It also made support MUCH simpler since you didn't have to worry about what fun new features a specific customer had added to their copy of the product.
Anyway, to get back to the my main point. GPL came along well after the tradition of free software had been established. I do not see it as essential to maintaining the free sharing of software. There ARE forces at work that will reduce this practice, but I don't believe GPL hinders those efforts, they being software patent madness and the US Department of "Homeland Security". (should I add that even in the days of the Soviet Union, Soviet computer people were both contributing and consuming free software. I am sure GPL had no impact what so ever there)
It should go without saying, but IBM OSes have been very expensive since the dawn of time (well, the start of the computer industry). The pricing has always be linked to the hardware costs, and those have not changed much in the big iron mainframe world (well, you get a great deal more power for the same big dollars now, but no small dollar machines). I do think the suggestion of some form of "hobbiest" license makes a lot of sense, particularly since with the tremendous power of today's PC hardware it would not be unreasonable for someone to implement a full emulation of the mainframe on it for training, development, and test purposes.
I really have to laugh at placing Sun and IBM in the same catagory. No way. Sun, even its biggest systems, is still a workstation and server platform while IBM's mainframes are BIG iron. I do wonder where the IBM mainframe clones have gone? Amdahl anyone?
While I have no particular animosity toward Intel, other than it is important for there always to be competition to push them, I do not think they need to be let off the hook. Itanium has been around a very long time. You may think of it as new technology, but that is more because of the lack of acceptance in the marketplace, not because it has only recently been released. What was happening all of these years since Itanium was initially launched?
Additionally, while the Itanium instruction set takes a different approach to those of Intel's competitors, they are not the only company introducing new CPUs. I do not remember such problems when other 64bit CPUs with their own, new, unique instruction sets were launched by Digital, HP, IBM or Sun to name just a few. These days, the competitive landscape has been radically reduced. Digital no longer exists and its Alpha architecture is owned by Intel. HP, while it still owns its PA-RISC architecture, is trying to migrate its customers to Itanium, though it is hard to say what will really happen to PA-RISC since no one seems anxious to adopt Itanium. IBM also has picked up Itanium, so who knows what will happen to their RISC architecture? That leaves Sun, and while SPARC has always been the weaker of the RISC architectures, it seems to be the primary remaining competitor to Itanium and Intel. Of course, who knows how much longer Sun will survive as an independent company? Maybe they are the next to be gobbled up by IBM or HP, both already commited to Itanium, so what happens to SPARC?
Finally, it is hard to say exactly where AMD fits in all of this. Its 32-bit line provides excellent competition to Intel's 32-bit Pentium family (now at P4), and the AMD 64-bit architecture looks like a nice increment beyond the now very old x86 32-bit architecture. But, in terms of major pressure on future CPU architectures? I just don't know where that competition will be coming from... Maybe China, Russia, Japan, India? Places not noted for their hi-tech prowess, but with lots of experience in fabrication and lots of affordable talent?
While I would like to be as optimistic as this post, I can't say there is a vedict yet. I think the fundimentals seem very clear. Almost every business depends on software to make it go, and that software needs to reflect each business business practices, making off-the-shelf software only a base element. It takes new software, or software modifications and/or integrations, to support changes in a business' practices. Thus, while right now there is a slump in the general economy, and a what I would say is a depression in the software industry, I do believe that things have to come back some soon as businesses can not afford not to innovate, and that will require supporting software.
There is lots of room for improvement in how software is developed. It would seem that the state of the art would be a IDE, with Visual Studio the model that everyone is shooting for. But, while Visual Studio.Net is slick and possibly the best of its class of tool, it is by no means the state of the art for software development. And even that state of the art that few get to use in "real life" leaves lots of room for innovation and improvement. The challenge, as I see it, is that entry into the field is getting quite expensive, both due to the complexity required of an inovative product and non-technical factors. This challenge was overcome in the past in many other fields and at some point it will be overcome in software (at which time all of us will end up having to learn yet another set of very new skills that will make object orientation, c++, Java/c#, and IDEs seem like the dark ages). It is not possible, though, to predict when this is likely to occur.
I do not believe we will ever see the software industry return to the way it was before the current crisis. In fact, I suspect it will resurface with a new look. But, I just as strongly believe that there will be room for the solid software professional with a deep knowledge of software while being literate and articulate, thus able to communicate well with the business side of the operation.
ACM Communications, the primary publication of the Association for Computing Machinery, the Computer Science professional society, contains a special feature on Digital Rights Management. I have not finished reading all of the articles in this section, but my take thus far is that ACM US has taken an active role in attempting to provide technological guidance and advise to Congress on this issue and with respect to both proposed legislation and court cases testing laws in this area. Yet, the pressure that has been brought to bear by the big copyright holder interests thus far have far outweighed those of the technologists and their commercial interests.
Additionally, there have been a continuing series of articles in this publication as well as others produced by ACM providing additional analysis of the topic along with recommendations for action.
Give it a good read... its both encouraging that such comprehensive analysis is in fact being brought to the highest levels of government, and discouraging in that thus far it seems to have made little difference.
Finally, this might be a good reason to join ACM to add to the strength of their(our) voice.
And not to put too fine a point on it, I DID provide for an out in my initial posting. There ARE cases when one needs to optimize the heck out of the code, to fit time, space, or both. The number of such cases has been decreasing at a very rapid pace. Even many embedded systems now have lots of CPU time and memory (both RAM & persistent) at their disposal. Just check out recent issues of ACM Communications, IEEE Software, & Dr. Dobbs.
Finally, while 5% of the total installed base seems a significant fraction, this isn't always the case. For the little guy, even getting 1% is hitting it big. On the other hand, for the big vendor, they may choose to give up on that 5% in exchange for less expense in their software development.
Long ago the developer/hardware equation was changed. Originally, hardware was far more expensive than the people developing the software. That was when investing lots of energy into hand-optimizing was the proper tradeoff.
Years ago, before 1GHz was considered a short term possibility, hardware costs had decreased and software costs had increased, to the point where it is the DEVELOPER who is the most expensive piece of the equation. Thus, we now are at the point where, with the exception of a few very specialized segments, we do whatever it takes to optimize the developer time in building software. That is not to say that developers can be careless and wasteful. But that developers should not waste time optimizing code. IFF performance is an issue, THEN one takes measurements and optimizes critical areas consuming the majority of time. Beyond that, it just isn't worth it. Today's 3GHz machines with GB of RAM only reinforce that this is the only appropriate approach to software development.
I did my time as a third shift operator as a student summer employee at IBM's TJ Watson Research Lab. This was the summer between my freshman and sophomore years of college, so I had only had two semesters of formal computer work, and a year of work on APL and BASIC before that.
I spent that summer pulling listings off the line printers (remember them?), cutting graphics output printed on thermal paper that was scattered around the floor by its printer (that was state of the art graphics then), and hanging tapes (primarily for the nightly backups). In my idle time, and there was LOTS of it, I read the manuals on the systems from the company library, and then showed the full time permanent operation staff how to customize their environment and write what we now call shell scripts to automate their environment.
Yup, a college sophomore that was very much wet behind the ears was showing the veterans of many years how to make their jobs easier simply by programming their function keys and writing very short shell scripts. That speaks pretty loudly of the type of people who were on the operations staff. And this was at IBM's front line research lab, and they were very clearly hand-picked from everyone available in the general area which had many very large IBM facilities.
Once in a while we had to deal with an emergency...... a jet plane landing in the machine room, oh, I mean a full head crash on a 3350 disk drive, the washing machine size disks with removable platter assemblies of many (was it 12) platters, and using the very first generation Winchester heads.... a minor flood caused by a huricane and a clogged stairwell drain. Nothing like water hitting high voltage power in a transforming closest for some excitment. Though, once we were done trying to defend the machine room with mops, we just sat around until power was restored and then saw who could seek out all of the devices that needed powering-up (it was a HUGE machine room and there was no map) and who got THEIR machine up first.
The normal staff was three third shift operators, including me. We were running three of what were then the biggest monsters IBM made, and possibly the biggest in the world at that time... three 370 model 168 mainframes, with what I believe was the maximum available CPU, Memory, and I/O subsystems.
It was a very interesting, if sleepless, summer. The insights into who the operators were, and thus who the users of many system level tools are, have lasted a career.
By the way, this was the summer of 1976.
(did someone say they wanted to hear some of that great IBM company folklore I heard that summer? I don't think that non-disclosure for that job prevents me from sharing anything at this late time)
You make light of this machine's accomplishments, but in its time, it was truly a wonderful machine. Just the thought that there was this computer and you could actually take it with you on a trip so you had it available where ever you were going was just fantastic. While there were a few that pre-dated it (there was a similarly sized APL machine, the IBM APL 5100), this was truly a revolutionary machine. It was such a shame when they announced the Osborne 2 prematurely and EVERYONE decided to stop buying the current model and buy the new one when it came out. But, out of this disaster people realized just how powerful the idea of a computer you could haul around really was.
I have used just about all of the various Unix flavors and platforms at one time or another as well as RedHat Linux.
OS:
Solaris - Tends to be clean, fast, and buggy. The most patches I've ever applied on a non-Microsoft platform was Solaris.
HP-UX - a Solaris clone intended to make porting from Solaris very easy. Generally, I found it to be a very good Solaris with many fewer bugs.
AIX - Very clean and very fast. A terribly literal SysV Unix, thus frequently rather spartan and basic. But, it cranks things out quickly and reliably.
Tru64 - Well, almost not worth mentioning since Digital was gobbled up by Compaq which was gobbled up by HP. If not for the fact that it was definitely at the top of my list for high quality Unix platforms. Stay away from DECnet (there for VMS sites) and DECWindows (ditto) and its a pretty nifty platform. The one downside... Hate that digital uses a software licensing system for pretty much all of its products. Now that HP owns what once was Digital, it can be pretty painful to buy a license.
Linux - Requires a lot more hands-on time to get things up and running properly, a bit unpredictable what quality of component one will encounter. For example, its VM system just does not compare to that in AIX or Tru64. If you don't have enough RAM to fit everything in real memory, paging performance just doesn't approach the competition.
Hardware:
SPARC: Always bringing up the rear in price/performance, but marketed extremely well.
HP-PA/PA-RISC: Far ahead of SPARC in price/performance, but much later to see ports of your favorite software. Not going to be around much more as HP is hot to drop it for IA64.
Alpha: Also far ahead of SPARC, usually playing leapfrog with HP-PA/PA-RISC & PowerPC. Not going to be around much more, as HP is hot to drop it and their own HP-PA/PA-RISC in favor of IA64.
PowerPC: Another very strong RISC platform, also with a questionable future due to IA64 plans.
The bottom line: Sun, while not necessarily the best Hardware or Software vendor, has provided a very important competitive presence in the industry. As they head down the path to low cost Intel platform solutions with Linux software, from where is the competition and inovation in Hardware and software going to come? It sure doesn't look like Sun will continue to be a significant player.
It was a great language, originally created to DESCRIBE and design computer hardware architectures, but a few of the wizards at IBM's Watson Lab decided to implement it as a programming language. Still the only language I know of that can multiply a pair of matrices with a single NATIVE operator! No user coding required.
Of course, with all of this power APL also invented the write-only program and the one-liner, both predecessors to the obfuscated C contest... and I must say that those using C and C++ can only dream of approaching the unreadability that was APL. Then again, only APL used its own character set... it used single symbols derived from mathematics as its operators. No constructing multi-character composite operators in that language... just custom terminals, or... a special type-ball for your selectric based 2741 terminal (there's a trip down memory lane).
To return to my point, though.
APL died for a number of reasons, but probably the biggest was how totally undigestible its code became with very little effort. I wrote some fairly simple programs, relatively well written for APL, avoiding the tradition of making everything a one-liner, actually including comments... you know those things written in intelligible English that explains what a program does... oh, yeah, I haven't seen any of those in years in my C/C++ shops... and yet as little as a few months after I wrote those beautiful, elegent APL programs I had no clue how they worked and could only change them by re-writing them. A very good reason for a language to die.
The shame of it all is the APL was probably the best language ever invented for solving hard-core math and arithmetic problems, yet it died before it could even run on its natural platform, the super-pipelined vector processors such as the Cray-1 and its relatives.
My favorite example... I was working on a large C++ software system. There was a major piece of business logic that included a sort of a very large list. The list was using the double-linked list C++ template class and was taking a very long time to execute. I was told to change the implementation to PERL since PERL can sort so much faster than C++. Instead, I investigated a bit, found the problem (the use of the linked list template class) and switched to a more appropriate structure (I used one of the hash based templates). My management was shocked when the C++ code was suddenly executing 100s of times faster and I hadn't converted the whole thing to PERL. Just for fun, we converted a piece of it to PERL so we could try the same sort. The PERL sort took two to three times the time that the C++ code did once I switched to a more appropriate class.
My basic rationale was... PERL has to eventually boil down to C++ logic, so why would PERL be FASTER? An interpreted language working with abstract datatypes vs. a compiled language that uses somewhat more concrete types... and as you move more toward C, you get to very basic data types that map directly to hardware.
There certainly are times when scripting is the right solution. But, compiled languages do have their strengths as well. One needs to keep these in mind when choosing, rather than just blindly thrash from one language to the next.
Note that with a secure "licensing facility", software vendors can and will have their software expire. Thus, you will purchase a license to use a software product for a specific period of time and it will not be usable after that period of time without additional expenditure to purchase additional licensing.
This has very interesting unintended consequences that I'm seeing today. Here's the example... I am working on something that requires me to use some old legacy operating systems. One of those is the old Digital Tru64 Unix product. So, off we go and buy some hardware from eBay, and then the required Tru64 software. That would be enough, right. Wrong. Digital implemented within all of their operating systems a licensing mechanism that requires one to obtain a license seperate from the OS media. That was fine when Digital was its own company and could justify staffing an office for issuing these licenses (assuming one buys into purchasing those licenses). But, today Digital does not exist. Even worse, Compaq does not exist, having been gobbled up by HP. HP probably had no idea when they purchased Compaq that there was this need for an office that issues licenses for the software that Digital had produced. So, today, while HP gladly will accept your money to purchase these licenses, it takes a significant amount of time as there is ONLY ONE PERSON staffing that office. What happens when that person finds a new job (not likely in today's software economy but what about tomorrow's)? Or retires? Who will issue the licenses then? Or, will it just become impossible to issue these licenses, thus rendering it impossible to use those products?
Elizabeth Haydon is a delightful author who has come on the scene only recently. Her initial book Rhapsody was absolutely stunning in its writing. She has now added three (or is it four now) books to her writing and they are all of the same high quality.
The aspects I most enjoy are her wonderfully flowing writing, so strong that she is one of the few to follow in the footsteps of Ursula Le Guin. I have a very hard time not trying to read her books in a single sitting, which would be a bad thing to do given their length and my reading speed (we're talking about a couple days minimum to read straight through to the end).
I had a great old Canon printer, but have been having big problems finding ink cartidges that fit it (even via the web). Thus, when I finally needed a new printer I went HP because I've NEVER had a problem finding the proper cartridge for any of their printers. I'll seriously consider vendors other than HP for my next printer when I can be sure I won't run into the availability problem again.
I find I spend a LOT of my time debugging or working around bugs in the tools. Nothing is more aggrevating and frustrating than getting exactly where you need to get to find a bug in a program only to have the debugger itself crash with a SEGV. Between that, and the inability of many of these tools to work easily in the modern dynamic library and distributed computing environments, life is far more difficult than it should be using the existing model.
Thus, I want a debugger that absolutely does not crash. I want it to let me set breakpoints in any routine, even if it resides in a dynamically loaded library that is not currently/yet loaded. I want to be able to view ALL of my data in their abstract datatype form without having to look up their definitions and then manually guide the debugger through doing the job. And, I want a debugger that always honors my commands and does not lock up (which I find far too often with the GUIs layered on top of command line debuggers).
I am a long time member of ACM (since 1977) and have watched research into debugger technology over those years. I have seen attempts at dramatically different conceptual models for debugging, but none seem to have been successful. In fact, even most graphical IDEs appear to have been failures. I suspect this is all because ultimately things boil down to mapping program behavior to the code and verifying results are those that are expected for that code. There is little room for new or different models and abstractions so long as code remains the same. And, given the modern evolution of coding, it will be quite some time before the term "Visual" means something more than being part of the Microsoft Visual Studio suite while continuing to use good old text based software language(s).
Real Programmers don't use symbolic debuggers. They use hex/octal crash dumps.
While it does make sense to be thoughtful and produce your best attempt at working code on the first try, clearly this is an imperfect process (for why, see a recent article in IEEE Software... software development is Knowledge Acquisition, known to be a rather imperfect activity). Thus, choosing not to use the most efficient techniques available means you choose to take more time than necessary. You really should not need to handicap yourself by eliminating a productive and useful tool to have the discipline to do the development of the software properly.
This approach of getting your foot in the door via something other than permanent employement USED to be a great way to get started. I just don't believe it works anymore. Companies have cut their use of contractors and consultants even more than their permanent staff. They simply look at the numbers and see they pay their existing staff less than contract and consulting houses. So, while it seems they have gone crazy and laid off all of their own talent, in many cases they have been even more aggressive in reducing these temporary expenses.
I have been in the Software industry for well over 24 years. There have been many bad times in the economy over that time, yet this is the only time that I've seen this industry in a full out depression. I can not guess where it is going, nor when it will start getting better.
In the past, as the economy turned from bad to worse, software folks actually did quite well. This was because bad economic times meant having to do more with what one already had. Since people saw hardware as the asset a company already had, they would invest in major software improvements to make better and more complete usage of that hardware.
Today we have hardware so cheap that we have the equivalent of a supercomputer sitting on desks of those recently laid off as that hardware cost much less than the people using it or the people programming it. Companies that are hurting badly buy more hardware in the hopes of reducing the number of people they employ.
The whole world seems turned upside down and who knows what it will look like when we come out the other end.
I do believe that things have to get better for the following reasons:
1 - Businesses need to change and evolve to grow. Even businesses in trouble work at change as that is the way they will dig out of the red ink.
2 - Business processes are increasingly integrated with Software. Thus, to change the operations of a business requires change to software.
Right now companies are postponing change, trying to hang tight with what they have right now. That can not last forever. When companies decide they have to move forward, then software development will have to pick up to support that change. And that means a better market for software professionals.
I do agree that companies are in need a good talent. But, I don't believe they understand how to go about hiring that talent. Thus, when good talent is out on the market, they are having a terrible time as the companies that they can most help, and thus are the best potential employers, are so overwhelmed by the hoards of applications and their lack of understanding of how to hire the people they need, that many good people are going unemployed.
I totally agree that networking is the only way to get hired right now. But, even if you are a great engineer and do a great job of networking it can be hard to find that company that not only needs your talents, but also can figure out how to find and hire you.
Your claim that there is no unemployment problem, only a problem of potential employees choosing not to accept a low-enough wage to become employed. This might possibly be true in a pure market setting, but we are not dealing with such a pure situation. There are a multitude of additional factors on both sides of the potential employment agreement.
Let me mention but a couple of these factors.
A potential employer considers far more than the cost of the potential employee when making a hiring decision. Does the employer have more work than the current staff can complete? Would the additional work that more staff would complete increase revenues, in other words pay for themselves. There are plenty of companies where the answers to these questions are no, and thus there is no opportunity at any price.
Potential employers when considering a candidate also pass judgement on whether that potential employee will be a happy contributing member of the company. Frequently an extremely qualified candidate willing to take a major pay cut will loose out to the less qualified candidate. The issue here is that it is assumed that the more experienced candidate is far too qualified to truly be content with the position, thus even though they may indicate they are willing to take on that position at the offered (extremely low) pay rate, the potential employer will choose the less qualified candidate on the assumption that that individual will be happier and thus a better contributor.
The are plenty of other considerations that come into play in a potential hiring situation. These, though, illustrate that a great engineer with excellent credentials willing to work for very low pay may still be unable to secure a job.
Slashdot Mirror
I have been involved in software reuse since the mid-1980s and possibly even earlier. There has been lots of energy expended on the problem of making existing implementations extensible, one of the strengths of OO technology, though not requiring OO. The big piece that has always been missing has been a major concerted effort focused on facilitation matching a developer's needs with existing software.
... I have long felt that CASE tools, yup those tools that are totally out of vogue right now, would be of greatest value if they had a dual function. Their primary function would continue to be as a means of describing architecture, design, or code, but a secondary function would be to, in the background, perform a continuous search of existing work looking for matches. I have never seen a tool that does this, yet this seems a tremendously valuable function.
There are many mechanisms that can assist such as:
1 - technical reviews. When these happen, you get a number of co-workers together to review your work. Not only does this assist in ensuring that direct work (architecture, design, code) is correct, but it also provides an opportunity for all those involved in the review to search their knowledge of pre-existing "parts", be they architectural, design, or actual code, and to suggest you investigate them. Of course, if you're like me, then actual review meetings where a number of people sit down and examine your work just do not happen any more. Thus this form of identifying existing work that can be reused no longer functions.
2 - CASE tools
3 - personal memory - only works for those items you already are familiar with, which frequently gets voided when changing jobs.
4 - institutional memory - this is similar to the technical review mechanism, yet is less well defined. The real question here is HOW does an individual tap into an institutional memory? Documentation search? This is far less than perfect even if all work was well documented. Code search? Even worse at turning up matches to needs.
So... the bottom line is that it truly is VERY difficult to match up needs of a software development effort with the existing software that is available.
Once case in point... I worked on a very large project for an FAA (Federal Aviation Administration) contract. One mechanism I needed was a circular buffer/queue. These seem very straight forward to implement, and an obvious place to use an existing piece of design/code. Well, even after extensive search and review I could not find such a part and built my own. Later, I discovered there were at least six independent implementations of a circular buffer/queue in this single project team. All of them were general enough to meet the other implementation's needs, yet somehow none of us knew of the others' overlapping work. If we couldn't coordinate the reuse of these six independent efforts (and that means we all built the same basic algorithm, found the same set of bugs... and yes, using our code management tool I was able to see the same bugs being fixed in each implementation... and thus a total and unnecessary duplication of effort), how in the world will we ever solve the problem of reusing work outside the single project team, or outside a company?
There are some examples of wild success with reuse... though they seem to me to be more success though definition. All of those shell scripts that are built from individual command line tools are examples of reuse, where each command line tool represents a unit of software available for reuse. But, I think we all think of reuse more at the code module level... a function, or class, or small library. And it is at this level that I think we fail miserably, and it is my contention that we fail because we can't easily find the candidates for reuse.
I think the larger problem is that this administration tends to take a "damn the torpedoes, full speed ahead" approach. If they think something is the right thing to do, they ignore any voices to the contrary and just forge ahead. This is exactly how you end up with problems like Chernobyl (read the book, its in there) and the two space shuttle crashes. As tragic as the shuttle crashes are, they do not have a direct long term impact on world health and safety. That is not the case when it comes to Nuclear power. It is scary indeed to think of what these folks might do to pursue their ideological views without letting a few inconvenient facts get in their way.
To tie into another posting... we may just get that operational test where they, with a fully operating core they dump the coolant. If things work as they are engineered and as the design intends, sure, things will likely be pretty much safe. BUT, what if things go wrong. We've just created our own disaster without any means of undoing it.
Why don't we throw a few live ICBMs at, say, a couple of our own cities to test SDI? Who says you can't test it?
I suspect a lot of people were not around in the pre-FSF/GPL days. The various Universities with computers would share their software with each other, and pretty much anyone else who had a computer.
As soon as two machines could exchange files via UUCP, people and organizations were sharing software for no charge. There were even cases before any kind of communications based transfer were possible where, if you supplied the tape and shipping, you could get a vast array of software for that day.
Of course, back then even the software you paid the big-bucks for was distributed in source code. I was involved with one of the big players of the time when everyone realized there was a workable alternative to shipping source code to every customer and asking them to compile and install every software product. Customers would write their own patches and utilities based on the insights into commercial product code offered by the source distributions. I should add, though, that given the huge burden of building products (a large percentage of even a large mainframe's disk space was consumed for many hours building just one of the major products), customers were right at the head of the line insisting on binary software distribution. It also made support MUCH simpler since you didn't have to worry about what fun new features a specific customer had added to their copy of the product.
Anyway, to get back to the my main point. GPL came along well after the tradition of free software had been established. I do not see it as essential to maintaining the free sharing of software. There ARE forces at work that will reduce this practice, but I don't believe GPL hinders those efforts, they being software patent madness and the US Department of "Homeland Security". (should I add that even in the days of the Soviet Union, Soviet computer people were both contributing and consuming free software. I am sure GPL had no impact what so ever there)
It should go without saying, but IBM OSes have been very expensive since the dawn of time (well, the start of the computer industry). The pricing has always be linked to the hardware costs, and those have not changed much in the big iron mainframe world (well, you get a great deal more power for the same big dollars now, but no small dollar machines). I do think the suggestion of some form of "hobbiest" license makes a lot of sense, particularly since with the tremendous power of today's PC hardware it would not be unreasonable for someone to implement a full emulation of the mainframe on it for training, development, and test purposes.
I really have to laugh at placing Sun and IBM in the same catagory. No way. Sun, even its biggest systems, is still a workstation and server platform while IBM's mainframes are BIG iron. I do wonder where the IBM mainframe clones have gone? Amdahl anyone?
While I have no particular animosity toward Intel, other than it is important for there always to be competition to push them, I do not think they need to be let off the hook. Itanium has been around a very long time. You may think of it as new technology, but that is more because of the lack of acceptance in the marketplace, not because it has only recently been released. What was happening all of these years since Itanium was initially launched?
Additionally, while the Itanium instruction set takes a different approach to those of Intel's competitors, they are not the only company introducing new CPUs. I do not remember such problems when other 64bit CPUs with their own, new, unique instruction sets were launched by Digital, HP, IBM or Sun to name just a few. These days, the competitive landscape has been radically reduced. Digital no longer exists and its Alpha architecture is owned by Intel. HP, while it still owns its PA-RISC architecture, is trying to migrate its customers to Itanium, though it is hard to say what will really happen to PA-RISC since no one seems anxious to adopt Itanium. IBM also has picked up Itanium, so who knows what will happen to their RISC architecture? That leaves Sun, and while SPARC has always been the weaker of the RISC architectures, it seems to be the primary remaining competitor to Itanium and Intel. Of course, who knows how much longer Sun will survive as an independent company? Maybe they are the next to be gobbled up by IBM or HP, both already commited to Itanium, so what happens to SPARC?
Finally, it is hard to say exactly where AMD fits in all of this. Its 32-bit line provides excellent competition to Intel's 32-bit Pentium family (now at P4), and the AMD 64-bit architecture looks like a nice increment beyond the now very old x86 32-bit architecture. But, in terms of major pressure on future CPU architectures? I just don't know where that competition will be coming from... Maybe China, Russia, Japan, India? Places not noted for their hi-tech prowess, but with lots of experience in fabrication and lots of affordable talent?
While I would like to be as optimistic as this post, I can't say there is a vedict yet. I think the fundimentals seem very clear. Almost every business depends on software to make it go, and that software needs to reflect each business business practices, making off-the-shelf software only a base element. It takes new software, or software modifications and/or integrations, to support changes in a business' practices. Thus, while right now there is a slump in the general economy, and a what I would say is a depression in the software industry, I do believe that things have to come back some soon as businesses can not afford not to innovate, and that will require supporting software.
.Net is slick and possibly the best of its class of tool, it is by no means the state of the art for software development. And even that state of the art that few get to use in "real life" leaves lots of room for innovation and improvement. The challenge, as I see it, is that entry into the field is getting quite expensive, both due to the complexity required of an inovative product and non-technical factors. This challenge was overcome in the past in many other fields and at some point it will be overcome in software (at which time all of us will end up having to learn yet another set of very new skills that will make object orientation, c++, Java/c#, and IDEs seem like the dark ages). It is not possible, though, to predict when this is likely to occur.
There is lots of room for improvement in how software is developed. It would seem that the state of the art would be a IDE, with Visual Studio the model that everyone is shooting for. But, while Visual Studio
I do not believe we will ever see the software industry return to the way it was before the current crisis. In fact, I suspect it will resurface with a new look. But, I just as strongly believe that there will be room for the solid software professional with a deep knowledge of software while being literate and articulate, thus able to communicate well with the business side of the operation.
ACM Communications, the primary publication of the Association for Computing Machinery, the Computer Science professional society, contains a special feature on Digital Rights Management. I have not finished reading all of the articles in this section, but my take thus far is that ACM US has taken an active role in attempting to provide technological guidance and advise to Congress on this issue and with respect to both proposed legislation and court cases testing laws in this area. Yet, the pressure that has been brought to bear by the big copyright holder interests thus far have far outweighed those of the technologists and their commercial interests.
Additionally, there have been a continuing series of articles in this publication as well as others produced by ACM providing additional analysis of the topic along with recommendations for action.
Give it a good read... its both encouraging that such comprehensive analysis is in fact being brought to the highest levels of government, and discouraging in that thus far it seems to have made little difference.
Finally, this might be a good reason to join ACM to add to the strength of their(our) voice.
And not to put too fine a point on it, I DID provide for an out in my initial posting. There ARE cases when one needs to optimize the heck out of the code, to fit time, space, or both. The number of such cases has been decreasing at a very rapid pace. Even many embedded systems now have lots of CPU time and memory (both RAM & persistent) at their disposal. Just check out recent issues of ACM Communications, IEEE Software, & Dr. Dobbs.
Finally, while 5% of the total installed base seems a significant fraction, this isn't always the case. For the little guy, even getting 1% is hitting it big. On the other hand, for the big vendor, they may choose to give up on that 5% in exchange for less expense in their software development.
Long ago the developer/hardware equation was changed. Originally, hardware was far more expensive than the people developing the software. That was when investing lots of energy into hand-optimizing was the proper tradeoff.
Years ago, before 1GHz was considered a short term possibility, hardware costs had decreased and software costs had increased, to the point where it is the DEVELOPER who is the most expensive piece of the equation. Thus, we now are at the point where, with the exception of a few very specialized segments, we do whatever it takes to optimize the developer time in building software. That is not to say that developers can be careless and wasteful. But that developers should not waste time optimizing code. IFF performance is an issue, THEN one takes measurements and optimizes critical areas consuming the majority of time. Beyond that, it just isn't worth it. Today's 3GHz machines with GB of RAM only reinforce that this is the only appropriate approach to software development.
I did my time as a third shift operator as a student summer employee at IBM's TJ Watson Research Lab. This was the summer between my freshman and sophomore years of college, so I had only had two semesters of formal computer work, and a year of work on APL and BASIC before that.
... a jet plane landing in the machine room, oh, I mean a full head crash on a 3350 disk drive, the washing machine size disks with removable platter assemblies of many (was it 12) platters, and using the very first generation Winchester heads. ... a minor flood caused by a huricane and a clogged stairwell drain. Nothing like water hitting high voltage power in a transforming closest for some excitment. Though, once we were done trying to defend the machine room with mops, we just sat around until power was restored and then saw who could seek out all of the devices that needed powering-up (it was a HUGE machine room and there was no map) and who got THEIR machine up first.
I spent that summer pulling listings off the line printers (remember them?), cutting graphics output printed on thermal paper that was scattered around the floor by its printer (that was state of the art graphics then), and hanging tapes (primarily for the nightly backups). In my idle time, and there was LOTS of it, I read the manuals on the systems from the company library, and then showed the full time permanent operation staff how to customize their environment and write what we now call shell scripts to automate their environment.
Yup, a college sophomore that was very much wet behind the ears was showing the veterans of many years how to make their jobs easier simply by programming their function keys and writing very short shell scripts. That speaks pretty loudly of the type of people who were on the operations staff. And this was at IBM's front line research lab, and they were very clearly hand-picked from everyone available in the general area which had many very large IBM facilities.
Once in a while we had to deal with an emergency...
The normal staff was three third shift operators, including me. We were running three of what were then the biggest monsters IBM made, and possibly the biggest in the world at that time... three 370 model 168 mainframes, with what I believe was the maximum available CPU, Memory, and I/O subsystems.
It was a very interesting, if sleepless, summer. The insights into who the operators were, and thus who the users of many system level tools are, have lasted a career.
By the way, this was the summer of 1976.
(did someone say they wanted to hear some of that great IBM company folklore I heard that summer? I don't think that non-disclosure for that job prevents me from sharing anything at this late time)
You make light of this machine's accomplishments, but in its time, it was truly a wonderful machine. Just the thought that there was this computer and you could actually take it with you on a trip so you had it available where ever you were going was just fantastic. While there were a few that pre-dated it (there was a similarly sized APL machine, the IBM APL 5100), this was truly a revolutionary machine. It was such a shame when they announced the Osborne 2 prematurely and EVERYONE decided to stop buying the current model and buy the new one when it came out. But, out of this disaster people realized just how powerful the idea of a computer you could haul around really was.
I have used just about all of the various Unix flavors and platforms at one time or another as well as RedHat Linux.
OS:
Solaris - Tends to be clean, fast, and buggy. The most patches I've ever applied on a non-Microsoft platform was Solaris.
HP-UX - a Solaris clone intended to make porting from Solaris very easy. Generally, I found it to be a very good Solaris with many fewer bugs.
AIX - Very clean and very fast. A terribly literal SysV Unix, thus frequently rather spartan and basic. But, it cranks things out quickly and reliably.
Tru64 - Well, almost not worth mentioning since Digital was gobbled up by Compaq which was gobbled up by HP. If not for the fact that it was definitely at the top of my list for high quality Unix platforms. Stay away from DECnet (there for VMS sites) and DECWindows (ditto) and its a pretty nifty platform. The one downside... Hate that digital uses a software licensing system for pretty much all of its products. Now that HP owns what once was Digital, it can be pretty painful to buy a license.
Linux - Requires a lot more hands-on time to get things up and running properly, a bit unpredictable what quality of component one will encounter. For example, its VM system just does not compare to that in AIX or Tru64. If you don't have enough RAM to fit everything in real memory, paging performance just doesn't approach the competition.
Hardware:
SPARC: Always bringing up the rear in price/performance, but marketed extremely well.
HP-PA/PA-RISC: Far ahead of SPARC in price/performance, but much later to see ports of your favorite software. Not going to be around much more as HP is hot to drop it for IA64.
Alpha: Also far ahead of SPARC, usually playing leapfrog with HP-PA/PA-RISC & PowerPC. Not going to be around much more, as HP is hot to drop it and their own HP-PA/PA-RISC in favor of IA64.
PowerPC: Another very strong RISC platform, also with a questionable future due to IA64 plans.
The bottom line: Sun, while not necessarily the best Hardware or Software vendor, has provided a very important competitive presence in the industry. As they head down the path to low cost Intel platform solutions with Linux software, from where is the competition and inovation in Hardware and software going to come? It sure doesn't look like Sun will continue to be a significant player.
It was a great language, originally created to DESCRIBE and design computer hardware architectures, but a few of the wizards at IBM's Watson Lab decided to implement it as a programming language. Still the only language I know of that can multiply a pair of matrices with a single NATIVE operator! No user coding required.
Of course, with all of this power APL also invented the write-only program and the one-liner, both predecessors to the obfuscated C contest... and I must say that those using C and C++ can only dream of approaching the unreadability that was APL. Then again, only APL used its own character set... it used single symbols derived from mathematics as its operators. No constructing multi-character composite operators in that language... just custom terminals, or... a special type-ball for your selectric based 2741 terminal (there's a trip down memory lane).
To return to my point, though.
APL died for a number of reasons, but probably the biggest was how totally undigestible its code became with very little effort. I wrote some fairly simple programs, relatively well written for APL, avoiding the tradition of making everything a one-liner, actually including comments... you know those things written in intelligible English that explains what a program does... oh, yeah, I haven't seen any of those in years in my C/C++ shops... and yet as little as a few months after I wrote those beautiful, elegent APL programs I had no clue how they worked and could only change them by re-writing them. A very good reason for a language to die.
The shame of it all is the APL was probably the best language ever invented for solving hard-core math and arithmetic problems, yet it died before it could even run on its natural platform, the super-pipelined vector processors such as the Cray-1 and its relatives.
My favorite example... I was working on a large C++ software system. There was a major piece of business logic that included a sort of a very large list. The list was using the double-linked list C++ template class and was taking a very long time to execute. I was told to change the implementation to PERL since PERL can sort so much faster than C++. Instead, I investigated a bit, found the problem (the use of the linked list template class) and switched to a more appropriate structure (I used one of the hash based templates). My management was shocked when the C++ code was suddenly executing 100s of times faster and I hadn't converted the whole thing to PERL. Just for fun, we converted a piece of it to PERL so we could try the same sort. The PERL sort took two to three times the time that the C++ code did once I switched to a more appropriate class.
My basic rationale was... PERL has to eventually boil down to C++ logic, so why would PERL be FASTER? An interpreted language working with abstract datatypes vs. a compiled language that uses somewhat more concrete types... and as you move more toward C, you get to very basic data types that map directly to hardware.
There certainly are times when scripting is the right solution. But, compiled languages do have their strengths as well. One needs to keep these in mind when choosing, rather than just blindly thrash from one language to the next.
Note that with a secure "licensing facility", software vendors can and will have their software expire. Thus, you will purchase a license to use a software product for a specific period of time and it will not be usable after that period of time without additional expenditure to purchase additional licensing.
This has very interesting unintended consequences that I'm seeing today. Here's the example... I am working on something that requires me to use some old legacy operating systems. One of those is the old Digital Tru64 Unix product. So, off we go and buy some hardware from eBay, and then the required Tru64 software. That would be enough, right. Wrong. Digital implemented within all of their operating systems a licensing mechanism that requires one to obtain a license seperate from the OS media. That was fine when Digital was its own company and could justify staffing an office for issuing these licenses (assuming one buys into purchasing those licenses). But, today Digital does not exist. Even worse, Compaq does not exist, having been gobbled up by HP. HP probably had no idea when they purchased Compaq that there was this need for an office that issues licenses for the software that Digital had produced. So, today, while HP gladly will accept your money to purchase these licenses, it takes a significant amount of time as there is ONLY ONE PERSON staffing that office. What happens when that person finds a new job (not likely in today's software economy but what about tomorrow's)? Or retires? Who will issue the licenses then? Or, will it just become impossible to issue these licenses, thus rendering it impossible to use those products?
Elizabeth Haydon is a delightful author who has come on the scene only recently. Her initial book Rhapsody was absolutely stunning in its writing. She has now added three (or is it four now) books to her writing and they are all of the same high quality.
The aspects I most enjoy are her wonderfully flowing writing, so strong that she is one of the few to follow in the footsteps of Ursula Le Guin. I have a very hard time not trying to read her books in a single sitting, which would be a bad thing to do given their length and my reading speed (we're talking about a couple days minimum to read straight through to the end).
I had a great old Canon printer, but have been having big problems finding ink cartidges that fit it (even via the web). Thus, when I finally needed a new printer I went HP because I've NEVER had a problem finding the proper cartridge for any of their printers. I'll seriously consider vendors other than HP for my next printer when I can be sure I won't run into the availability problem again.
I find I spend a LOT of my time debugging or working around bugs in the tools. Nothing is more aggrevating and frustrating than getting exactly where you need to get to find a bug in a program only to have the debugger itself crash with a SEGV. Between that, and the inability of many of these tools to work easily in the modern dynamic library and distributed computing environments, life is far more difficult than it should be using the existing model.
Thus, I want a debugger that absolutely does not crash. I want it to let me set breakpoints in any routine, even if it resides in a dynamically loaded library that is not currently/yet loaded. I want to be able to view ALL of my data in their abstract datatype form without having to look up their definitions and then manually guide the debugger through doing the job. And, I want a debugger that always honors my commands and does not lock up (which I find far too often with the GUIs layered on top of command line debuggers).
I am a long time member of ACM (since 1977) and have watched research into debugger technology over those years. I have seen attempts at dramatically different conceptual models for debugging, but none seem to have been successful. In fact, even most graphical IDEs appear to have been failures. I suspect this is all because ultimately things boil down to mapping program behavior to the code and verifying results are those that are expected for that code. There is little room for new or different models and abstractions so long as code remains the same. And, given the modern evolution of coding, it will be quite some time before the term "Visual" means something more than being part of the Microsoft Visual Studio suite while continuing to use good old text based software language(s).
Real Programmers don't use symbolic debuggers. They use hex/octal crash dumps.
While it does make sense to be thoughtful and produce your best attempt at working code on the first try, clearly this is an imperfect process (for why, see a recent article in IEEE Software... software development is Knowledge Acquisition, known to be a rather imperfect activity). Thus, choosing not to use the most efficient techniques available means you choose to take more time than necessary. You really should not need to handicap yourself by eliminating a productive and useful tool to have the discipline to do the development of the software properly.
This approach of getting your foot in the door via something other than permanent employement USED to be a great way to get started. I just don't believe it works anymore. Companies have cut their use of contractors and consultants even more than their permanent staff. They simply look at the numbers and see they pay their existing staff less than contract and consulting houses. So, while it seems they have gone crazy and laid off all of their own talent, in many cases they have been even more aggressive in reducing these temporary expenses.
I have been in the Software industry for well over 24 years. There have been many bad times in the economy over that time, yet this is the only time that I've seen this industry in a full out depression. I can not guess where it is going, nor when it will start getting better.
In the past, as the economy turned from bad to worse, software folks actually did quite well. This was because bad economic times meant having to do more with what one already had. Since people saw hardware as the asset a company already had, they would invest in major software improvements to make better and more complete usage of that hardware.
Today we have hardware so cheap that we have the equivalent of a supercomputer sitting on desks of those recently laid off as that hardware cost much less than the people using it or the people programming it. Companies that are hurting badly buy more hardware in the hopes of reducing the number of people they employ.
The whole world seems turned upside down and who knows what it will look like when we come out the other end.
I do believe that things have to get better for the following reasons:
1 - Businesses need to change and evolve to grow. Even businesses in trouble work at change as that is the way they will dig out of the red ink.
2 - Business processes are increasingly integrated with Software. Thus, to change the operations of a business requires change to software.
Right now companies are postponing change, trying to hang tight with what they have right now. That can not last forever. When companies decide they have to move forward, then software development will have to pick up to support that change. And that means a better market for software professionals.
I do agree that companies are in need a good talent. But, I don't believe they understand how to go about hiring that talent. Thus, when good talent is out on the market, they are having a terrible time as the companies that they can most help, and thus are the best potential employers, are so overwhelmed by the hoards of applications and their lack of understanding of how to hire the people they need, that many good people are going unemployed.
I totally agree that networking is the only way to get hired right now. But, even if you are a great engineer and do a great job of networking it can be hard to find that company that not only needs your talents, but also can figure out how to find and hire you.
Your claim that there is no unemployment problem, only a problem of potential employees choosing not to accept a low-enough wage to become employed. This might possibly be true in a pure market setting, but we are not dealing with such a pure situation. There are a multitude of additional factors on both sides of the potential employment agreement.
Let me mention but a couple of these factors.
A potential employer considers far more than the cost of the potential employee when making a hiring decision. Does the employer have more work than the current staff can complete? Would the additional work that more staff would complete increase revenues, in other words pay for themselves. There are plenty of companies where the answers to these questions are no, and thus there is no opportunity at any price.
Potential employers when considering a candidate also pass judgement on whether that potential employee will be a happy contributing member of the company. Frequently an extremely qualified candidate willing to take a major pay cut will loose out to the less qualified candidate. The issue here is that it is assumed that the more experienced candidate is far too qualified to truly be content with the position, thus even though they may indicate they are willing to take on that position at the offered (extremely low) pay rate, the potential employer will choose the less qualified candidate on the assumption that that individual will be happier and thus a better contributor.
The are plenty of other considerations that come into play in a potential hiring situation. These, though, illustrate that a great engineer with excellent credentials willing to work for very low pay may still be unable to secure a job.
Just imagine being those engineers... no matter what you do you still end up working for Microsoft (assuming the do the deal).