Slashdot Mirror

...in theory. In actual practice, prices are random.

I have not seen any open source projects that you can use as a platform for building a trading system.

I have built an intraday (within one day) trading system in Java. I'm afraid that this system is not open source either. This system runs one or more models that look for intraday trading signals. The Java software submits buy and sell orders. It is multi-threaded and runs one thread per stock. I have been very happy with the software performance. A long running "server" like this seems to benefit from Sun's HotSpot compiler. The system is web services based (e.g., it runs on Tomcat).

I used Interactive Brokers for my market data and order infrastructure. I was concerned about the quality of the Interactive Brokers tick data (the trade by trade data). Interactive Brokers consolidates their tick data feed so you get a consolidated tick about ever 250 msec. For my system this has been adequate. If you want to run on Linux or use Java there are few inexpensive options for real time data feeds. Information may way to be free, but market data is expensive.

I have some web pages on the alternatives that I explored as platforms for a Java/Linux based trading system. These notes can be found on my web page Software for Constructing a Market Trading System

I don't have any moderation points to add to Helevius' karma, but I can send my things for posting an informative article.

Its pretty clear to anyone paying attention that the fact that software vendors like Microsoft pay no price for security failures in their products means that they don't have much incentive to fix them (see Why are there still e-mail viruses? ). So I was inclined to agree with what seemed to be the theme of Geekonomics. But from looking at the quotes above and the authors blog, it's pretty clear that the author doesn't have much of a clue. He tries to excuse this by stating that, well, he's writing for executives, so he can't go into technical detail. This is really just an excuse for not putting the effort in to completely explain the problem. I am in the middle of reading David Leavitt's excellent novel The Indian Clerk , about the Indian number theorist Ramanujan. Leavitt manages to clearly portray pre-World War I England, number theory and academic mathematics. The author of Geekonomics really comes off as yet another consultant whose writing a book to get those hourly fees from those very same clueless executives. I am glad that Helevius has saved me from buying this turkey of a book.

You certainly have a right to your opinion. It's great that there's a match between employers who do this sort of thing and people who like that sort of interview. I wrote a web page, Calculating Permutations and Job Interview Questions on this topic. What do I suggest for interview questions instead? How about detailed questions about the projects that the applicant has worked on, what exactly they did on these projects, how the solved the problems they encountered, how the software they wrote was structured, how they worked with other members of the team, how they dealt with conflict...

I will note, just in passing, that Soviet Russia has not existed for some time so the present tense of the sentence below seems misplaced.

In Soviet Russia, the government controls the commerce.

I'm assuming you're referring to TOR, not TOR (though he's pretty cool too).

I also think Freenet and Darknet type networks will play increasingly important roles in the inexorable globalization of free speech. What's needed is a way to create secure, historied pseudonyms that are peer validated, verifiable by signature, but incapable of being route-traced. If done right, such a system could potentially put freedom of speech and trade beyond the reach of government suppression.

taniwha, I am quite certain I reviewed several early diagrams (which naturally I cannot find now), and I am quite certain that originally, Transmeta was talking about clockless processing. The reference clearly indicates that I'm not imagining things, but it may have been nothing more than hype. It would appear as if Transmeta went a different direction, however. Maybe Transmeta knew about the work at Intel (which came first?) and drummed it up, or maybe the efforts at Transmeta were not enough, and that engineer left for Intel to try it somewhere else? We really need an inside opinion here. Anyone know someone at Transmeta who would care to comment?

I read the New York Times report on the ACM study this morning. All I could think of was that old joke about catching your significant other in bed with someone else. "Really", they tell you, "it's not what it looks like. Who are you going to believe, me or your lying eyes".

My resume is published on my web page. So one way I judge hiring is by recruiter calls (obviously this lacks something in the scientific rigor catagory). The other way I judge this issue is by press reports, which I've collected in an annotated bibliography that is at the end of my web page An Economics Question. Many of these press accounts describe the experiences of other engineers in today's job market. There are a few conclusions that I draw from the current engineering employment environment:

1. There are still interesting jobs that pay decently out there. However, pay and stock options are definitely not what they were, even in 1992, much less 1999.

2. Job instability is way up. The days when you could get another engineering job relatively easily if you had a good background are over. This greatly increases the risk of working for a start-up, since you could experience many months of unemployment if the start-up fails and you're out of a job. The problem here is that most start-ups do not compensate you for this increased risk. They pretty much give you the same pay and stock options that you got a decade ago. But in this new environment you stand the risk of losing your savings or even your house because of a long period of unemployment.

3. Job security is also way down. There remains a big pool of engineers looking for work and employers definitely have the attitude that they can always hire another engineer, so you're a disposable, interchangable commodity. With many software development jobs there is always the threat that your project will be "offshored" or that when you complete it, maintenance will be offshored and you'll be out of work.

4. While hiring is at best tepid in the United States and Europe, hiring is booming in India. Employment demand for Indian engineers who graduate from schools with education comparable to schools in the US or Europe has entirely outstripped demand. The good news is that this is forcing salaries in India up. But my lying eyes tell me that what is fueling the demand in the Indian job market are "first world" jobs that are being outsourced.

With the ACM report working engineers are faced with that question of "who are you going to believe, the ACM or your lying eyes". My lying eyes tell me that the story told by the ACM does not reflect the employment experience of the ACM membership.

I read the New York Times report on the ACM study this morning. All I could think of was that old joke about catching your significant other in bed with someone else. "Really", they tell you, "it's not what it looks like. Who are you going to believe, me or your lying eyes".

My resume is published on my web page. So one way I judge hiring is by recruiter calls (obviously this lacks something in the scientific rigor catagory). The other way I judge this issue is by press reports, which I've collected in an annotated bibliography that is at the end of my web page An Economics Question. Many of these press accounts describe the experiences of other engineers in today's job market. There are a few conclusions that I draw from the current engineering employment environment:

1. There are still interesting jobs that pay decently out there. However, pay and stock options are definitely not what they were, even in 1992, much less 1999.

2. Job instability is way up. The days when you could get another engineering job relatively easily if you had a good background are over. This greatly increases the risk of working for a start-up, since you could experience many months of unemployment if the start-up fails and you're out of a job. The problem here is that most start-ups do not compensate you for this increased risk. They pretty much give you the same pay and stock options that you got a decade ago. But in this new environment you stand the risk of losing your savings or even your house because of a long period of unemployment.

3. Job security is also way down. There remains a big pool of engineers looking for work and employers definitely have the attitude that they can always hire another engineer, so you're a disposable, interchangable commodity. With many software development jobs there is always the threat that your project will be "offshored" or that when you complete it, maintenance will be offshored and you'll be out of work.

4. While hiring is at best tepid in the United States and Europe, hiring is booming in India. Employment demand for Indian engineers who graduate from schools with education comparable to schools in the US or Europe has entirely outstripped demand. The good news is that this is forcing salaries in India up. But my lying eyes tell me that what is fueling the demand in the Indian job market are "first world" jobs that are being outsourced.

With the ACM report working engineers are faced with that question of "who are you going to believe, the ACM or your lying eyes". My lying eyes tell me that the story told by the ACM does not reflect the employment experience of the ACM membership.

I agree, but let's not forget about Cormack and the Doom series that uses OpenGL!

That said, this site has explainations and code: http://www.bearcave.com/misl/misl_tech/wavelets/

First of all, as much as I apparently seem intelligent enough, I think speculation is very difficult in this direction exactly because the "geek masses" are on average intelligent and as such unpredictable. But since it's an interesting topic, I'll give it a try ;-)

Concerning "disenchanted enough [...] to take matters into their own hands": Yes, definitely, and I think this has already happened when it comes to the purely virtual domain. Slashdot readers generally seem to think that someone using bad, insecure software is at the very least partly to blame if something bad happens to his/her system as a result. I consider this "taking matters into own hands" as well since it's opposed to the general opinion that when some poor slob gets his unprotected windows system compromised it's only the bad evil hackers to blame. The fact that the legal system still considers itself to have the final say even here, doesn't change the techies' opinion on this.

As to "next civil war": I think physical violence is not something a supposedly intelligent group of the population is likely to resort to as long as there's any alternative at all. And I don't think this situation is likely to occur, which coincides nicely with the next thing you mention:

The useful "weapons" of the tech population are twofold, IMHO.

Firstly, the power to severely disrupt worldwide communication. I don't think they/we are likely to resort even to malicious attacks against important networks, but with the brains behind the internet removed, I don't think it would be long before the internet would degrade to a useless chaos (if you think the internet is sufficiently established for the suits & beancounters to be able to sustain it, consider, as a random example, what would happen if the Apache foundation decided to no longer fix any problems and/or make improvements to facilitate new technologies). While this does not directly stop rediculous lawsuits, I do think that situations where an example is being set will at some point trigger protests that will be heard by the general public and ultimately be acted upon by the not-so-technical, but sufficiently sensible part of the population.

Secondly, there is the ability of the tech people to take a significant part of their matters "underground". The idea of a "Darknet" has been mentioned in the media, combine this with using strong cryptography and even steganography and you're looking at a network that will be impossible to regulate and likely impossible to detect. It won't help situations like the one discussed in the article though, where the border between the virtual world and the physical world is concerned.

Ofcourse ultimately, only time will tell.

Sorry I don't have any moderator points, or I'd "mod" you up. You are exactly right. Without disposable income there is no one to buy all the stuff that the factories all over the world are churning out. Currently purchasing power is supported by credit. Money taken out of home equity, credit cards and the massive trade deficit. This will not continue forever (or even for that much longer).

I finished a essay on this in 2004 titled An Economics Question, which includes an extensive set of annotated, largely web accessible, references.

I would rather pay a few dollars more for something made in the US that provides US jobs that pay rock bottom prices and hollow out our economy. This would still allow competition within the US (and kill off the buggywhip makers) but would still provide a healthy local economy. People forget that the free trade religion in the US is certainly not shared by countries like China and Japan.

Is there something I don't know about my domain, bearcave.com? I don't understand the IRC reference. There was a bearcave.ORG, which was a site for hairy gay men and bearcave.NET, a site run by network savy "Bears". Some of these are discussed here (another one of those bearcave links).

Anyway, would you care to let me know if I've missed something? You can reach me at iank at bearcave.com. Thanks.

I've tried to work on projects where we developed new software as much as possible. However, taken all together I think I've probably spent many months of aggregate time reading other peoples code, modifying it, adding new features and fixing bugs. I hate doing this work, not only because of poorly designed software but because most software I've worked on is almost entirely undocumented (e.g., no comments and no external documentation).

Some posters have mentioned comments and documentation in this thread. But in my opinion, it has not been discussed in strong enough terms. For a long rant on this topic, see my essay Software and Documentation. I'll summarize some of the points I try to make in this essay here.

There were only a few people in this thread who wrote that they think that software source code is self documenting. This is a good sign. In the past this was a widely held view. Perhaps it is becoming less popular. Perhaps pigs are becoming airborne.

To those of you who wrote that your code is self documenting: you are simply lazy and unprofessional.

Sure I can understand what well written code does. But I cannot understand why it does what it does. I can show you three hundred lines of wavelet signal processing code. You can immediately understand what the code is doing. But you will have no clue what-so-ever about why it does what it does unless you understand the thinking behind the code. The same is true of any complex software.

Another problem is that even though I might understand what components do and perhaps even why they are doing what they do, getting an overall understanding of the architecture of a large application can be difficult and time consuming.

By not writing comments in your code and, for large applications, providing external documentation on the architecture, you are saving your time and effort, but forcing others to spend effort that you could have saved them.

Even if you have no consideration for anyone else who must maintain the application after you write it, comments will help you find bugs in your own code. Again and again I have found that when I explain my code in a comment I am forced to look over it again. In doing this I have found that the code does not do what I intended or that I left something out.

When it comes to documentation there is rarely enough. Even if you make an effort to document your code, when you are in the middle of it you will not document things that you find obvious. When you return later, these obvious issues will have been forgotten and are not obscure.

Documentation must be maintained along with the code. And it should be extended where it is incomplete or unclear. English (or what ever your language is): it's another kind of software.

Developers seem to fall into two camps: those who believe documentation is important and who do it and those who do not. I've managed software groups where I mentioned documentation in people's performace reviews and it still did not good. So far I have not been able to get someone who does not document to do it.

I have thought that perhaps one solution would be to hold code reviews where one of the major features that was looked at was understandability in the code. But I have not experimented enough to know if this will actually get people to document their software.

Literate programming not only involves writing well designed and well informed algorithms. It also includes writing documentation that explains these algorithms.

The parent post rambles around a bit but I take it that the major point is that the infrastructure in India is broken and that without infrastructure support you can't move into the modern world.

There is more to infrastructure than just phones, highways and clean water, although these may be the most important part. Infrastructure also includes a working legal system and something approaching "the rule of law".

And then there is the violence between Hindus and Muslims. And so on and so on...

I don't see India as being a superpower in the next century. I see China in this role.

Newfoundlands are wonderful dogs. In fact, at the risk of courting controversy, I'd say that Newfi's are the worlds best dog. But...

If you live with a Newfi you have to get used to drifts of hair piling up. Especially in the spring during that dreaded time when a Newfi "blows their coat".

So I imagine one of these little Roomba's doing its thing over the tile floor when it encounters a drift of Newfi hair. The Roomba would start to suck up the hair, fill up, get clogged and die. Newfi 1, Roomba 0

Outsourcing has been moving "up the food chain". From call centers selling products to help desk support. And from help desk support to software development, chip design, paralegal work and any other job that can be done on a computer.

One of the scary things about offshore outsourcing is that it appears to be eating into more and more high paying jobs. These are the jobs that were supposed to belong to first world "knowledge workers". After hollowing out our manufacturing and losing high paying manufacturing jobs, then losing "knowledge worker" jobs, the fear is that all that will be left are "service sector" jobs that pay poorly and have few benifits.

These fears should not be dismissed and they are certainly echoed in the slashdot community. But the issue is more complex than the pay differential between India and Silicon Valley.

People in the US are notoriously insular. We don't travel to foreign countries as much as people in Europe since the United States is huge. We tend to assume that people everywhere are either just like us or want to be just like us (just listen to they way G.W. Bush talks about Iraq).

When we look at India from afar we see a democracy. India has a British derived legal system, just like we do. Many Indians have excellent english language skills. Many of us have gone to school and worked with people from India. So there should be no problem moving jobs from the US to India where the job can be done for much less cost.

How simplistic this view is was brought home to me recently when I read Suketu Mehta's book Maximum City: Bombay Lost and Found

India is a huge country. Maximum City is about just one part of India, Bombay. I have no idea who different Bangalore is. The picture that Mehta paints of India is of a country whose legal system is broken on every level.

A broken legal system has a number of implications. One implication is that without a civil tort system, where a lawsuit can be successfully brought, there is nothing to stop someone from stealing your intellectual property. If you pay someone to develop software there is nothing to stop them from selling it to someone else. If you partner with someone there is nothing to stop them from appropriating shared intellectual property or the money that you put into the partnership.

The legal system is broken on a criminal level as well. Extortion exists more or less without check. Wealthy people must be careful to avoid displays of wealth, lest they be targeted for extortion. Wealthy Indians who do not pay are killed or their family members are kidnapped. There is no reason to assume that a foreigner in India will be immune.

The police in India practice torture and have squads that take part in extra-judicial killings. One chapter in Maximum City starts out with an account of a muslim baker being burned alive during one of the Hindu anti-muslim riots.

This is not Kansas, Dorothy.

Cost in the western world are much higher. But the legal system in the western world works well in comparison to the legal system in India or China. While there is corruption in the police and civil service in the United States it is not a way of life as it is in India or China.

The book Mr. China by Tim Clissold tells a story of how the investment group that Clissold worked for lost $400 million (US) on a number of Chinese joint ventures. After three years of struggle, Clissold, who spoke Mandarin Chinese fluently, was totally burned out.

China is changing at a dizzying rate. The conditions of a few years ago that Clissold described may no longer be the same. But even in China the "rule of law" is in the process of evolving. China is still the land of the "Red Princes" and those who go against the oligarchy can

They are using the Windows Media Rights Manager to encrypt their content. This has not been cracked yet.

(1) Yes it has. It has been broken in the past and WILL be broken in the future. Without Paladium-like hardware support to physically keep the private key away from the user, DRM is simply an impossible promise . It amazes me that otherwise intelligent people can take it seriously when looking at how it works. Even with hardware support it is sketchy, it is almost a given that the hardware scheme will be broken as well over time. There is no DRM scheme on earth that has not been broken yet. The entire concept is flawed.
And regardless of all of that, the analog hole makes it all pointless anyway.

(2) Then it is not p2p in any form. Windows Media Rights Manager works quite simply as I have described in my previous post. How do you release something on a p2p network and control it after n+1 steps? Setting up a system that distributes DRMed content from A to B is easy (see itms, napster2, etc). I have yet to see anyone solve the problem of passing DRM content from A to B to C unless you are using a single key to encrypt everything, in which case you lose individual control anyway.

Finkployd

It's human nature to respond to put the best possible light on a negative situation that doesn't appear to be changeable.

This may be somewhat OT, but I think it's a good example of this cognitive dissonance phenomenon: I am a social conservative (strongly support the right of an armed citizenry, believe abortion should be illegal during all 9 months, for example) who is not voting for either the Republican or Democrat presidential candidate. I simply can't see myself voting for someone who has proven himself as incompetent as Bush has, even though I actually agree with him on most of the issues I find important. (The Iraq war and the environment are exceptions.) I found it at turns amusing and exasperating so see how my conservative friends tried to defend Bush's "puzzled chimp" performance in the first debate: "It was 9PM Eastern time, and that's late at night for him," "I don't think he did that bad," "He's a plain-spoken man," etc. Imagine their reaction if things had been switched and Kerry had performed that dismally. There would have been a lot of gloating and pointing out that his fate was sealed.

Now, back onto the topic: Good luck with your theory that only programming grunt work is going to be offshored. Yeah, that's what we said about manufacturing some years back, maintaining that the real "brain work" will stay in the U.S. Not a chance.

Just take a look at what Google says about the topic. I found one of the first hits, "Offshore Outsourcing World" to be particularly interesting, and chilling. Ironically, the article talks about google itself.

I actually don't see any alternative to free trade, and firmly believe that capitalism is the only way to go (conservative there, again). But with the last barriers to global competition rapidly coming down, a re-distribution of wealth is in progress on a global scale. That means painful adjustments for those who have gotten used to having more of it than most of the world's people.

I am a registered patent agent, licensed to practice law in patent matters before the U.S Patent & Trademark Office. To get to where I'm now at, I've had to get a four-year technological degree, pass a really tough exam, and learn how to write by working under some experienced patent attorneys for that past five years or so. (Self-promotional but generally informative info here.)

So, does that mean my career is safe? See for yourself.

I went back and read my post. Perhaps I did not make point clearly.

I love computer science. I've done a lot in my career that I'm proud of (see my resume). I am fortunate to have a job at a government lab that will never be moved overseas. If you really love computer science then you should go into this field. And, as I noted, at least the CS geeks are not as gross as the Wall Street traders.

But the profession is not what it was five years ago. So go into it with your eyes open. This is the same advice that I'd give anyone, regardless of gender.

One thing that has changed in the last twenty years is that there are many more women in science and technology. I already mentioned Ingrid Daubechies, an applied mathematican who was responsible for developing advanced signal processing algorithms (the Daubechies Wavelets).

I recently went to a talk by a woman who is a lab head at MIT. She is working on processes to get biological viruses to create inorganic nanotubes. She has won a variety of scientific awards and is clearly on the fast track to being a major contributer in her field.

In high school it may be difficult to see that there are women like these out there. When you get to a University it will be different. So by all means, go for it. You can be one of these women. If you love computer science and applied math, this is a great field. But you will start your career in a very different world than I did. For most of my career there has been a shortage of software engineers. I have been able to go from company to company looking for the next cool project. These days are gone and they may never return.

I went back and read my post. Perhaps I did not make point clearly.

I love computer science. I've done a lot in my career that I'm proud of (see my resume). I am fortunate to have a job at a government lab that will never be moved overseas. If you really love computer science then you should go into this field. And, as I noted, at least the CS geeks are not as gross as the Wall Street traders.

But the profession is not what it was five years ago. So go into it with your eyes open. This is the same advice that I'd give anyone, regardless of gender.

One thing that has changed in the last twenty years is that there are many more women in science and technology. I already mentioned Ingrid Daubechies, an applied mathematican who was responsible for developing advanced signal processing algorithms (the Daubechies Wavelets).

I recently went to a talk by a woman who is a lab head at MIT. She is working on processes to get biological viruses to create inorganic nanotubes. She has won a variety of scientific awards and is clearly on the fast track to being a major contributer in her field.

In high school it may be difficult to see that there are women like these out there. When you get to a University it will be different. So by all means, go for it. You can be one of these women. If you love computer science and applied math, this is a great field. But you will start your career in a very different world than I did. For most of my career there has been a shortage of software engineers. I have been able to go from company to company looking for the next cool project. These days are gone and they may never return.

In the case of run once applications that are CPU intensive, like compilers, there is no question that Java can be slow (see my web page on why it is reasonable to consider compiling Java into native code)

I've been working on the design of a trade engine, which can support multiple trading applications. I'm an experienced C++ developer, but for the trade engine I'm planning on using Java (running on Linux and storing transactions in PostgreSQL).

One of the core issues in many financial applications, like a trade engine, is that they must record transactions in a database. Once you start inserting into a database it is likely that the database will become the bottleneck (unless you are using something like an in memory database (e.g., like the one from TimesTen). So the performance advantage that C++ can deliver over Java does not seem to apply for database applications. Sometimes database operations seem so slow that I fell like programming in smoke signals would make no difference.

The class libraries available for Java are more extensive than any environment I've worked in. In C++ you could only match Java's library by purchasing all of Rogue Wave's libraries. And even then, Rogue Wave can be rather obscure. In addition to the huge class library, Java gives you garbage collection, which makes development faster.

Java also provides many advantages to enterprise applications. For example, you can use an application server like Resin to start up your services and to support remote calls (via SOAP). You can also have dynamically generated web pages that display portfolio postions and other changing data.

I've been programming heavily in C++ for many years. While I have a love hate relationship with C++'s complexity, I never thought that I would use Java heavily. But I've been working on more web services applications that access a database. The huge class library that is available with Java is a great advantage when it comes to developing these applications. Sure I can pay thousands of dollars to Rogue Wave and get some of the same features in C++ that I get for free using the Sun and Apache Java libraries. But why? The higher performance of C++ is of little use to most applications that references a database, since these applications are usually bottlenecked in the database. And when it comes to web services (e.g., XML processing, servlets, dynamically generated web pages), C++ cannot compare to Java. And then there is garbage collection, which makes develoopment faster.

There are still applications that I would not write in Java. For example, compilers or other algorithmically intensive applications that are CPU bound. There are also times when I can simplify my code by using C++ features like operator overloading (see my wavelet packet transform algorithm for example). But these applications are now in such a minority in the work I'm doing that I worry that my C++ "chops" will get rusty.

I went to a talk by Stroustrup where he discussed C++ (some cool algorithms to support linear algebra computation), future developments and C+++ vs. Java. He promotes C++ as a systems programming language. For things like operating systems, virtual machines, hardware drivers and compilers. He trumpted C++ as the most widely used programming language. What does not seem to have occured to Stroustrup is that systems level applications, where C++ shines, are a small minority of the code programmers write. My view is that C++'s star is fading.

So yeah, I'd say Java is heavily used on Linux. At work I'm part of a group developing a distributed database application in Java (this runs on top of a relational database, so Java's performance is not an issue), hosted on Linux. I'm in the process of setting up a Linux/PostgreSQL system on which to develop a financial trading application, again in Java, using XML and web services.

I work at the Lawrence Livermore National Labs in Livermore, California. At least at Lawrence Livermore (LLNL) there are excellent summer internship programs. Each "Directorate" has a summer internship program. The directorates include "Computation" (the directorate I work in), materials science, engineering, physics, chemistry and molecular biology. Internships are available for both undergrads and graduate students. At the graduate student level there are divisions that also hire people outside of the sciences (like East Asia experts).

You need to apply early, usually the year before (by December 31 in some cases) or early in the year (by January or so). Some programs require letters from your professors. So start early. At least at LLNL you need to be a US citizen. There is a basic background check for summer jobs.

Hiring at "The Labs" ebbs and flows, depending on funding. If you take part in a summer program and you impress people during your internship, there is a better chance that you will be offered a job on graduation.

The projects are interesting and it is a good chance to get experience. A friend of mine's daughter is working in an internship in materials science. They are giving her access to the MEMS silicon fabrication facility (which, she tells me, costs more than they are paying her for the summer). Summer projects in computation in the past have involved networking and cryptography.

The summer programs are closed for this year. You can find them described on the LLNL web site (llnl.gov). The directorates do the hiring, so please don't send me your resume. I can't help in this area.

There are a number of other national labs with summer programs. Labs that come to mind are Los Alamos (LLNL's "sister lab"), Oak Ridge, Argonne, the Pacific Northwest National Lab. There is also an Dept. of Energy engineering lab in Idaho, but I don't know if they hire summer students.

Finally, for what it's worth my heart goes out to anyone who is graduating in CS or engineering these days. In my twenty four years working in this industry I have never seen times like these (and that includes the 1992 down turn). As others have noted, the problem is that there is unemployment among experienced engineers, so this makes it doubly difficult for new grads.

The irony is that from what I've read, hiring is booming for engineers in India (see my essay An Economics Question). This is one reason I'm grateful to have a job at a National Lab. My work requires a security clearance and it will never be moved overseas.

I can see your point about privacy. It is true that once you allow something to read email it could be abused. But to balance this is the fact that, at least for me, email would be useless without a spam filter.

Privacy is not an issue in my case. I use text only email on Linux (email never touches my Windoz system for security reasons). I run a spam filter for my own email account, so it is my program that reads my email, not someone elses. I read my email on a shared Linux system run by the ISP that hosts my domain (my ISP is webquarry.com).

As far as I know, the RBL approach would not work in my case. I do discard some email one the basis of the domain name, which is far less efficient than the RBL. My spam filter keeps a log of some of the header information from the email it discards. A fair amount of spam is going through fixed domain names these days (e.g., like the infamous tekmailer).

One of the problems I had with the commonly used spam filters was that it was unclear to me how to install them in the case where I am simply piping my email to them. I was also concerned about resource usage, since I am using a shared system. So like a typical programmer I wrote my own spam filter in C++. It is probably 80 to 90 percent efficient. Enough spam still gets through that I'm going to take another look at SpamAssassin and see if I can get it to run with a "procmail" forward. It is just too time consuming to constantly hack the spam filter for the latest evil spammer trick (recently they have been sending spam to my email address from the other valid user on my domain, where I don't check content).

As time goes on DSPAM (and SpamAssassin for that matter) become more and more sophisticated, incorporating more complex algorithms. What I also find striking is that many of these algorithms appear to be compute intensive. These spam filters seem to be designed for server side ISP level email filtering. I would expect that a computer would have to be dedicated to running this anti-spam software.

Also, as a number of posters have noted, configuring these spam filters takes some effort and education on the part of the user.

This level of resource comsumption is fine for an ISP, but it seems problematic for a single user or someone using a shared system.

I only use Linux shell based email to avoid worms and viruses. I've had my domain since 1995, so I got a lot of spam. As the tide of SPAM increased, overcoming the primative SPAM filter I was using I looked at both SpamAssassin and DSPAM. But these tools do not appear appropriate for a shared Linix system like the one that hosts my domain. So, in classic "reinvent the wheel fashion", I wrote my own SPAM filter. It is just a simple (compared to these tools) rule based filter. It filters out enough SPAM that email it not totally useless. This email filter can be found here. The email filter is written in C++ in an attempt to minimize resource usage. It is published as open source.

I too would think that a portfolio of past work would demonstrate to a prospective employer that I can engineer software. I have published thousands of lines of Java and C++ code on my web site www.bearcave.com.

What constantly strikes me as odd is that people I've interviewed with are more or less unwilling to look at my work (note that this work belongs to me and has been done on my own time). And those who do interviews that consists largely of programming problems still insist in having me code up their little algorithm on the white board. As I note on the web page linked to above, I think that in part this is because the interviewer does not know what else to ask.

When I have been a hiring manager I've looked for people who really loved software and had engineered large systems. Given this they can learn what ever else I need them to know.

I too would think that a portfolio of past work would demonstrate to a prospective employer that I can engineer software. I have published thousands of lines of Java and C++ code on my web site www.bearcave.com.

What constantly strikes me as odd is that people I've interviewed with are more or less unwilling to look at my work (note that this work belongs to me and has been done on my own time). And those who do interviews that consists largely of programming problems still insist in having me code up their little algorithm on the white board. As I note on the web page linked to above, I think that in part this is because the interviewer does not know what else to ask.

When I have been a hiring manager I've looked for people who really loved software and had engineered large systems. Given this they can learn what ever else I need them to know.

Such optimism and belief in the capitalist system. You should be living down here in the lower 48.

My view paraphrases something that Churchill said: Capitalism is the worst possible system. Except for all of the others. Yes I believe in the power of markets and their ability in many cases to effiently allocate resources and generate wealth. But capitalism also has its "red claw" side as well.

Basicly wealth follows a power law distribution called the Pareto distribution. The result of this for as long as capitalism has existed is that a few percent of the population control most of the wealth. In republics there is a constant struggle to shift this curve so that there is more distribution to more people. Right now the wealthy, at least in the US are winning the distribution struggle.

The US has been living off of debut. This is true of individuals and of the country as a whole. The vaunted standard of living in the US is not really much better than Europe or Canada. And income has largely stagnated in the last decade or so as well paying manufacturing jobs are moving offshore.

Perhaps average income has increased, but not the median income. The wealthy are doing well and until 2000 technology workers were doing well. Now may professions are threatened by offshoring. For more on this see my essay An Economics Question.

should go here. -- Tired of political dishonesty? Vote for the Lemon Party!

The economist Brian Arthur is one of the proponents of the theory of path dependence. In path dependence something is adopted for reasons that might be determined by chance (e.g., the adoption of MS/DOS) or by some related feature (C became popular in part because of UNIX's popularity).

The widespread use of C and C++, languages without bounds checking in a world where we can afford bounds checking, is not so much a matter of logical decision as history. C became popular, C++ evolved from C and provided a some really useful features (objects, expressed as classes). Once C++ started to catch on, people used C++ because others used it and an infrastructure developed (e.g., compilers, libraries, books). In sort, the use of C++ is, to a degree, a result of path dependence. Once path dependent characteristics start to appear, choices are not necessarily made on technical virtue. In fact, one could probably say that the times when we make purely rational, engineering based decisions (feature X is important so I'll use language Y) are outweighed by the times when we decide on other criteria (my boss say's we're gonna use language Z).

Someone mentioned that we over estimate the value of our data. That's probably true.

While I acknowledge this, I've thought of the archiving issue too. I've been working on my web site www.bearcave.com since 1995. The material published on this web site represents the largest work I've completed that does not belong to someone else. I intend to keep adding to it. In the long run it may represent the largest work I've accomplished in my life.

Egotist that I am, I'd like it to survive me. I have searched and I did not find any web repository except for the Internet Archive, which attempts to archive the Internet. The Internet Archive has archived bearcave.com, so there is some chance that my work will be around when I'm not. The way things are going there will probably come a time when you can carry around the current Internet Archive in your pocket, so the costs of archiving should drop, which also provides some hope that the Internet Archive data itself will survive.

Unfortunately, the Internet Archive is not an ideal solution. Given bandwidth issues, they cannot afford to update too frequently. Also, while the Internet Archive is locally searchable, I don't think that is is searchable by search engines like Google. So material on the Internet Archive is not as accessible as other material on the Web.

There appears to be a possible business here (perhaps at the non-profit level). I'd be willing to pay money into an escrow account and a monthly fee to have my web site scanned weekly. The when I die my web site would no longer be scanned and my data be available to the web on the new site.

The problem with such a business is that it would probably have to be set up as a non-profit. The concentration of an archiving business is to pay its bills and survive in the long term, not make lots of money for its founders or shareholders.

There are some technical complexities as well. Internal links between web site pages would have to be changed so that they worked at the new location. But it should not be too difficult to write conversion software.

An interesting post on the differences between Indian and US development. Thanks for posting manavendra. Like many US software engineers, I have more than a passing interest in our competition in India.

There is an odd dynamic here. The view with some people who outsource work from the US to low wage countries like India seems to be that if the people who are doing the outsourced work are not as efficient, it doesn't matter because they are so cheap. These same people are either unable to analyze or ignorant of software lifecycle cost.

I'm sure that there are development groups in India that are as good and efficient as those in the US. But for the sake of argument even if it were true that all Indian groups were mired in paperwork and ISO-standardization it would not matter, since they are viewed as cheap labor. Outsourcing now seems to have reached the level of management fad, where the sole justification is short term cost. If you run a corporate IT department you probably have to justify to your masters why you are not outsourcing.

The story of Daniel Robbins and Gentoo Linux seems to me to be an example of software as art and Daniel as a starving artist. And yes, I realize that many other people were involved in the Gentoo project. But one of the leading forces behind Gentoo seems to be leaving because he can't afford to take part in the project anymore.

The world does not owe artists, writers or software engineers working on open source/ Free Software a living. But what is interesting to me is that if, for the sake of argument, some commercial entity, like Red Hat, were to come along and start selling Gentoo at some point in the future, Daniel Robbins and the rest of the Gentoo developers would get as much as the Linux developers got from Red Hat going public (e.g., very little).

If software engineering jobs were not moving overseas and our income was not under constant downward pressure this might not be such a big deal. There is a lot to be said for doing something you love. For many people money can't replace this. But when it gets to the point where you can't pay your bills or are unemplyed, survival becomes the important issue.

Speaking for myself, the current state of our industry throws into question any open source project that can be picked up by slick marketeers and resold to end users. Since I'm not independently wealthy, why should I work for free? I have to wonder if Daniel Robbins is not asking himself similar questions as he looks at the state of his finances.

For more on this see my essay Freedom Can be Slavery

You act like the victims of the system are responsible for it. Would you say that the victims of the Great Cultural Revolution got what they deserved because they supported Mao? Or would you say that Stalin's victims also got what was coming to them because they supported the Bolsheviks?

Obviously the effects of globalization and offshoring are nothing like the excesses of communism. But offshoring and globalization do produce their share of misery. Just because people live in a capitalist society does not mean that they agree to a race to the bottom with people in India and China.

The United States is a diverse place. Over half of the voters voted against G.W. Bush (remember, we have a strange electoral system and an arguably corrupt Supreme Court, so the majority did not win the election). An increasing number of people don't believe that invading Iraq was a good idea. A majority of people in the US finally understand that there were no weapons of mass destruction in Iraq.

Most people don't understand what is happening to them. Their incomes are not going up. If they are lucky enough to have a job, there is no job stability. But they are just trying to get by. Between the stress of making a living and numbing their mind with television there is not a whole lot of reflection. So most people just believe what they are told. And right now all the corporate and government economists are telling that that offshoring is good for them. "Yes you are losing your job, but you'll get a better, higher paying job to replace it". It takes people time and sometimes serious misfortune to understand that the system is not working for them. In the Great Depression it took 25% unemployment.

Capitalism and trade protection when it comes to moving jobs to low wage countries is not incompatible. If things keep going as they are, there is no question that there will be protectionist laws. Just as there have been with auto imports. People in the West are not going to quietly sit there while etheir income is averaged with the incomes of the huge populations of India and China.

At least in the US there is still a lot of print media that provides different perspectives on reality. With the exception of India, which has a more or less free press, this is not true in most of Asia. At at least here if enough voters make a fuss, change will happen.

I've written a long, and perhaps turgid essay that touches on many of these issues. See An Economics Question. This essay includes a growing list of web published references.

The Eudaemonic Pie was reissued as Newtonian Casino (both titles by Thomas Bass). These books recount the efforts by Norman Packard, Doyne Farmer and others to build a computer that could predict roulette.

Thomas Bass is also the author of the book The Predictors. Packard and Farmer when on to found Prediction Company which builds models that attempt to do short term stock market prediction. My review of The Predictors can he found here. I don't find Thomas Bass the most reliable of reporters. My retrospective on his book The Predictors book can be found here

The Eudaemonic Pie was reissued as Newtonian Casino (both titles by Thomas Bass). These books recount the efforts by Norman Packard, Doyne Farmer and others to build a computer that could predict roulette.

Thomas Bass is also the author of the book The Predictors. Packard and Farmer when on to found Prediction Company which builds models that attempt to do short term stock market prediction. My review of The Predictors can he found here. I don't find Thomas Bass the most reliable of reporters. My retrospective on his book The Predictors book can be found here

What surprises me is not that a product, which is not due to ship for months, has announced that they are using the Transmeta processor, but that Transmeta is still around (see Transmeta: hype and processor performance).

Other than support for the underdog, the fascination in the Linux community with Transmeta has been the Linux (or Linus) connection. This connection is now tenuous at best. It's time to look at Transmeta with a less romantic eye.

Originally Transmeta was founded to provide a high performance processor for laptops (I interviewed with them during this phase). They claimed that they were going to be the next Intel (truely, a quote). Unfortunately, the Transmeta processor could not beat Intel and AMD processors. So then they refocused on producing a low power chip for portable devices. But then Intel and others came out with low power processors. So there is no real edge that I can detect for Transmeta. I am only surprised that they are still around.

what's the big deal with these compiled/interpreted arguments? Can't you compile java source if you want, no JVM required? Isn't it also true that most performance problems are located in a very small portion of the code, but it's extremely difficult to figure out where (ahead of time)? So why not just write it all in java and then mix 'n' match interpreted java, compiled java, FORTRAN, Guile, or whatever?

If Java could be natively compiled...

You might be interested in this page.

I'm not saying that hotspot isn't impressive, but it isn't anywhere near native code yet.

And becuase of this, I refuse to even acknowledge Sun sponsored benchmarks - they're rigged. So don't respond with Sunworld, Javalobby, or Sun.com benchmarks.