IBM's "Deep Computing"

ZDNet is reporting that IBM is forming a "$29M research institute". The assigned scientists will focus on using supercomputers to model real-life scenarios. Apparently, they plan to release some software for visual data representation in the near future under an open source license. Update: 05/25 09:40 by J : IBM is forming an alliance with Pacific HiTech where PHT will support IBM's DB2 under Linux. It also seems DB2 will ship with a special version of PHT in the future.

Comments

This is very good for the scientific community

[sluke]

Increasingly over the past few years, scientists have turned to computationally intensive solutions to all kinds of problems. As a student who has changed institutions in the past few years I have worked on two markedly different projects that involve computational power that was unheard of several years ago.

The real news in this announcement is that IBM is opening up supercomputers to researchers. Processing power of this magnitude was previously very, very difficult to come by. (either you worked at a national lab, or you got a grant from NCSA) Otherwise you chugged along on whatever hardware your funding would buy, perhaps a four processor SGI or so. And you could be guaranteed that you would wait on the order of a week for any meaningful results.

That said, this software IBM is creating doesn't seem to be terribly necessary to me. As I said earlier, people have been using computationally intensive methods to solve problems for over five years now. Only in fairly rare cases can a scientist not find code written by collegues that will preform the necessary computations with little modification.

In fact the whole idea seems a little bit misguided. In the article they talk about using supercomputers to model precise weather patterns. Lorentz showed a while back that precise weather patterns are chaotic, thus rendering most modeling quite useless! Unless this data explorer tool is a big improvement over already available packages it will fall by the wayside. The supercomputing time, however is one of the best gifts a computer company has given to science to date.

Vote "No" on Knee-Jerk Reactions

[Anonymous Coward]

IBM is not 'pouring' $29 million into an open-source project. The $29 million is going into this research institute in general; the fact that one of the tools which this institute plans to use is being open-sourced does not indicate any sort of silly subsidizing of the software project; remember, it's already completed and in use in mission-critical situations, which were mentioned in the article. Although the software does look interesting, what IBM is really doing here is supporting free (or open-source, if you do insist) software in the ways it was always intended to be used; that is, this IBM-supported organization is a -research- institute, and the tools and findings of their research will be freely available to anyone. Of course, most of us would be hard-pressed to recreate their research, given our general lack of access to hardware as powerful as theirs. The point is that this is neither a commercial (esr, 'open-source') nor moralist (rms, 'free software') venture; the software is open because it is the most appropriate avenue of distribution for the software, given its intended use.

Gift Horse....

[fornix]

I honestly believe that most large companies such as IBM are just using open-source as a method to gain publicity and free development out of the open-source community.

Perhaps so. But isn't this a perfectly acceptable way to get good press? Even if the license is not perfect, lots of people could potentially benefit from this.

Options:

1. You can't drive my Ferrari
2. You can drive it all you want - use it as a chick magnet even, but you can't change the license plates or get a job using it as a taxi without my permission
3. You can do with it what you please.

Option 2 is still a good thing. Even if they retain some measure of control, it's still a good gesture - they didn't have to do this. If someone tweaks their code to make their project fly better, and then returns to patch to IBM, then others will still benefit from the patch.

I would like to hereby encourage other software companies to seek publicity by giving me cool software to use.

Re:Big companies and open-source

[Bruce Perens]

Gary,

Are you really reading the licenses? Netscape might want to control it's source, but they don't prevent you from making a fork of the program that you control. Their license grants you that privilege.

I've been working hard to get the licenses to the point that they're fair for us. I wouldn't dream of claiming these are my successes alone, but we saw a lot of improvement in the Qt 2.0 license, the APSL 1.1, and there are more coming, big ones. Sure, the companies get some benefit from this. But there's a quid-pro-quo, a balance between what we give and what we get back.

Let's be a bit more constructive with specific complaints about the licenses, not vague ones, please.

Thanks

Bruce Perens

Weather is good too!

[Anonymous Shepherd]

There are some places really good weather prediction on the scale of only a few hours would be absolutely wonderful:

Hurricane tracking, ie being able to take an enormous amount of ground and satellite data and doing the 'deep processing' IBM is talking about. Even if one can only see for an hour ahead, that could save millions in loss and lives by having that extra hour to prepare.

Tornado tracking, being able to predict the path of a tornado, or even predicting its formation. If a deep procesing unit were permanently spent in each state in the tornado belt, and fed a constant stream of satellite and ground data, even being able to predict a tornado by 5 minutes would be of great benefit, and being able to predict it's path by even a couple hundred meters would be a god send.

Weather for satellite and shuttle launch. If we can see the weather for the next week with reasonable accuracy with today's methods, deep computing may allow us a reasonable view for up to a month or so. Precise weather is not necessary, just as long as we can avoid storms and heavy cloud cover!


-AS

Why this matters

[jabbo]

Here is my take on what is going on, having worked at IBM (doing contract work, not as an IBMer), having worked with larger-scale IBM hardware, and having talked to a fair number of IBMers.

IBM is supporting the needs of their customers.

That's it. If you are in a position to pay for their services (and IBM support is not cheap), you are important. If you can help make IBM's image more palatable to the people who buy IBM support (this includes research centers and universities), you are somewhat important. (their viewpoint)

Consider that IBM recently built a Beowulf in a day specifically to demonstrate how powerful their Intel-based hardware was. Consider that IBM is happy to support Java and Emacs on S/390s to make their paying customers keep buying services. Consider that the vast majority of IBM's new revenue is coming from custom e-commerce solutions as hardware, software, and support packages.

IBM stands for big iron and reliable, powerful solutions. Not raging I/O (though the mainframes are good at that), not integrated-everything (though OS/2 acts kind of like that), but relentless, dependable solutions that will be supported ten years down the road. There are a great many systems in service at IBM that are at least that old.

Anything that makes IBM more attractive to their core customers will be supported. DX is not the crown jewel of IBM Research -- they provide algorithm design services to monster companies like Monsanto, and IBM has been the only company AFAIK to turn a steady profit selling supercomputers. (The SP/2 is basically a Beowulf with a much higher-performance switch.)

There is no hidden agenda here -- IBM wants you to give them your money and be happy to keep doing so, especially if you are a large business or government institution. If supporting Linux and opening up the source to esoteric supercomputing tools or next-generation compilers makes more customers choose IBM, that's what gets done.

Opening the source will make DX a better product and increase demand for hardware, which IBM conveniently provides, and makes IBM look like the anti-Microsoft in some peoples' eyes. So they do it. A Beowulf built out of, say, Netfinity boxes is easier to maintain because of the hardware-diagnosing features (LightPath for example), so they exhibit the power of a Beowulf.

One day someone from IBM was wondering why the Java-Apache project "Cocoon" wasn't using the IBM XML4J parser anymore. Stefano (Mazzochi, the guy who started the whole Java-Apache thing, and an Apache core developer) replied that it was because open-source tools do not grow momentum without a feeling of participation (eg. "I built this from SCRATCH"). For simple or generally useful tools I think he's right, but for stuff like DX -- which as an internal IBM Research project was put together over the course of 18 months by some of the smarter people at IBM, without interference from marketing, and with the direct support of a company VP -- I can't see how starting over would help. If the whole codebase is released, the potential for so many cool tie-ins and hacks that it's unreal. (if not... well, I think a lot of people will be quite disappointed) The word that comes to mind when describing DX's data model is "profound" -- it is built on the notion of expression data filtering in terms of the mathematics of fiber bundles, independent of the content flowing through the fibers.

There is also a distributed version of DX, which (who knows?) might be Just Right for making Beowulf useful to more than just scientists. Visualization tools (like DX, or AVS) can offer insights into huge datasets that simple reports cannot -- clustering of sales data, or hit rates for candidate drugs as a function of ethnicity.

Anyways, this isn't some version of corporate insanity. IBM wants to sell you stuff and will do whatever it takes to make that happen. If a ton of positive publicity is generated along the way, so much the better!