I implemented and rolled out a secured Oracle web application with a 5-user concurrent limit ($2000) by choking the number of Apache children down to five.
Real fucking hard. Could we have afforded an unlimited web license? Probably. But for $28K less we got a working install which services everyone who needs it (maybe 500 people, never more than 5 at one instant in time).
Duuuuhhhh...
Henry Ford said something along the lines of "people who don't think ahead never have enough time; people who do, never have too little". Same thing goes for money, licenses, what have you.
In any event, around the release of Postgres 6.6 (currently projected for 5 months or so), a lot of these license questions will be moot -- Postgres should achieve functional parity with Oracle at that point, and you'll have a free alternative.
I change my mind on whether Java sucks or rules about once a week, but it would be nice for the Jakarta product (i.e. Apache-Jserv Part Too) to work with AOLserver. I've been waiting around to find out whether Sun will provide its javax.net.security.x509 classes or whether I "get to" write those for Jakarta... maybe I should think about how to make the next version of Apache-Jserv work with AOLserver, as it is going to be my webserver of choice now that its source is open (at least until Apache 2.0 comes out).
On the other hand, smart guys like William Crawford routinely turn up on Philip's discussion boards, so someone else may have already started doing this. I don't know (or care) how servlets worked in the old nsdynamo implementation, but Apache-Jserv is a very nice tool, as it makes load-balancing and distributed execution trivial. Jakarta/Apache-Jserv 1.2 plus AOLserver would be very pleasant to work with for platform-agnostic geeks... there are many things to recommend Java (that evil, slow, strongly-typed, bad-for-the-web language from Sun Proprietary Products, Inc and Alan Baratz... "it's just a reference implementation, we can't be held responsible!").
Oracle is gluing XML support onto 8io but it looks like a ploy to lock you into their tools. Fsck that.
My personal experiences with XML/XSL/XQL are that the whole wad of them currently make my life harder rather than easier. YES, I KNOW IT'S A MORE ELEGANT SOLUTION. But so was Common Lisp.
>The best [...] distro is lagging more and more > because you feared they were too big and > badmouthed them on any possible occasion. Now > noone dares to aproach them.
Who said anything bad about Debian? Corel's perfectly happy to work with them as I see it.
The J5000 is "only" $20000 for the minimal dual CPU configuration, though I remember the dual-CPU cards for the V-Class machines being something like $32000. Maybe they just don't sell many V-Class boxes, whaddya think?;-)
Doing a little bit of guesstimating and hand-waving we could arrive at... oh, about $6000 to $8000 depending on who you believe.
(ie. I don't recall seeing any PA-8500s at auction recently... doesn't seem like an eBay kinda item)
Of course, I have always had the luxury of ignoring agencies and talking directly to my prospective boss. I'm not planning on changing these habits in the near future.
Why can't people sell their Real Machines with Linux? They worked with those Wacky Canucks to port Linux to the PA-RISC, the 8500 w/1.5MB on-chip cache is faster than fast, so why not ditch the skanky old HP-sUX and sell Linux on it?
Hey, they could even enhance Linux a little! It's not like anyone in their right mind wants to run a commercial Unix on a workstation anymore. (oh alright, maybe Irix or Solaris, but not HPUX)
The PA-RISC is a weird but well-engineered chip, and the 8500 is the last of them AFAIK... it would be nice to walk up to some random Linux box in a couple of years and discover it was running That 64-bit Chip That Got Steamrolled By Intel.
Oh well, at least Intel isn't as evil as MS... instead of just destroying HP's stuff and forcing more x86 (the architectural equivalent of BASIC) down everyone's throats, they're actually sticking their necks out and producing something that works. Maybe. In the second generation...
>> turning the encryption off is a command >> decision that you "get paid the >> big bucks for."
Gee, if I were a hacker, I'd *never ever* wait until a big event (eg. market goes to hell) to start dumping to disk if I had managed to hack into a decent-sized ISP (or worked there and was a pissy sort of person). The prestige for showing an online broker to be vulnerable has to be pretty significant, especially if you moonlight as a "security consultant" or whatever.
Maybe I'm just a wuss, but it seems like s/get paid/get fined/g; is a distinct possibility if the ruse is uncovered. (It's also a tacky thing to do)
I suppose that with those sorts of loads, you could make a case for it being statistically infeasible to pull any real information out without a huge amount of disk space to dump the packets onto and a lot of time to pore through them... but people don't change their passwords very often, and you could probably assemble useful information in a reasonable amount of time. And a decent lawyer should have no trouble spooking a jury into overreacting if a trial came to pass.
Either way I submit that the magnitude of the negative publicity that would ensue would make such a decision very hard to justify.
Why not colocate at, say, Above.Net, and rely on their monster pipes for the big loads? It's not like it would cost that much more, and you rely on an extremely high caliber of technical staff to keep things running.
>> Encryption makes a 30% performance hit
In my experience you're off by almost an order of magnitude, in terms of CPU load. If you're only talking about packet throughput, then yeah, the handshake, key exchange, and renegotiation every few minutes adds about 30%. It seems like CPU power is usually the bottleneck in doing SSL transactions on big fat pipes, though.
>> people who have gotten MBAs seem to only trust >> other people with MBAs
I had the misfortune of working at one of the top business schools in the country for about a year, and this is what I perceived: MBAs without a physical science or engineering background are categorically inept at technical decisions, no matter how much they think they have learned by reading InfoWorld. Negotiation is an MBA's strong point; following through is Someone Else's Job, as best as I could make out. So why don't they recognize that they are likely to make more money (enough to offset the cost) if they hire the best (and most expensive) technical staff? Beats me...
'Cause otherwise you're presenting an opening for someone else to gain publicity as Those Guys That Suck Less (tm) and steal your mindshare and profits. That can't possibly be lost on MBAs. (can it?)
From the very beginning, Greg Abrams and the people at IBM research have been very helpful and accomodating, resulting in 4 tarball releases in the 4 weeks since DX was opened up. OpenDX now runs on (at least) Linux, Linux-AXP, LinuxPPC, MkLinux, FreeBSD, SunOS, Tru64, Irix, Solaris, SunOS, AIX, and HP-UX, and the client is even known to run on Windows with Cygwin. A great deal of this has been made possible by user-contributed patches being applied to the tree at IBM Research, resulting in new tarballs.
There is not yet a publicly available CVS tree, and the mailing lists are explosively overactive (I have no idea how the guys working on this project for IBM get anything else done), but they have been extraordinarily helpful and receptive to suggestions. Lesstif is being closely examined and apparently improved as a side effect (since most Linux users do not purchase Motif, duh). Distributed DX and an RPM are in the works thanks to the efforts of the IBM crew, and I am trying to SWIG parts of DX so it can be scripted from the web with minimal effort. And for whatever it's worth, I have started working on DX2Octave again now that I have access to DX on my machines.
As far as the code goes, it's a little crufty, but anyone who tells me that (for example) the original Mozilla codebase was any cleaner is insane. And OpenDX worked out-of-the-box, the day it was released, on many platforms. Plus, IBM is licensing some of its own patents to outside developers by releasing DX in its working entirety. They went the extra mile than Netscape did not, which is why I bring up Mozilla vs. DX.
I am very pleased with IBM Research's involvement in their open-sourced projects, and IMHO they are a great example for other companies to follow.
Fermilab has plans to build a 2000-node cluster in the near future but is putting off purchasing all the nodes until the last second to maximize their value.
re: Rackmounts
They're more expensive, and typically the machine rooms at large Beowulf installations have enough space for whatever they choose to use. It's not like Los Alamos has to pay for space at the colo when they add a new pile of Alphas.
>> Would it be newsworthy if ESR was invited to >> speak at Sun Microsystems or SGI?
Probably, but not nearly as newsworthy. The reason is obvious to me -- ESR has publicly stated his contempt for Bill Gates after being brushed off by Gates some time ago.
A more reasonable comparison would be to diehard EFF members sitting down to chat with Scott "No Privacy" MacNealy, or perhaps Michael Dell speaking at Apple on the company's future...
One of my friends in SF pointed out that she ought to get some new, high-profile parts to keep her movie career swingin'. She did the artsy Permanent Midnight and some other forgettable films, but dammit, she's way too hot and too solid an actress to be marginalized!
>> Unfortunately I do not have the time to write >> either an introductory tutorial on scientific >> computing
The shorter version: see under "numeric integration" and/or "matrix/tensor manipulation"...
>> Er.. Jon, who do you think used what you call >> "deep computing" before? Some kids in a garage?
some graduate student at MIT with a secretary, maybe (viz. the original Connection Machine). Granted this is not representative, but then again, things like Beowulf (don't start) are making embarrassingly parallel simulation much more accessible to the average lab or business. And that appears to be the (muddled) point that Katz is getting at, i.e. that IBM has effectively recognized that Joe Average might have something to contribute back to the field. Not trivial.
Beyond that, there's some sort of relatively understandable source for people to go look at now when you need to explain why on earth you'd want a 1024-node Origin 2000 or a real Beowulf. "No, it doesn't play Quake any faster..."
>> Jon, I don't think you understand what >> supercomputers do.
Kaa, I don't think you are acknowledging what some people use supercomputers to do. Realtime visualization ("intelligence amplification" in some peoples' jargon) is at least as useful as numerical simulation (more often complementary to it, as a tool for extracting useful conclusions) and provably more so than AI in a general sense.
If you do understand this, then you're purposely ignoring these uses in order to flame Katz, which, while tempting, is irresponsible. Not least because the general readership of Slashdot isn't going to rebut you, and you know it.
>> They have not magically acquired any >> problem-solving technology. All they do is >> crunch numbers, usually vectors and matrices, >> really really fast. The class of problems >> suitable for these machines is not big at all.
Now you're really misleading people. Relational databases are quite useful to the general computing public -- you are using one whenever you post to Slashdot or make a withdrawal at the ATM. This single application alone has probably done more to advance the practice of high-end computer engineering than AI, the NSA, and the NSF put together. (the military... well, that's inevitable) Anyways, not everyone uses their SP2 (or what have you) for CFD, molecular mechanics, or other noble intellectual pursuits. In fact, I'd bet that a minority of them do.
One of the major uses of ultra-high-end (nonspecialized, i.e. non-vector-processor) machines is to serve as the backend for OLAP (database analysis and decision support) in huge corporations. Data visualization ala IBM's suite of mining products is a major application for these people, and (perhaps equally so, though not necessarily using the same toolset) for scientific users who get to sift through reams of simulation results. It's a whole hell of a lot easier to render a fracture simulation in realtime after applying the appropriate transforms to the experimental results than it is to try and grasp the same results as raw data. (although sometimes the opposite is true -- whoever said "a debugger doesn't replace good thinking, but good thinking doesn't replace a debugger" must have had the same experience) Likewise (and this is where I am coming from) a good tool for getting useful conclusions from protein folding simulations or ligand docking can literally be worth millions of dollars (esp. to Big Pharma).
>> Believing that increased specialized processing >> power will solve the world's political and >> social problems is naive at best.
To put it gently. This is the crux of my argument against your post, perhaps paradoxically. The tools and thought from top-notch researchers (which IBM hires quite a few of) are critical to the effective use of big iron. That's why, in my estimation, the formation and dialogue with the public about "Deep Computing" (what a silly name -- I'd rather see "the Grand Challenge Institute") by IBM actually is significant. Besides, maybe some kids in a garage will find enough use for a pile of P90's running DDX to get a grant and do something useful. Don't rule it out, and don't forget that developing similar tools from scratch would waste months/years of their life. VTK, a competing model to DX, has been open for quite some time, and research applications of it have been quite clever -- there was even an article in Linux Journal on how to use VTK for engineering simulation analysis a while ago. If you think that making the tools to create better predictive models available is inconsequential, maybe you haven't had to come up with one in a while! It's a real pain in the ass -- as you seem to point out.
>> You are >> confusing ability to solve a problem (e.g. >> build a good predictive model) and raw >> computing power.
I hope he's not, but I wanted to say the same about your post. I'm not supporting Katz in general -- his wide-eyed optimism bothers me -- but I do think you were overly harsh and might turn some people off from a vibrantly interesting field which (thank god) is getting some of the recognition and money which it deserves.
>>>> If some of the most specialized existing data >>>> on the planet were focused on specific >>>> medical problems, treatment and research be >>>> greatly accelerated.
>> The meaning of this sentence is beyond me. Does >> it mean that if medical researchers read each >> others publications we would be able to...
So an heuristic approach to relating useful information within the avalanche of academic literature produced each month would be unhelpful to active researchers? Realtime visualization of otherwise indistinguishable tissues (see this month's _Scientific American_ and try not to vomit when they refer to visualization as "Virtual Reality") is not an advance for neurosurgeons? Sifting faster and more effectively through the flood of genomic and proteomic data published each day is of no interest to patients or insurers?
Have you been working in CFD, many-body simulations, or some other "Grand Challenge" field for so long that you have forgotten about the mundane uses that the unwashed masses have for big iron? Katz may not necessarily know what he's talking about, but this happens to be correct. And your puny little Ultra won't put a dent in most of these problems. Making tools for using real Big Iron more affordable and visible could be the difference between budgeting $3 million for a Microsoft junkware upgrade and buying a UE10K or setting up a farm of parallel & distributed compute nodes at some places.
If you want to continue this dialogue offline, for better or worse (please feel free to flame the shit out of any hyperbole in my reply, for starters), please do so. I am about 8 months out of the loop WRT real supercomputing, but the release of the DX source and patents was as exciting to me as most anything in recent memory. More importantly, it looks like I'm going back to the Big Iron, so we may be able to use these tools for day-to-day business, even more so than at my current job (where the market research/data analysis crew was delighted that tools like DX are now available for use on lower-end hardware -- they can afford to wait a week for results I used to get in 30 minutes). All in all I view IBM's announcements as very significant, far more so than the latest JVM or the newest Microsoft vaporware update, and I agree with Katz in that respect.
As for politicians... well, you're right, that part of the article is beyond hope. However, people at places like the Santa Fe Institute actually do work on simulating social and economic developments, so Katz may not be 100% off base in that respect. I don't know enough about the accuracy of those simulations to say.
This has about as much heft as me claiming that no one has the right to tell anyone what to do. It may be true in my way of thinking, but the fact is that a society cannot function like that.
Besides, it's not your systems in question here, so what you think is irrelevant.
Here is my take on what is going on, having worked at IBM (doing contract work, not as an IBMer), having worked with larger-scale IBM hardware, and having talked to a fair number of IBMers.
IBM is supporting the needs of their customers.
That's it. If you are in a position to pay for their services (and IBM support is not cheap), you are important. If you can help make IBM's image more palatable to the people who buy IBM support (this includes research centers and universities), you are somewhat important. (their viewpoint)
Consider that IBM recently built a Beowulf in a day specifically to demonstrate how powerful their Intel-based hardware was. Consider that IBM is happy to support Java and Emacs on S/390s to make their paying customers keep buying services. Consider that the vast majority of IBM's new revenue is coming from custom e-commerce solutions as hardware, software, and support packages.
IBM stands for big iron and reliable, powerful solutions. Not raging I/O (though the mainframes are good at that), not integrated-everything (though OS/2 acts kind of like that), but relentless, dependable solutions that will be supported ten years down the road. There are a great many systems in service at IBM that are at least that old.
Anything that makes IBM more attractive to their core customers will be supported. DX is not the crown jewel of IBM Research -- they provide algorithm design services to monster companies like Monsanto, and IBM has been the only company AFAIK to turn a steady profit selling supercomputers. (The SP/2 is basically a Beowulf with a much higher-performance switch.)
There is no hidden agenda here -- IBM wants you to give them your money and be happy to keep doing so, especially if you are a large business or government institution. If supporting Linux and opening up the source to esoteric supercomputing tools or next-generation compilers makes more customers choose IBM, that's what gets done.
Opening the source will make DX a better product and increase demand for hardware, which IBM conveniently provides, and makes IBM look like the anti-Microsoft in some peoples' eyes. So they do it. A Beowulf built out of, say, Netfinity boxes is easier to maintain because of the hardware-diagnosing features (LightPath for example), so they exhibit the power of a Beowulf.
One day someone from IBM was wondering why the Java-Apache project "Cocoon" wasn't using the IBM XML4J parser anymore. Stefano (Mazzochi, the guy who started the whole Java-Apache thing, and an Apache core developer) replied that it was because open-source tools do not grow momentum without a feeling of participation (eg. "I built this from SCRATCH"). For simple or generally useful tools I think he's right, but for stuff like DX -- which as an internal IBM Research project was put together over the course of 18 months by some of the smarter people at IBM, without interference from marketing, and with the direct support of a company VP -- I can't see how starting over would help. If the whole codebase is released, the potential for so many cool tie-ins and hacks that it's unreal. (if not... well, I think a lot of people will be quite disappointed) The word that comes to mind when describing DX's data model is "profound" -- it is built on the notion of expression data filtering in terms of the mathematics of fiber bundles, independent of the content flowing through the fibers.
There is also a distributed version of DX, which (who knows?) might be Just Right for making Beowulf useful to more than just scientists. Visualization tools (like DX, or AVS) can offer insights into huge datasets that simple reports cannot -- clustering of sales data, or hit rates for candidate drugs as a function of ethnicity.
Anyways, this isn't some version of corporate insanity. IBM wants to sell you stuff and will do whatever it takes to make that happen. If a ton of positive publicity is generated along the way, so much the better!
For all the bitching I did initially, I now feel that Oracle is pretty good. And running it with the JDBC drivers being called from servlets in Apache Jserv reduces the moron factor exponentially (eg. how much damage can be done by someone who isn't really on the ball).
Informix is supposed to be nice too, and their support staff is enthusiastic. However, I have heard reports from a friend of mine (a DBA at Cornell, where they are rolling out Informix for payroll on AIX) that it is less than 100% robust.
So on my current project, I have learned to like Oracle and apparently the feeling is mutual;-).
Works fine for my employer. Actually, that's a lie -- it works vastly better than the previous NT/Sybase "solution". It was $2000, though, for 5 users.
I guess you get what you pay for when it comes to RDBMSes. Of course, you could fix Postgres if you don't want to pay. You have the source, yes?
If you manage to get Postgres running to your satisfaction you can take advantage of 64-bit platforms that Linux runs on, which pretty much evens the playing field. That's a big if, though.
Regardless, I think people are being overly dramatic about the database situation on Linux.
unless you have used it. I was working on trying to get around several of the (patented) techniques used in DX so as to build my own version of it... I guess I can finish writing that DX2Octave module that I was going to finish when I was at Cornell.
This software is *very* impressive. The screen shots on www.almaden.ibm.com don't convey what's really cool about DX -- check out this page for a better idea of how DX makes really nasty visualization tasks simple.
This rules as much as SGI releasing XFS, not to put too fine a point on it. (Although SGI appears not to be releasing LVM support for XFS -- d'oh.)
Looks like I've got my activity for the rest of this week's afternoons (at least). Now I just need to send notes to everyone I tried to explain DX to saying "Just download it, it's Free now".
Re:The letter I wrote to the editor of time.com
on
Time Review of Linux
·
· Score: 2
>>Your readers may also benefit from knowing that >>Gnome, while developed with help from Red Hat, >>is also open source software and is available >>from www.gnome.com...
They'd probably benefit even more from knowing it was at www.gnome.ORG, since there's some silly company at www.gnome.com.
Between FreeBSD, RedHat, and Debian, FreeBSD was probably easiest, followed by RedHat, and then Debian. Of course an AIX build was easier than any of these, and an NT install harder than any of these.
So there's my opinion. The whole "3 distros" nonsense makes it sound like there isn't much to do/but/ install Linux, i.e. why bother/using/ it when you can go get another distro to install?
Anyways, people can just buy a Dell or VA box with the OS preinstalled if they're concerned about the installation process.
Oh yeah, that XFS thingy. Wow, wouldn't that be cool on a Beowulf?;-) (actually, yes, if you built a SAN like the Bulk Data Server on it)
Whut else... um, Time Digital is now running a series on Linux... Mozilla is actually GOING SOMEWHERE... and people are saying "Hurd" again. Intel investing in VA... Compaq supporting Linux on Alpha... Sun supporting Linux on Sparc64... SGI selling boxes with Linux on them... IBM selling boxes with Linux on them... HP selling Linux support... TUCOWS putting up Linux crap (?)...
What new media formats are there for Windows? I haven't booted into it (except in vmware, to test some client-side SSL stuff in IE4) since 1998 so forgive my asking;-)...
Well, yeah, but then the FreeBSD guys believe that all the SysV folks will See The Light and switch to a simpler (but nonstandard) initialization sequence.
Again, the world keeps spinning. The great thing is that with Linux, we can pick up after SGI, fix the bugs, and still have a better system than when it started. I don't really think that the BSD folks would mind that, either, if SGI wanted to use FreeBSD or NetBSD as the in-house OS of choice.
There's a lot that I hate(d) about Irix, and when I saw the CERT advisory about "4dgifts" and other unpassworded default accounts I decided that security wasn't top priority on Irix.
Still, their I/O has always been top notch. If we can have better, fully reentrant I/O in Linux and ditch the crufty old bits of Irix, why not?
Incidentally, the crabby old men are usually right, as I'm sure you were implying. But sometimes they don't see the big picture.
>>...if SGI decides to hold close it's dead IRIX >>technology the community will lose a very good >>scalable operating system.
With sieve-like security and a tiny user base. I used to use Irix and loved it, but come on... the world keeps on spinning. Cellular Irix will probably show up on the ultra-high-end; go get yerself a O2K, maaan. As for the midrange, who cares? Linux scales as well as Irix on an O2...
I'm overjoyed that SGI is bringing in the heavy I/O artillery for Linux. Unless you really despise all us unwashed Linux users, you should be too. AOL will probably be enough (by themselves) to drive Irix *support*, but maybe not *development*, especially on the low end. (AOL runs AOLserver on O2Ks with Sybase as the main backend; they're keeping all 3 of these in business by my estimation;-))
Incidentally, Irix goes to 128-way on the big CC-NUMA systems. It effortlessly did 20-way on our (straight SMP) Onyx when I was at Cornell... I don't disagree that it rules, but try explaining that to a PHB that thinks GUI hooks in a server OS kernel are Modern.
They OSS'ed OpenVault, why wouldn't they do the same (or similar) with XFS? Well, methinks they may take the opportunity to engineer something better. I went and bought the Be book on Filesystem Design when I realized the level of flexibility the VFS gives you. It's pretty cool.
Anyways, SGI == I/O and we should all rejoice. The chances of NT retaining a lead in brute-force I/O (which is a big, big hangup for Linux in the scalability/multithreaded department) should now be slim-to-none. Hah, Hah... and we all thought SGI had sold out. Maybe they just pulled an IBM.
Comparing (stable, slower) Apache + no Squid to (fast, unstable) IIS instead of something like Zeus or Boa is, in fact, bullshit. Apples and oranges -- or have you never run a goddamn server?
The next question is who has the money to buy a quad Xeon with quad fast Ethernet NICs, but can't scrape together the change to get a gigabit NIC and switch instead? Uh, I'll take "no one" for $1000, Alex.
Slashdot Mirror
I implemented and rolled out a secured Oracle web application with a 5-user concurrent limit ($2000) by choking the number of Apache children down to five.
Real fucking hard. Could we have afforded an unlimited web license? Probably. But for $28K less we got a working install which services everyone who needs it (maybe 500 people, never more than 5 at one instant in time).
Duuuuhhhh...
Henry Ford said something along the lines of "people who don't think ahead never have enough time; people who do, never have too little". Same thing goes for money, licenses, what have you.
In any event, around the release of Postgres 6.6 (currently projected for 5 months or so), a lot of these license questions will be moot -- Postgres should achieve functional parity with Oracle at that point, and you'll have a free alternative.
I change my mind on whether Java sucks or rules about once a week, but it would be nice for the Jakarta product (i.e. Apache-Jserv Part Too) to work with AOLserver. I've been waiting around to find out whether Sun will provide its javax.net.security.x509 classes or whether I "get to" write those for Jakarta... maybe I should think about how to make the next version of Apache-Jserv work with AOLserver, as it is going to be my webserver of choice now that its source is open (at least until Apache 2.0 comes out).
On the other hand, smart guys like William Crawford routinely turn up on Philip's discussion boards, so someone else may have already started doing this. I don't know (or care) how servlets worked in the old nsdynamo implementation, but Apache-Jserv is a very nice tool, as it makes load-balancing and distributed execution trivial. Jakarta/Apache-Jserv 1.2 plus AOLserver would be very pleasant to work with for platform-agnostic geeks... there are many things to recommend Java (that evil, slow, strongly-typed, bad-for-the-web language from Sun Proprietary Products, Inc and Alan Baratz... "it's just a reference implementation, we can't be held responsible!").
;-)
Oracle is gluing XML support onto 8io but it looks like a ploy to lock you into their tools. Fsck that.
My personal experiences with XML/XSL/XQL are that the whole wad of them currently make my life harder rather than easier. YES, I KNOW IT'S A MORE ELEGANT SOLUTION. But so was Common Lisp.
>The best [...] distro is lagging more and more
> because you feared they were too big and
> badmouthed them on any possible occasion. Now
> noone dares to aproach them.
Who said anything bad about Debian? Corel's perfectly happy to work with them as I see it.
;-)
The J5000 is "only" $20000 for the minimal dual CPU configuration, though I remember the dual-CPU cards for the V-Class machines being something like $32000. Maybe they just don't sell many V-Class boxes, whaddya think? ;-)
Doing a little bit of guesstimating and hand-waving we could arrive at... oh, about $6000 to $8000 depending on who you believe.
(ie. I don't recall seeing any PA-8500s at auction recently... doesn't seem like an eBay kinda item)
and you no longer have to deal with the idiots.
Of course, I have always had the luxury of ignoring agencies and talking directly to my prospective boss. I'm not planning on changing these habits in the near future.
Why can't people sell their Real Machines with Linux? They worked with those Wacky Canucks to port Linux to the PA-RISC, the 8500 w/1.5MB on-chip cache is faster than fast, so why not ditch the skanky old HP-sUX and sell Linux on it?
Hey, they could even enhance Linux a little! It's not like anyone in their right mind wants to run a commercial Unix on a workstation anymore. (oh alright, maybe Irix or Solaris, but not HPUX)
The PA-RISC is a weird but well-engineered chip, and the 8500 is the last of them AFAIK... it would be nice to walk up to some random Linux box in a couple of years and discover it was running That 64-bit Chip That Got Steamrolled By Intel.
Oh well, at least Intel isn't as evil as MS... instead of just destroying HP's stuff and forcing more x86 (the architectural equivalent of BASIC) down everyone's throats, they're actually sticking their necks out and producing something that works. Maybe. In the second generation...
Meanwhile the PA-RISC chips work fine already.
>> turning the encryption off is a command
>> decision that you "get paid the
>> big bucks for."
Gee, if I were a hacker, I'd *never ever* wait until a big event (eg. market goes to hell) to start dumping to disk if I had managed to hack into a decent-sized ISP (or worked there and was a pissy sort of person). The prestige for showing an online broker to be vulnerable has to be pretty significant, especially if you moonlight as a "security consultant" or whatever.
Maybe I'm just a wuss, but it seems like
s/get paid/get fined/g;
is a distinct possibility if the ruse is uncovered. (It's also a tacky thing to do)
I suppose that with those sorts of loads, you could make a case for it being statistically infeasible to pull any real information out without a huge amount of disk space to dump the packets onto and a lot of time to pore through them... but people don't change their passwords very often, and you could probably assemble useful information in a reasonable amount of time. And a decent lawyer should have no trouble spooking a jury into overreacting if a trial came to pass.
Either way I submit that the magnitude of the negative publicity that would ensue would make such a decision very hard to justify.
Why not colocate at, say, Above.Net, and rely on their monster pipes for the big loads? It's not like it would cost that much more, and you rely on an extremely high caliber of technical staff to keep things running.
>> Encryption makes a 30% performance hit
In my experience you're off by almost an order of magnitude, in terms of CPU load. If you're only talking about packet throughput, then yeah, the handshake, key exchange, and renegotiation every few minutes adds about 30%. It seems like CPU power is usually the bottleneck in doing SSL transactions on big fat pipes, though.
>> people who have gotten MBAs seem to only trust
>> other people with MBAs
I had the misfortune of working at one of the top business schools in the country for about a year, and this is what I perceived: MBAs without a physical science or engineering background are categorically inept at technical decisions, no matter how much they think they have learned by reading InfoWorld. Negotiation is an MBA's strong point; following through is Someone Else's Job, as best as I could make out. So why don't they recognize that they are likely to make more money (enough to offset the cost) if they hire the best (and most expensive) technical staff? Beats me...
'Cause otherwise you're presenting an opening for someone else to gain publicity as Those Guys That Suck Less (tm) and steal your mindshare and profits. That can't possibly be lost on MBAs. (can it?)
There is not yet a publicly available CVS tree, and the mailing lists are explosively overactive (I have no idea how the guys working on this project for IBM get anything else done), but they have been extraordinarily helpful and receptive to suggestions. Lesstif is being closely examined and apparently improved as a side effect (since most Linux users do not purchase Motif, duh). Distributed DX and an RPM are in the works thanks to the efforts of the IBM crew, and I am trying to SWIG parts of DX so it can be scripted from the web with minimal effort. And for whatever it's worth, I have started working on DX2Octave again now that I have access to DX on my machines.
As far as the code goes, it's a little crufty, but anyone who tells me that (for example) the original Mozilla codebase was any cleaner is insane. And OpenDX worked out-of-the-box, the day it was released, on many platforms. Plus, IBM is licensing some of its own patents to outside developers by releasing DX in its working entirety. They went the extra mile than Netscape did not, which is why I bring up Mozilla vs. DX.
I am very pleased with IBM Research's involvement in their open-sourced projects, and IMHO they are a great example for other companies to follow.
DX homepage, with downloads and license info
OpenDX.org
re: Big-Ass Clusters
Fermilab has plans to build a 2000-node cluster in the near future but is putting off purchasing all the nodes until the last second to maximize their value.
re: Rackmounts
They're more expensive, and typically the machine rooms at large Beowulf installations have enough space for whatever they choose to use. It's not like Los Alamos has to pay for space at the colo when they add a new pile of Alphas.
>> Would it be newsworthy if ESR was invited to
>> speak at Sun Microsystems or SGI?
Probably, but not nearly as newsworthy.
The reason is obvious to me -- ESR has publicly stated his contempt for Bill Gates after being brushed off by Gates some time ago.
A more reasonable comparison would be to diehard EFF members sitting down to chat with Scott "No Privacy" MacNealy, or perhaps Michael Dell speaking at Apple on the company's future...
Maybe it's time for a letter-writing campaign. ;-)
otherwise agreed, though I'm not sure that it was actually illegal for AT&T to sell Sys[i]r[j].
>> Unfortunately I do not have the time to write
...
>> either an introductory tutorial on scientific
>> computing
The shorter version: see under "numeric integration" and/or "matrix/tensor manipulation"...
>> Er.. Jon, who do you think used what you call
>> "deep computing" before? Some kids in a garage?
some graduate student at MIT with a secretary, maybe (viz. the original Connection Machine). Granted this is not representative, but then again, things like Beowulf (don't start) are making embarrassingly parallel simulation much more accessible to the average lab or business. And that appears to be the (muddled) point that Katz is getting at, i.e. that IBM has effectively recognized that Joe Average might have something to contribute back to the field. Not trivial.
Beyond that, there's some sort of relatively understandable source for people to go look at now when you need to explain why on earth you'd want a 1024-node Origin 2000 or a real Beowulf. "No, it doesn't play Quake any faster..."
>> Jon, I don't think you understand what
>> supercomputers do.
Kaa, I don't think you are acknowledging what some people use supercomputers to do. Realtime visualization ("intelligence amplification" in some peoples' jargon) is at least as useful as numerical simulation (more often complementary to it, as a tool for extracting useful conclusions) and provably more so than AI in a general sense.
If you do understand this, then you're purposely ignoring these uses in order to flame Katz, which, while tempting, is irresponsible. Not least because the general readership of Slashdot isn't going to rebut you, and you know it.
>> They have not magically acquired any
>> problem-solving technology. All they do is
>> crunch numbers, usually vectors and matrices,
>> really really fast. The class of problems
>> suitable for these machines is not big at all.
Now you're really misleading people. Relational databases are quite useful to the general computing public -- you are using one whenever you post to Slashdot or make a withdrawal at the ATM. This single application alone has probably done more to advance the practice of high-end computer engineering than AI, the NSA, and the NSF put together. (the military... well, that's inevitable) Anyways, not everyone uses their SP2 (or what have you) for CFD, molecular mechanics, or other noble intellectual pursuits. In fact, I'd bet that a minority of them do.
One of the major uses of ultra-high-end (nonspecialized, i.e. non-vector-processor) machines is to serve as the backend for OLAP (database analysis and decision support) in huge corporations. Data visualization ala IBM's suite of mining products is a major application for these people, and (perhaps equally so, though not necessarily using the same toolset) for scientific users who get to sift through reams of simulation results. It's a whole hell of a lot easier to render a fracture simulation in realtime after applying the appropriate transforms to the experimental results than it is to try and grasp the same results as raw data. (although sometimes the opposite is true -- whoever said "a debugger doesn't replace good thinking, but good thinking doesn't replace a debugger" must have had the same experience) Likewise (and this is where I am coming from) a good tool for getting useful conclusions from protein folding simulations or ligand docking can literally be worth millions of dollars (esp. to Big Pharma).
>> Believing that increased specialized processing
>> power will solve the world's political and
>> social problems is naive at best.
To put it gently. This is the crux of my argument against your post, perhaps paradoxically. The tools and thought from top-notch researchers (which IBM hires quite a few of) are critical to the effective use of big iron. That's why, in my estimation, the formation and dialogue with the public about "Deep Computing" (what a silly name -- I'd rather see "the Grand Challenge Institute") by IBM actually is significant. Besides, maybe some kids in a garage will find enough use for a pile of P90's running DDX to get a grant and do something useful. Don't rule it out, and don't forget that developing similar tools from scratch would waste months/years of their life. VTK, a competing model to DX, has been open for quite some time, and research applications of it have been quite clever -- there was even an article in Linux Journal on how to use VTK for engineering simulation analysis a while ago. If you think that making the tools to create better predictive models available is inconsequential, maybe you haven't had to come up with one in a while! It's a real pain in the ass -- as you seem to point out.
>> You are
>> confusing ability to solve a problem (e.g.
>> build a good predictive model) and raw
>> computing power.
I hope he's not, but I wanted to say the same about your post. I'm not supporting Katz in general -- his wide-eyed optimism bothers me -- but I do think you were overly harsh and might turn some people off from a vibrantly interesting field which (thank god) is getting some of the recognition and money which it deserves.
>>>> If some of the most specialized existing data
>>>> on the planet were focused on specific
>>>> medical problems, treatment and research be
>>>> greatly accelerated.
>> The meaning of this sentence is beyond me. Does
>> it mean that if medical researchers read each
>> others publications we would be able to
So an heuristic approach to relating useful information within the avalanche of academic literature produced each month would be unhelpful to active researchers? Realtime visualization of otherwise indistinguishable tissues (see this month's _Scientific American_ and try not to vomit when they refer to visualization as "Virtual Reality") is not an advance for neurosurgeons? Sifting faster and more effectively through the flood of genomic and proteomic data published each day is of no interest to patients or insurers?
Have you been working in CFD, many-body simulations, or some other "Grand Challenge" field for so long that you have forgotten about the mundane uses that the unwashed masses have for big iron? Katz may not necessarily know what he's talking about, but this happens to be correct. And your puny little Ultra won't put a dent in most of these problems. Making tools for using real Big Iron more affordable and visible could be the difference between budgeting $3 million for a Microsoft junkware upgrade and buying a UE10K or setting up a farm of parallel & distributed compute nodes at some places.
If you want to continue this dialogue offline, for better or worse (please feel free to flame the shit out of any hyperbole in my reply, for starters), please do so. I am about 8 months out of the loop WRT real supercomputing, but the release of the DX source and patents was as exciting to me as most anything in recent memory. More importantly, it looks like I'm going back to the Big Iron, so we may be able to use these tools for day-to-day business, even more so than at my current job (where the market research/data analysis crew was delighted that tools like DX are now available for use on lower-end hardware -- they can afford to wait a week for results I used to get in 30 minutes). All in all I view IBM's announcements as very significant, far more so than the latest JVM or the newest Microsoft vaporware update, and I agree with Katz in that respect.
As for politicians... well, you're right, that part of the article is beyond hope. However, people at places like the Santa Fe Institute actually do work on simulating social and economic developments, so Katz may not be 100% off base in that respect. I don't know enough about the accuracy of those simulations to say.
>If M$ refused to support it's customer base
They already do.
> USENET is not the Internet police. No one is.
This has about as much heft as me claiming that no one has the right to tell anyone what to do. It may be true in my way of thinking, but the fact is that a society cannot function like that.
Besides, it's not your systems in question here, so what you think is irrelevant.
Here is my take on what is going on, having worked at IBM (doing contract work, not as an IBMer), having worked with larger-scale IBM hardware, and having talked to a fair number of IBMers.
IBM is supporting the needs of their customers.
That's it. If you are in a position to pay for their services (and IBM support is not cheap), you are important. If you can help make IBM's image more palatable to the people who buy IBM support (this includes research centers and universities), you are somewhat important. (their viewpoint)
Consider that IBM recently built a Beowulf in a day specifically to demonstrate how powerful their Intel-based hardware was. Consider that IBM is happy to support Java and Emacs on S/390s to make their paying customers keep buying services. Consider that the vast majority of IBM's new revenue is coming from custom e-commerce solutions as hardware, software, and support packages.
IBM stands for big iron and reliable, powerful solutions. Not raging I/O (though the mainframes are good at that), not integrated-everything (though OS/2 acts kind of like that), but relentless, dependable solutions that will be supported ten years down the road. There are a great many systems in service at IBM that are at least that old.
Anything that makes IBM more attractive to their core customers will be supported. DX is not the crown jewel of IBM Research -- they provide algorithm design services to monster companies like Monsanto, and IBM has been the only company AFAIK to turn a steady profit selling supercomputers. (The SP/2 is basically a Beowulf with a much higher-performance switch.)
There is no hidden agenda here -- IBM wants you to give them your money and be happy to keep doing so, especially if you are a large business or government institution. If supporting Linux and opening up the source to esoteric supercomputing tools or next-generation compilers makes more customers choose IBM, that's what gets done.
Opening the source will make DX a better product and increase demand for hardware, which IBM conveniently provides, and makes IBM look like the anti-Microsoft in some peoples' eyes. So they do it. A Beowulf built out of, say, Netfinity boxes is easier to maintain because of the hardware-diagnosing features (LightPath for example), so they exhibit the power of a Beowulf.
One day someone from IBM was wondering why the Java-Apache project "Cocoon" wasn't using the IBM XML4J parser anymore. Stefano (Mazzochi, the guy who started the whole Java-Apache thing, and an Apache core developer) replied that it was because open-source tools do not grow momentum without a feeling of participation (eg. "I built this from SCRATCH"). For simple or generally useful tools I think he's right, but for stuff like DX -- which as an internal IBM Research project was put together over the course of 18 months by some of the smarter people at IBM, without interference from marketing, and with the direct support of a company VP -- I can't see how starting over would help. If the whole codebase is released, the potential for so many cool tie-ins and hacks that it's unreal. (if not... well, I think a lot of people will be quite disappointed) The word that comes to mind when describing DX's data model is "profound" -- it is built on the notion of expression data filtering in terms of the mathematics of fiber bundles, independent of the content flowing through the fibers.
There is also a distributed version of DX, which (who knows?) might be Just Right for making Beowulf useful to more than just scientists. Visualization tools (like DX, or AVS) can offer insights into huge datasets that simple reports cannot -- clustering of sales data, or hit rates for candidate drugs as a function of ethnicity.
Anyways, this isn't some version of corporate insanity. IBM wants to sell you stuff and will do whatever it takes to make that happen. If a ton of positive publicity is generated along the way, so much the better!
For all the bitching I did initially, I now feel that Oracle is pretty good. And running it with the JDBC drivers being called from servlets in Apache Jserv reduces the moron factor exponentially (eg. how much damage can be done by someone who isn't really on the ball).
;-).
Informix is supposed to be nice too, and their support staff is enthusiastic. However, I have heard reports from a friend of mine (a DBA at Cornell, where they are rolling out Informix for payroll on AIX) that it is less than 100% robust.
So on my current project, I have learned to like Oracle and apparently the feeling is mutual
Works fine for my employer. Actually, that's a lie -- it works vastly better than the previous NT/Sybase "solution". It was $2000, though, for 5 users.
I guess you get what you pay for when it comes to RDBMSes. Of course, you could fix Postgres if you don't want to pay. You have the source, yes?
If you manage to get Postgres running to your satisfaction you can take advantage of 64-bit platforms that Linux runs on, which pretty much evens the playing field. That's a big if, though.
Regardless, I think people are being overly dramatic about the database situation on Linux.
unless you have used it. I was working on trying to get around several of the (patented) techniques used in DX so as to build my own version of it... I guess I can finish writing that DX2Octave module that I was going to finish when I was at Cornell.
This software is *very* impressive. The screen shots on www.almaden.ibm.com don't convey what's really cool about DX -- check out this page for a better idea of how DX makes really nasty visualization tasks simple.
This rules as much as SGI releasing XFS, not to put too fine a point on it. (Although SGI appears not to be releasing LVM support for XFS -- d'oh.)
Looks like I've got my activity for the rest of this week's afternoons (at least). Now I just need to send notes to everyone I tried to explain DX to saying "Just download it, it's Free now".
>>Your readers may also benefit from knowing that
>>Gnome, while developed with help from Red Hat,
>>is also open source software and is available
>>from www.gnome.com...
They'd probably benefit even more from knowing it was at www.gnome.ORG, since there's some silly company at www.gnome.com.
Failure to proofread is bad for all of us...
Between FreeBSD, RedHat, and Debian, FreeBSD was probably easiest, followed by RedHat, and then Debian. Of course an AIX build was easier than any of these, and an NT install harder than any of these.
/but/ install Linux, i.e. why bother /using/ it when you can go get another distro to install?
So there's my opinion. The whole "3 distros" nonsense makes it sound like there isn't much to do
Anyways, people can just buy a Dell or VA box with the OS preinstalled if they're concerned about the installation process.
...what else matters?
;-) (actually, yes, if you built a SAN like the Bulk Data Server on it)
;-)...
Oh yeah, that XFS thingy. Wow, wouldn't that be cool on a Beowulf?
Whut else... um, Time Digital is now running a series on Linux... Mozilla is actually GOING SOMEWHERE... and people are saying "Hurd" again.
Intel investing in VA... Compaq supporting Linux on Alpha... Sun supporting Linux on Sparc64... SGI selling boxes with Linux on them... IBM selling boxes with Linux on them... HP selling Linux support... TUCOWS putting up Linux crap (?)...
What new media formats are there for Windows? I haven't booted into it (except in vmware, to test some client-side SSL stuff in IE4) since 1998 so forgive my asking
LinuxPPC newbie help? Shit, send me an email!
Well, yeah, but then the FreeBSD guys believe that all the SysV folks will See The Light and switch to a simpler (but nonstandard) initialization sequence.
Again, the world keeps spinning. The great thing is that with Linux, we can pick up after SGI, fix the bugs, and still have a better system than when it started. I don't really think that the BSD folks would mind that, either, if SGI wanted to use FreeBSD or NetBSD as the in-house OS of choice.
There's a lot that I hate(d) about Irix, and when I saw the CERT advisory about "4dgifts" and other unpassworded default accounts I decided that security wasn't top priority on Irix.
Still, their I/O has always been top notch. If we can have better, fully reentrant I/O in Linux and ditch the crufty old bits of Irix, why not?
Incidentally, the crabby old men are usually right, as I'm sure you were implying. But sometimes they don't see the big picture.
>>...if SGI decides to hold close it's dead IRIX >>technology the community will lose a very good
;-))
>>scalable operating system.
With sieve-like security and a tiny user base. I used to use Irix and loved it, but come on... the world keeps on spinning. Cellular Irix will probably show up on the ultra-high-end; go get yerself a O2K, maaan. As for the midrange, who cares? Linux scales as well as Irix on an O2...
I'm overjoyed that SGI is bringing in the heavy I/O artillery for Linux. Unless you really despise all us unwashed Linux users, you should be too. AOL will probably be enough (by themselves) to drive Irix *support*, but maybe not *development*, especially on the low end. (AOL runs AOLserver on O2Ks with Sybase as the main backend; they're keeping all 3 of these in business by my estimation
Incidentally, Irix goes to 128-way on the big CC-NUMA systems. It effortlessly did 20-way on our (straight SMP) Onyx when I was at Cornell... I don't disagree that it rules, but try explaining that to a PHB that thinks GUI hooks in a server OS kernel are Modern.
They OSS'ed OpenVault, why wouldn't they do the same (or similar) with XFS? Well, methinks they may take the opportunity to engineer something better. I went and bought the Be book on Filesystem Design when I realized the level of flexibility the VFS gives you. It's pretty cool.
Anyways, SGI == I/O and we should all rejoice. The chances of NT retaining a lead in brute-force I/O (which is a big, big hangup for Linux in the scalability/multithreaded department) should now be slim-to-none. Hah, Hah... and we all thought SGI had sold out. Maybe they just pulled an IBM.
Long Live SGI.
Comparing (stable, slower) Apache + no Squid to (fast, unstable) IIS instead of something like Zeus or Boa is, in fact, bullshit. Apples and oranges -- or have you never run a goddamn server?
The next question is who has the money to buy a quad Xeon with quad fast Ethernet NICs, but can't scrape together the change to get a gigabit NIC and switch instead? Uh, I'll take "no one" for $1000, Alex.
You're not nearly as realistic as you think.