Reliable Java Threading w/ Linux 2.4?

mikej asks: "I'm involved in an ongoing saga that attempts to get an application running a large number of Java threads (30-60 sustained, 300+ peak) reliably. My last large hurtle was the context switching delay on the 2.2 kernel, which the 2.4 series has _nicely_ fixed. Now I'm running into problems keeping the JVM alive under threadload. The Sun and IBM java mailing lists and newsgroups are full of complaints concerning threading problems on Linux, and I'm faced with the choice of moving the system to Solaris (something that I very much do not want to do). Has anyone seen a system like this in action, and is there an especially successful combination of SDK/glibc/kernel?"

Comments

Which JVM?

[sohp]

There are many JVMs that run on Linux. Have you tried the IBM JVM? Green threads or native threads? Have you tried building a kernel with the number of threads (MR_TASKS) increased?

You can find some suggestiions for your problems at these two sites: jGuru Linux FAQ Java on Linux by Nathan Meyers

youre using the sun jvm you dolt.

[Zurk]

which has known threading issues. switch to blackdown..ive been using it for 450 threads sustained with no problems. use the FCS JDK 1.2.2 or 1.3

Re:youre using the sun jvm you dolt.

[Zurk]

and remember -- when in doubt use synchronised atomic spinlocks. specially with that kind of aggressive threading.

Re:youre using the sun jvm you dolt.

[Zurk]

actually thats mostly true -- but only if the application is written really correctly. with that many threads you HAVE to do a good job or you will have problems since each JVM handles threads differently. for example, java threads on windows arent as agressive as those on solaris -- ive seen applications which dont handle deadlocks correctly die on solaris/linux while working perfectly on windows even though the application was badly written. its fairly incredible to see this in action -- i was positively *stunned* since im used to writing heavily multithreaded code thats synced correctly and having it run across multiple machines/jvms in clusters correctly with no problems. ive seen my code behave exactly the same on windows nt/2k, solaris, hpus, aix and linux even though it had 450-500 odd threads striped across no less than 4 JVM instances and multiple clustered machines.
anyway for best results i recommend :
blackdowns JDK 1.2.2FCS on linux 2.2.17pre20 or 18 or 19preX
suns JDK 1.2.2 on solaris with 108940-07 patches
IBMs JDK 1.2.x for AIX with the patches
HPs JDK 1.1.7/8 for HP/UX on 11.0
ive had problems with all the rest...not tried linux 2.4.x though..too alpha for me. 2.2.x with ingos low latency patches beats even 2.4.x i think.
BTW, if you do want to test how heavily a JVM is multithreaded download the swingworker 2 java file from suns site and see if it produces nullpointerexceptions. if it doesnt then the JVM doesnt thread aggressively. on windows swingworker 2 works fine while it barfs on solaris 30-50% of the time. swingworker 3 fixed that bug but swingworker 2 makes a good test tool.

Re:youre using the sun jvm you dolt.

[mikej]

I will do so. I'm currently juggling 4 different SDKs for testing, but abandoned blackdown long ago. It won't be hard to start pounding on it again.

Re:youre using the sun jvm you dolt.

[mikej]

This is an excellent idea, though I'm not the one developing this codebase. The developers are very prone to the "if it's java, I can do anything in java and any problems are the sysadmin's job" mentality. I think they actually believe the whole "write once, run anywhere" hype. Frustrating, but I appreciate this suggestion.

Re:Threading is for weenies.

[Tower]

Real Men don't use Java...
--

We gave up...

[metacosm]

I had the same battle that you are having _before_ 2.4 and we decided that the money being spent on trying to fix the problem was better spent buying the big "S". So we did - the application runs very nicely on it as well. Do some quick math - how much are you spending trying to get it to work on Linux and how much will "S" run you - and let the dollars and sense decide for you.

Re:We gave up...

[ndfa]

A very very sound idea!

BUT to add to this Sun has done a lot with the new HotSpot server and threading with it is a lot better! With the Blackdown JVM i would consistently have a server (running maybe 15 threads) take down the JVM almost constantly if 'sunwjit' was used! I have seen this problem a lot less with the hotspot server which i am currently running on a slackware 7 system with linux 2.4.0!
One idea might be to setup Solaris 8 on an intel box, or get a Ultra 10 (they are pretty cheap) and see how things work there!
Also, there are many ways you could shoot yourself in the foot with that many threads ? Could GC be a problem ? Are you reusing the threads correctly ? Lots of stuff to think off that could cause problem! You might want to look at "Taming Java Threads" by Allen Holub... good read and could help you!
just my $0.02

Re:We gave up...

[mikej]

I'm beginning to come to a similar conclusion. The reponses I've gotten from people when I ask this question have generally been along the lines of "You're trying to do what? The problem is at least partially our codebase (the developers build and test on NT machines, and are skeptical when I report linux-specific issues (like Thread.interrupt() being dangerous), but also with pthread. Lower load helps, as (I'm hoping) will a move to a glibc 2.2 base.

Re:the big "S"

[mikej]

Well, direct accountability for bugs means nothing to me in this case. I can't argue that Java doesn't run better on Sun's platform, but I can guarantee that I'm reluctant to be pushed around like that. That kind of behavior is why I started using linux so long ago in the first place. My benches have shown Linux being considerably faster for what we're doing, and Solaris has been a giant pain in the ass to run. I have to install about 18 different gnu packages to get any kind of useful behavior out of a solaris machine. Frustrating.

Re:the big "S"

[ratboy666]

Just to let you know...

Solaris 8 has some GNU stuff -- bash, bzip2,
less, etc.

Ratboy666

Sorry, but

[SuiteSisterMary]

The Sun and IBM java mailing lists and newsgroups are full of complaints concerning threading problems on Linux, and I'm faced with the choice of moving the system to Solaris (something that I very much do not want to do).

Sorry, and I don't mean to be harsh, but that's just ignorant. If you're using Linux for Linux's sake, then shut up and live with the inherent problems. Linux simply nowhere near ready for the enterprise. No, let me back up. The Linux kernel isn't too bad; GNU/Linux the system is. Too many 0.x released bits, written by college fanboys in their spare time, with an 'if it compiles, it's done' mentality. This is bad. Also, the entire 'incremental updates' idea behind Linux as it stands is anathema to proper methodology; 'if it doesn't work,' they cry, 'patch the code!' 'Yes,' you say, 'that's fine, except that all code changes need to go through substantial regression testing, and besides, it should work in the first place.' I personally lost all respect for Linux on the server when the cron daemon that shipped (and enabled by default) on Mandrake 7.x would run jobs scheduled for 4 AM at 10 AM. Or 4 PM. Or whenever it felt like it. Guess what; if system A can't do what you need it to, but system B can, but you don't want to switch to B for undisclosed reasons, you have no business complaining about A's problems. And as an aside, you need to get somebody in charge to yell at the programmers. The problem with Java is that it promotes sloppy coding, because people brought up in Java tend to assume that the language will correct all problems. And they don't understand concepts like proper memory management. Feel free to get in touch if you want some more conversation on the joys of Linux vs quite a few other platforms in real deployment.

Re:the big "S"

[brsett]

It sounds more like you just don't know how to use Solaris, and you are too lazy to learn. GNU tools are superfluos on a Solaris System. Solaris has many flaws as an OS (or at least up to 2.6, I haven't ran 7 much or 8 at all), but kernel support for multithreading works fine since at least 2.5. You hit a flaw in linux, and yet you still want to use it. Its the whole sharp tool debate. You're doing surgery, and you've got access to a scalpel, but you prefer pocket knives cause normally you whittle. Linux makes alot of sense for some applications (e.g. imo they are good development boxes, as they are fairly cheap to upgrade), but when you want processing power (which is presumable why a program would be cranking out threads at that rate, because threads for threading sake is stupid, even for java programmers), don't be afraid to get a better tool (in this case a sun box, preferably not on x86 hardware, but maybe you guys are short on cash). Solaris machines should be stable with loads into the 80's and 90's.

the big "S"

[ClubPetey]

Last I saw the big "S" was free, save for media costs ($10 - $20). I've seen Linux releases for $120 at the local CompUSA. I've been at several Internet companies over the last few years and all of them have the Solaris/Linux debate at some point. In the end Solaris wins becuase:

There is direct acountablity and support for bugs.

Java is SUN, and thus runs better on SUN Software

Solaris x86 is faster in most cases than Linux

Solaris is easier to administrate than Linux (this really only is true if you use non-standard releases or multiple releases of Linux)
--
He had come like a thief in the night,