Slashdot Mirror
Are Linux Transactions Slower Than Win2k's?
FullClip asks: "In the July issue of PC Magazine, Red Hat Professional is compared to Windows NT/2000 on basis of ServerBench, which tests the maximum Transactions Per Second (TPS) for a given number of clients. Red Hat 6.1 (when tweaked) matched the performance of Windows, but showed a terrible decrease in performance at about 24 clients to a weeping 20 % of the level that Windows was able to maintain. Somehow this disturbs me.
Doesn't Linux perform better than that bad in client-server environments? If someone can point me to an non-FUD benchmark site, it would be appreciated..." Is this yet another case where benchmarks have been skewed severely to show a deficiency that doesn't exist? Or is this another area where Linux needs improvement? [Updated 6 July 2000 2:15 GMT by timothy] You may want to compare this with the far different results reported by SpecWeb.
Why is it always Linux vs. Windows?
Because that's what Linux advocates trump up. Ever since Linux became 'popular', advocates have been pitting it against the big bad evil Microsoft. Nevermind that until recently, Solaris was just as closed-source and dealt in the same underhanded tricks as Microsoft. Nevermind that they're two completely different types of operating systems aimed at two entirely different classes of people.
Basically, Linux people want Linux to be able to do everything that Windows can. They want it to be a robust server operating system. They want it to be an easy-to-use client operating system. They want it to run everything. They want to be the monopoly (but a monopoly of choice, not of force). Nevermind that Windows 2000 isn't trumped as the OS for everyone and Windows 98 isn't used in high-end server systems (and yet, advocates want Linux to do all of these tasks, and rule the hand-held market as well). And so, we get tests like this, Win2K vs. Linux, when really, what we should be getting is Win2K vs. Solaris (which I'm quite confident would blow Win2K out of the water).
Does Linux really want to compete at the levels of AIX and Solaris?
No, they want to compete with Windows. Windows is the enemy. Sound the alarms, and when Windows does something better than Linux, something is seriously wrong with the world (or so they would have you believe). Perhaps what would be a better suite of tests for Linux is one which isn't a comparison test at all, but rather one which looks for deficiencies so that people can start fixing them and quit debating about whether or not a comparison is valid.
Because that's what Linux advocates trump up. Ever since Linux became 'popular', advocates have been pitting it against the big bad evil Microsoft. Nevermind that until recently, Solaris was just as closed-source and dealt in the same underhanded tricks as Microsoft. Nevermind that they're two completely different types of operating systems aimed at two entirely different classes of people.
Perhaps a less biased way of saying this is "Because Windows is, arguably, the main competition for Linux. While AIX & Solaris are also viewed as competitors, due to Linux' current weakness in scalability, they are not considered direct competitors."
Now, that said, I'll respond by saying you're an idiot. Linux & Windows 2k ARE NOT designed for two different types of users. Both are designed for general use, high-end workstations low-to-mid end servers. In particular, in the context of the question, they are designed for EXACTLY the same market.
As far as AIX & Solaris, they are also the competition. But, most people who have the budget to run a high -end unix server have a reason to spend the money (Support, a boss that's an idiot, or a need for specialized capabilities or scalability that Linux & Windows don't allow). Linux is rapidly advancing, & is beginning to address the last two issues (scalability & features), but at present it's hard to directly compare Linux to some of the commercial Unixes. And of course, you again need to consider the context. Since the question was specifically in response to a benchmark comparing Linux to Win2k, why would you even expect AIX or Solaris to be brought up?
diff -u linux/net/ipv4/tcp.c:1.1.1.6
@@ -1575,7 +1575,7 @@
add_wait_queue(sk->sleep, &wait);
for (;;) {
- current->state = TASK_INTERRUPTIBLE;
+ current->state = TASK_INTERRUPTIBLE | TASK_WAKE_ONE;
Offhand, it looks like that particular change isn't in Red Hat 6.1 or 6.2. I don't know whether this would affect ServerBench performance, though. It's hard to tell without looking at the source.
Without going to far into it, I remember discussing a lot of this stuff with the guys doing those tests at the time. Those (fairly) low-down tweaks were attempts to see if the Linux setup was tripping up on something obvious (e.g. trying to auto-negotiate on the NIC) and whether it could be speeded up. That was because everyone was really quite shocked at the figures coming out, and went to some trouble -- including talking to Red Hat -- to attempt to eliminate configuration issues and the like, because everyone thought the numbers looked odd. But after a lot of effort, they still looked odd.
And you don't (or shouldn't) 'root' for any of the platforms you're testing when you benchmark. You go to a reasonable amount of trouble to make sure that you are testing what you think you are (and not some config hiccup that's hamstringing the results). But having done that, you sometimes still get a surprise. That's what happened here.
The server was a dual-proc machine. Win2k has a multi-threaded TCP/IP stack, linux 2.2.x doesn't. That probably accounts for most of the issue right there - at around 24 users, the single processor limitation of the Linux TCP/IP stack was reached, and the Win2k mahcine just split the load up.
Of course, IANALOLT (I am not Alan Cox or Linus Torvalds), but it seems the most likely explanation to me...
---- I made the Kessel Run in under 11 parsecs.
This just went up on the TPC website Monday, there is a monster leader in transaction processing price/performance and that is:
You will not believe this unless you see it!
Yes - but check out the hardware. 32 four-way pentium Xeon's, and over a terabyte of disc space, and an obscene amount of RAM. That is not a standard setup, although it was built with standard parts (trust me - I know the team which built it). That is not to say that the DB2 team isn't extremely pleased with this result :-)
Just because it's running on Windows 2000 does not automatically mean that there might not be better choices for an OS to support this benchmark. It's not even entirely clear to me that Windows NT might not have been faster here, given the benchmarks which MS put out on their own website showing that Windows 2000 does better in limited memory, but is worse than NT above 128MB (and these machines had a lot more than that). Remember that DB2 UDB has a shared-nothing architecture which that it scales extremely well and is additionally capable of using raw devices so the OS in question may not have a big impact on performance. And DB2 runs on most platforms out there, from OS/2, AIX, HP-UX, Solaris, Linux, Windows 9x/NT/2000, SGI, SCO, Dynix and various 64 bit platforms as well.
Of course, it would be nice to have some side-by-side benchmarks of DB2 UDB on Windows NT/2000 and DB2 UDB on Linux. There will almost certainly be some benchmarks on Linux sooner or later - since IBM has made Linux available for all its machines, it makes sense to publicise the performance of its flagship DB product on Linux as well.
Cheers,
Toby Haynes
P.S. I work on DB2 UDB development.
Anything I post is strictly my own thoughts and doesn't necessarily have anything to do with the opinions of IBM.
"Each was tested on it's own network, with 2 subnets of 24 servers, the windows network consisted of 48 PII's, whereas the linux network had the added advantage of having a Cray Supercomputer making requests at full charge on the 24th node of the first subnet..."
Eh...
I hate people talking how 2.4 will fix everything, 2.2 surely didn't.
Where did I say that 2.4 will fix everything?
I said that there is a specific problem, known in 2.2 that has turned up before, that is a potential explanation for this bad result.
There are other known (and fixed) scheduler problems.
Encountering any combination of these in 2.2 benchmarks is to be expected. Don't make these out to be more or less than indications that 2.2 had some obvious room for improvement.
I am sure that 2.4 will have more problems. However many problems that turned up in benchmarking 2.2 have been fixed (because they turned up in benchmarking 2.2), and preliminary benchmarks of 2.4 (eg the recent SpecWeb result where it nearly tripled Windows 2000 on a similar 4 CPU box) indicate this.
Now will 2.4 be ready for the enterprise, as they like to say? Not really. First of all until it has been through a few point releases, I would expect some significant bugs. (To be expected in any software.) Aside from that issue, it lacks many managability tools, a volume manager, more work needs to be done on failover, journaling filesystems are needed, etc. I have been convinced by Larry McVoy's argument that further work on SMP is not needed, NUMA (done through clustering and virtual operating systems) is.
These are known problems. Work is being done on them. However there will be room for complaint about Linux vs more mature systems for some time to come. However problems are getting solved, and Linux is moving up the food chain, fast.
Regards,
Ben
My usual seat in the cluetrain is at A HREF="http://pub4.ezboard.com/biwethey.ht
An immediate thought.
The "thundering herd" problem that was identified in Mindcraft and fixed in 2.4, isn't that still present in RedHat 6.1? (BTW calling it "Linux 6.1" really irritated me.) That could explain a sudden drop-off. It is not a problem, not a problem, then suddenly becomes a problem and as soon as you get a slow-down, you get a real traffic jam.
Just guessing...
Ben
My usual seat in the cluetrain is at A HREF="http://pub4.ezboard.com/biwethey.ht
I know this sounds strange but when I'm looking at designing a high transaction application or site I don't even LOOK at Windows or Linux. Does it suprise me that Linux doesn't scale to the enterprise market ? No, its written by individuals for lowish demand systems that they require, rather than by Company A who is implementing for Company B something that will cost several million pounds of development.
These sort of tests are IMO unfair to Linux. Should you use NT/W2K or Linux for your high transaction application/site ? The choice is more normally "Should I use True64, AIX or Solaris ?".
Linux works great for me as a webserver, as a client who takes a limited number of hits at a cheap price. If you want to scale you buy more boxes.
On the back end use a large end server with lots of RAM that has a massive IO throughput.
Does Linux really want to compete at the levels of AIX and Solaris ? Why not go for the niche, of cheap, reliable, and easy to scale horizontally.
An Eye for an Eye will make the whole world blind - Gandhi
Okay, they set the ethercard to full duplex and increased the queue depth on the scsi card. Fine -- makes sense. Stopped the "atime" updates. Makes sense.
... interesting. We're developing a new filesystem, and ended up ignoring bdflush completely to get good performance. Here's what those values mean:
/usr/src/linux/Documentation/filesystems/proc.txt:
/proc/sys/vm/bdflush
... they seem to have changed one of the "dummy" values... wonder why? Other than that, they appear to have increased the interval at which bdflush runs, meaning more stuff is hanging around in memory. It may be that at 24 clients, bdflush is banging on the filesystem too much. I would loveto see a graph of disk activity inluded with the results. Sometimes Linux will go through a silent-storm-silent-storm cycle as bdflush runs on a busy system. It would be interesting to see how a journaled filesystem would perform. I think Reiser does his own buffer-flushing rather than relying on bdflush runs to do it, meaning he has finer control over it. It would also be interesting to see this test run on FreeBSD, which does a better job keeping the disks busy.
/usr/src/linux/Documentation/IRQ-affinity.txt ... describes how to have specific CPUs handle specific IRQs -- like the mindcraft tests did with NT.
But they also did this:
echo 100 5000 640 2560 150 30000 5000 1884 2 >/proc/sys/vm/bdflush
From
Table 2-2: Parameters in
Value (default/tweaked)
nfract (40/100)
Percentage of buffer cache dirty to activate bdflush
ndirty (500/5000)
Maximum number of dirty blocks to write out per wake-cycle
nrefill (64/640)
Number of clean buffers to try to obtain each time we call refill
nref_dirt (256/2560)
buffer threshold for activating bdflush when trying to refill buffers.
dummy (500/150)
Unused
age_buffer (3000/30000)
Time for normal buffer to age before we flush it
age_super (500/5000)
Time for superblock to age before we flush it
dummy (1884/1884)
Unused
dummy (2/2)
Unused
Tweakers may also be interested in reading
ServerBench is not available in source code, and the testing was done by ZDNet. From what i know about ServerBench it uses a threaded IO model on NT, but a fork/process model on Linux. The Linux 'solution' is coded by ZDNet, with no possibility from us to influence/comment the design and approach used at all. Even under these circumstances we expect the 2.4 Linux kernel to perform significantly better in ServerBench than 2.2 kernels. The 2.2.1x (and late 2.3.x) kernels had some VM problems, and with increasing VM utilization (more clients) this problem could have been triggered.
SPECweb99 OTOH is a standardized benchmark with full source-code access (ServerBench are closed binaries), so all SPECweb99 implementational details are visible.
Nevertheless it's technically possible that ServerBench triggers performance bugs in Linux - we'd love to see the source to fix those bugs ASAP, if they are still present in 2.4.