High Performance Network Applications

An Anonymous Coward sent in this: "An article over at SysAdmin magazine seeks the truth while comparing network application performance under RH Linux, Solaris x86, FreeBSD 4.2, and Windows 2000. I'm a little suspicious of the writer's results, but you be the judge."

Comments

Re:This benchmark is baloney

[edhall]

Agreed -- it's been a long time since I've seen a "benchmark" as poor as this one. But I don't think Windows was treated any more poorly than the other OSes. It wasn't a fair test of any of them.

The "tuning" for the Unix systems consisted in bumping up the maximum number of file descriptors. That's it. The FreeBSD system in particular was left completely mistuned and clearly running out of socket resources -- they report that it was logging errors but seem entirely ignorant of what those errors were (beyond their being load-related) and how to correct them.

Polling is hardly the best system interface for multiplexing TCP connections on either Windows or FreeBSD. As you mention, completion ports are best for Windows. Kqueue is best for FreeBSD. It just happens that polling is used in the crappy commercial SPAM program they "benchmarked". (All the OSes support scatter/gather, BTW, so you can't claim Windows was treated unfairly by its omission.)

None of the systems were testing in a way that shows their actual capabilities. The article is just a thinly disguised commercial for a (barely-)cross-platform "bulk email" product.

-Ed

This benchmark is baloney

[spectecjr]

I'm sorry, but I can't see how this is a valid benchmark.

"As a real-world test, we measured how quickly email could be sent using our MailEngine software. MailEngine is an email delivery server, ships on all the tested platforms (plus on Solaris for Sparc), and uses an asynchronous architecture (with non-blocking TCP/IP using
the poll () system call). So that email was not actually delivered to our 200,000-member test list, we ran MailEngine in test mode. In this mode, MailEngine performs all the steps of sending mail, but sends the RSET command instead of the DATA command at the last moment. The SMTP connection is then QUIT, and no email is
delivered to the recipient. Our workload consisted of a single message being delivered to 200,000 distinct email addresses spread across 9113 domains. Because the same message was queued in memory for every recipient, disk I/O was not a significant factor. We slowly raised the number of simultaneous connections to see how the increased load altered performance."

Nice! So in other words, they used straight BSD sockets for their
implementation - which is NOT the way to get performance from Windows. You
need to use:

1. Asynchronous, Event based socket handling.
2. Completion ports.
3. Scatter/Gather buffering.

Polling is lousy no matter what way you do it. You'll lose most of your
performance spent going round a small loop.

Similarly you can infer that they used straight malloc() for their memory
handling, and most likely file handling - again very lousy
performance-wise on windows compared to the alternatives, such as
VirtualAlloc, CreateFile(), scatter-gather file handling and more.

As for the second test, we can guess (from their comments) that they're
using straight C++/C file operations under windows instead of tuning them to
the architecture, so of course performance is going to be lousy -- they're
benchmarking Microsoft's C runtime implementation, nothing more, nothing
less.

Also note that:
1. They don't provide details of which compiler they're using.

2. They don't provide details of the actual benchmark code for test 2.

3. They only tuned the Linux, FreeBSD and Solaris setups -- they should have
tuned Win2k server as well.

Sheesh. Talk about a crappy way to benchmark.

Simon

Ignorant of how to program Win2k

[throx]

The method used here for programming Windows 2000 is almost certain to guarantee slow results. Assuming he's written his code to use select() or even WaitForSingleObject() then he's signifiantly slowing down the system.

If you want to write high performance socket applications on Windows you MUST use I/O completion ports (something this article failed to mention at all). Most high load applications I've written using sockets have shown a 50% to 100% improvement in throughput for the same CPU load when switching to I/O Completion ports from a tradition (Unix style) asyncronous I/O model.

I'm not saying in this case that Win2k would beat Linux, just that the tests were skewed by the author's inadequate knowledge of writing high performance code on Windows 2000.

FreeBSD performance

[nsayer]

It's clear from their comments that they did not turn on Softupdates on the filesystems when they set up their FreeBSD machine for the testing. It's no wonder that they found disk I/O to be slower on FreeBSD, therefore.

Traditionally, Linux has traded speed for safety in filesystem meta data handling. FreeBSD has always refused to do so, insisting that metadata be updated synchronously. With softupdates, the metadata is cached, but the cache is flushed in the right order. The upshot is that you get the speed and the safety.

In short (too late), I am sure that their opinion of FreeBSD would improve markedly if they would set it up properly.

From what I see, just about every other OS represented has a defender saying exactly the same thing. That doesn't speak well for the thoroughness of the testing. I'll leave it at that.