Intel and BlueArc Set New Mail Server Record

louismg writes "With e-mail traffic continuing to explode, Intel and BlueArc announced this morning that the two companies have set a new SPECmail benchmark record in cooperation with CommuniGate Pro, offering a solution that can serve 30 million messages per day - 67% ahead of the previous record, owned by Sun Microsystems. Rather than clustering a lot of smaller servers together, large ISPs can now use fewer systems to handle massive traffic load."

Comments

The Aliens, there in it with the spammers

[Anonymous Coward]

And if there was no spam, I doubt there would be such a need for these machines.

Old Koreans Rejoice!

[LiquidCoooled]

There will be partying in the care homes when this news spreads (via email of course).

I always thought gamers drove performance.

[mikeophile]

Turns out it's really spammers.

Re:Oh oh .....

[LiquidCoooled]

Constructing templated emails and blasting them out to multiple servers doesn't require a full mail server.
They are better using customised software which doesn't care about inbound mail.

Re:Oh oh .....

[LiquidCoooled]

Shhhhhhhhhhhhhhhhhhhhhhhhhhh!

I don't want everyone to know.

By the way, are you interested in some viagra?

Sun Solaris 10 x86

[Alex]

CommuniGate Pro Dynamic Cluster - Backend Servers (4 systems)
Software
CommuniGate Pro: CommuniGate Pro v4.3.6 Operating System: Sun Solaris 10 x86

CommuniGate Pro Dynamic Cluster - Frontend Servers (5 systems)
Software
CommuniGate Pro: CommuniGate Pro v4.3.6 Operating System: Sun Solaris 10 x86

Redundant?

[Puls4r]

No, not the post.

Isn't part of the allure of smaller systems handling the specifically to get away from large dedicated systems that aren't nearly as reliable?

By now, google should have taught the world something - distributed computing with small cheap specced systems that can each be swapped with multiple redundancy is the way to offer both uptime, speed, and be cost effective.

It's nearly identical to the "lean" manufacturing techniques pioneered by the Japanese. Small cells that can increase or decrease output based on the amount of workers (systems) that are working that day. Very flexible.

After all, it's a COMPUTER.... do you really want it dedicated to just email, or can we use it for other tasks in the downtime.

Re:Redundant?

[Jeff DeMaagd]

Isn't part of the allure of smaller systems handling the specifically to get away from large dedicated systems that aren't nearly as reliable?

The allure of small systems is because of COST, not reliability. Large systems are and can be very reliable. Using consumer commodity computer parts by themselves is more likely to be less reliable, but if you set up failover clusters, then you get a cheaper overall system that is as reliable.

Re:Redundant?

[saider]

I did a little "reliability analysis" for some telecom equipment a while ago, so I am semi-informed (not a guru).

There are two terms that often get interchanged, when they shouldn't. Reliability is the ability of the system to run without repairs. Availability is the ability of the system to do its job.

So the large monolithic system can be built out of very good (and expensive) components that do not fail as much as commodity hardware. This will lead to fewer failures and better reliability.

The commodity hardware can be arranged so that redundancy ensures that if on component fails, then another will take its load. Since the damaged component needs to be serviced, the reliability is lower, although the availiability of the system is the same.

Reliability is used by planners to determine the labor costs in keeping a system running. Availability is used by planners to make uptime predictions and to take measures to provide a certian level of service.

Two similiar numbers that are used for different purposes.

Doesn't this create SPF?

[Anonymous Coward]

Using massive systems for handling mail invites a single point of failure (SPF), whereas using clusters of smaller systems for the same amount of money gives failover capability.

Of course, ISPs won't realize this.

Not that great

[Matts]

I didn't even know there was a SPECmail, but this figure doesn't seem too outstanding to me.

Firstly I assume this is just a raw delivery setup - no spam or virus filtering. You'd be amazed how much of a difference this makes to any real world setup.

Secondly, apache.org does over 2 million mails a day on a dual 2.4Ghz Xeon using an SMTP server written in Perl. And that's with full anti-virus (clamav) and lots of different anti-spam measures including SpamAssassin (which is known to be slow - I know because I used to be one of the developers).

I also know of commercial setups doing over 50m (legit, well - mostly) mails a day. Using an SMTP Server designed with performance in mind. Perhaps they should submit for SPECmail ;-)

So 30 million doesn't seem terribly amazing to me. Perhaps Communigate Pro isn't a very fast mail server.

Re:Not that great

[antic]

Probably worth noting that the Louis Gray who submitted this story is BlueArc's Corporate Communications Manager.

I thought that the blurb seemed a bit too slick to have come from anywhere but the company themselves. I hope there's no dodgy reason that Louis used a mac.com email address to submit the story instead of their work account.

Re:Not that great

[Matts]

Replying to myself...

So reading the full disclosure they used 4 quad xeons. That's 16 CPUs. Compared to apache.org's 2.

So using a pure perl SMTPD you can have this kind of throughput (~1m mails per day per CPU) with spam and virus filtering enabled.

No, I am not impressed with this benchmark.

Right, but...

[thc69]

...imagine a beowulf cluster of these...

*sigh*

On an NFS message store

[Temkin]

Interestingly enough, they set their record using a message store mounted on NFS. It had 140 FC/AL attached disks, and 14Gb of RAM.

Virtually every file handle an MTA writes to is opened "O_SYNC". One of the quickest ways to make Sendmail or other common MTA's go fast is to mount their delivery queues on a solid state disk. I'm betting this disk array is turning around the queues without ever committing the data to the platters. (Not that there's anything wrong with this...) I am left wondering if there isn't some bit if NFS trickery not reported in the config.

But looking at the Sun entry, the old record was set using 2 year old software, and a much smaller disk configuration. Sun will probably update their entry in the near future, just to reclaim the crown. Email is much more an I/O problem than a CPU problem. Sun used to push their mail server on much larger HW, but most ISP's don't want to buy big boxes these days. The small to medium sized boxes, connected to a SAN are more cost effective, permit redundancy and easier maintenance.

Re:On an NFS message store

[Tet]

But looking at the Sun entry, the old record was set using 2 year old software, and a much smaller disk configuration.

Indeed. It doesn't strike me as being a particularly impressive record when there are only a total of 18 entries submitted, and most of them are 3 or more years old. I'm sure that I could quite easily come up with a system capable of beating the previous record for a reasonable cost simply by using modern hardware and minimal configuration tweaking.

Re:On an NFS message store

[puppetluva]

We evaluated BlueArc. . . we didn't choose it because it was "too new" in the market, but its network filer has unbelievable performance (Note: I don't work for BlueArc and it makes me a little queasy to support a commercial product).

BlueArc does all of their fileserving via microcoded hardware. Instead of using a plain old OS to build the fileserver, they do it in hardware modules dedicated to the task. This means that there is no OS overhead and the different modules (TCP/IP, CIFS, NFS, iSCSI, etc) have their own memory and work together in a pipeline. The performance they get from this is really sick. (Think one Bluearc machine is 3 times faster than a cluster of the highest end Netapps).

Behind the hardware front-end, you have a hardware raid-system that is long on cache (It's actually the raid unit from Engenio -- formerly LSI) with shelves configured with raid 5. Bluearc actually treats all of these raid-5 shelves as disks that it does raid-1 across. This gives you redundancy and double-striping in hardware from end to end. Did I mention that it was fast? It is - just check out specs.org to see how fast the thing is.

Speed vs. functionality

[IGnatius T Foobar]

The speed of any MTA is going to be determined largely by how much work is being performed when each message is submitted. The fastest MTA, therefore, is going to be the one that does the least amount of processing.

• How about that spiffy big Postfix or Sendmail box you've got sitting out there, whose sole purpose in life is to act as a relay? Sure, it'll process millions of messages per day. It doesn't have to do much.
• What if you're running a virus checker like ClamAV or a spam checker like SpamAssassin? Those take up CPU cycles. Sure, delivery is slower, but value was added.
• What if your back end mail system is something like the Citadel groupware platform, where MIME content drives an event handler system? Again, delivery is impacted, but the functionality of the system depends on it.
• What if your org has a global directory and your mail hub is responsible for making complex routing decisions for each message? Again, delivery is impacted; it'll be slower than the mega-fast-box, but mail won't be delivered correctly otherwise!

So you see, it's not always just about raw speed. And besides, next year's hardware will be faster anyway :)

And how does this impact Australia?

[AnotherEscobar]

Thats all I want to know

Re:Not that great - try RTFA

[MarsBar]

This isn't just a mail relay, this is (from spec's site):

A standardized mail server benchmark designed to measure a system's ability to act as a mail server servicing email requests, based on the Internet standard protocols SMTP and POP3. The benchmark characterizes throughput and response time of a mailserver system under test with realistic network connections, disk storage, and client workloads.

So that includes users connecting, picking up email, deleting from their data store etc etc etc.

Disclaimer: I have two friends who work for Bluearc but have no other connection to the company

Funny math.

[Spazmania]

BlueArc's Titan sustained a performance level of 12,500 SPECmail messages per minute, or the equivalent of two and a half million SPECmail users, sending 30 million e-mail messages per day.

The math seems a little off...

12,500 messages/minute * 60 minutes/hour * 24 hours/day = 18M messages/day, not 30M.

That having been said, CGPro is fast as all heck so I can believe it topped the previous record.

The tester responds...

[thomtunes]

Just thought I'd add a few details and address some of the questions here. My name is Thom O'Connor and work for CommuniGate Systems (CGS), and was the one who put together and ran these tests - you can (mostly) verify this by looking at the comments in the source on the results page.

First off, on the SPECmail test itself. SPECmail is a standardized test (the only one I'm aware of for email) that attempts to closely regulate a level playing field for measuring email performance. It is critical to understand that this is not just measuring SMTP. The 30 million message a day text is a little vague, but it is important that this includes a distribution of delivery, relayed, and retrieved email. Sure, anyone can just relay many millions of messages an hour.

SPECmail does POP and SMTP, so the test measures not just MTA behaviour but also local delivery and then retrieval of the messages. The SPECmail test also uses Quality of Service (QOS) measurements such that a message injected via SMTP to the system MTAs (the CommuniGate Pro Frontend servers in this diagram) must then be delivered locally into the users' account, then be retrieved within 60 seconds. Satisfying the QOS criteria during the benchmark is often the most difficult part.

So, SPECmail itself just does POP and SMTP, which is a little 1990s I agree, but SPEC is coming out with a SPECimap test in the near future, and CGS is also very interested in seeing a SPEC VoIP/SIP test for measuring CommuniGate Pro's Real-Time capabilities.

A few others questions I've seen raised here:

1. The CommuniGate Pro Dynamic Cluster described in this test is fully and completely appropriate for production use in all aspects. In fact, if you're running a 2+ million user ISP on a CommuniGate Pro Dynamic Cluster, we'd recommend you to use these results as a guide for your architecture (although load balancers should be added to the gateway point for all inbound connections). In fact, CGS has ISP customers running architectures which match the layout of the described system almost exactly. All systems in the Cluster service all accounts - you could lose 4 Frontend Servers and 3 Backend Servers, and all users could still access their email (albeit with decreased capacity).

2. HyperThreading was disabled in the BIOS because the downloadable Solaris 10 x86 operating system would not (yet?) support the Intel x86_64 Potomoc chipset properly. That said, on top of the recent security vulnerabilities on the topic, we have also discovered miscellaneous threading and even NFS issues related to having HyperThreading enabled on Linux 2.6, FreeBSD 5.4, and Solaris 10 x86 systems.

3. On NFS...NFS is used safely and securely in this test. The integrity of data storage is one of the major criteria that the SPEC organization closely evaluates when reviewing a SPECmail submittal. Obviously, there are many ways to cheat and/or cut corners using Solid State Disks, unsafe RAM for message queueing, and other techniques that you would never want to use on your production message system. However, the test described here was performed using a standard (albeit excellent) BlueArc Titan Storage System with write caching only in NRRAM and using proper mount options and layout for security, redundancy, and data integrity.

Hope this clears up any misconceptions. Obviously, I'm clearly biased about the work here, but assembling and then passing a SPECmail test of this size is a gigantic effort. If anyone thinks