Slashdot Mirror
HP Publishs First Linux TPC-C Benchmarks
The first ever official TPC-C benchmark on a Linux system has been published.
This was run on a cluster of 32 HP servers with Intel Xeon CPUs, running Redhat Linux and Oracle RDBMS. The system had over 18 terabytes of storage, and cost over 2 million US dollars. Performance was higher than a similar system running on MS Windows.
Having a Beowulf cluster...
Oh never mind.
Is about 1,100 difference in the TPC-C that big of a deal? I dont know what TPC-C measures so for all I know it could be the difference between a yugo and a delorean.
One wonders where the slowdown lies. It would be nice to find out if it lies withing the Win32 version of Oracle or within Windows. Seeing as Windows does have the top score in the Clustered TPC-C results, I'm leaning more towards Oracle being the slow one. (everyone else here would disagree no doubt)
If you look at the top results in a price/performance comparison, Windows 2000/SQL server are still winning.e sults.asp?resulttype=cluster&version=5
http://www.tpc.org/tpcc/results/tpcc_price_perf_r
It's interesting, though, that Red Hat had the cheapest Oracle implementation.
This just in: ditors at Slashdot ar no longr prmittd to us th lttr '' in articl titls!
On the other hand, the difference in performance is 17.21 vs. 18.46, i.e. approximately 6%. I think that the result is inconclusive (except that Linux can complete in this area at all).
Some more TPC-C benchmarks of big iron kit... found here
Jinx.
Linux cluster:
Windows cluster:
Note that the number of clients in the windows tests is higher 24 instead of 16), with smaller CPU's. Also, the server's aren't identical.
Besides from the small differences in setup, it's plain that hardware-costs greatly outnumber software costs. Yeah, linux has a small bit more performance (less than 1%) for a bit lower price (6%) but these aren't real shocking numbers. Of course, I'll get flamed for not bashing microsoft, but the difference really isn't that big.
--
If code was hard to write, it should be hard to read
Look at the availability date: 03/05/03. That's six months from now.
If you really want to compare 2 operating systems, you should rule out as many factors as possible. The windows cluster was set up differently (it had less memory, for example). I would guess that on the same hardware, the difference would be even less...
yeah, but it says the clients on the windows test had 48 cpus each!! while the linux clients only had 2 :)
(pretty sure its a typo.. but thats what it says)
I do not hazard a guess, but it is surely interesting what Linux can do best, instead of comparing it to Windows, only.
As many other people have pointed out, the system configuration was vastly different despite being 'similar' - yes, I want to see Linux out perform Windows but I would rather see it run on identical systems! When benchmarks are weighted like this, it just makes Linux look like the inferior product which is not the kind of image it should be getting for itself!
"Hey! Unless this is a nude love-in, get the hell off my property!!"
It is a benchmark, after all ;-)
The nice thing about statistics, benchmarks, or anything else that tries to assign a number with some sort of value to something that's much more complicated is there are so many factors in the complicated thing, and thus ways to fsck up. Therefore, there are equally many ways to fudge.
There are 32 xeon processors used, but only 16 computers.
Sorry that this might be off topic and cheerleading (right place though).
GNU/Linux impresses me a great deal. I've been running it for a few years now, mainly at home and in some positions when I can get away with it and its easy to take it for granted. But its articles like this that make me step back and think about what as really been achived by a bunch of mainly unpaid developers who provide all their source code for free.
Sure there a few places where Windows is more suitable but everyday, these few places get fewer.
So to any Linux developers and supporters who read this, thank you all.
A journey of a thousand miles starts with a brutal anal raping at airport security
Clients are dual proc machines, not the server.
I read the article, just not well
Nope, I agree. The power of open source solutions performance-wise is not that it's faster out of the box, it's that you have room to do all sorts of cool tweaking.
I like Redhat, but in my past experience, redhat isn't the best alternative if you want to do some serious tweaking.You can recompile most stuff to optimize it, find the best FS for your needs, tweak caching of stuff, tweak DB configuration, etc...
There is often more performance to be gained from this than you'd think.
-- Black holes are, where God is dividing by zero.
Of course, I'll get flamed for not bashing microsoft
:)
Don't be silly.
From this FAQ you'll see:
In general, TPC benchmarks are system-wide benchmarks, encompassing almost all cost dimensions of an entire system environment the user might purchase, including terminals, communications equipment, software (transaction monitors and database software), computer system or host, backup storage, and three years maintenance cost. Therefore, if the total system cost is $859,100 and the throughput is 1562 tpmC, the price/performance is derived by taking the price of the entire system ($859,100) divided by the performance (1562 tpmC), which equals $550 per tpmC.
Most people would focus on the hardware cost, but in reality the highlighted maintenance cost took the precedence.
Most midrange UNIX server has outragous maintenance cost. The maintenance cost of a UNIX server in the third year could be exceeding the cost of the hardware itself. It's due to the fact that older parts are difficult to find, thus make maintaining older servers more difficult. Besides, they really want to cut older production lines in favor of newer servers production.
x86 platform is known to have flat and lower maintenance cost, due to the low cost hardware and high compability with older hardware, i.e. older parts can be found easily. That's why Microsoft could easily beat the TPC pissing races.
Doomsday finally comes to Microsoft when Linux is entering the database market. Although at this moment big corps are still offering Linux maintenance with cost comparable to UNIX package, that's not surprising when Linux engineers are not as abandon as MCSE. However, it'll not be the case in the future. I think Microsoft would eventaully lose this pissing race in the long run.
>Windows is not interesting to me. Its not penetrating the back end, even though Microsoft marketing pretends it can.
heh heh. you said "penetrating the back end".
There was a followup message on the lkml from hp discussing how these numbers can improved even further, you can see it at http://marc.theaimsgroup.com/?l=linux-kernel&m=103 237327405644&w=2
The Windows TPC benchmark was on last years hardware (IBM xSeries 370, released last year).
The Oracle TPC benchmark is on next years hardware (HP ProLiant DL580R - not available till May 2003).
Crowing about how performance is higher under Linux is FUD. It's not a fair comparison. Or didn't the story submitter understand that hardware always affects performance?
The reason to use Linux is all aspects of its openness and compatibility with other systems. With Linux, you aren't locked into a single vendor. You use tools and APIs that have been around for nearly two decades and are available, in multiple implementations, from dozens of vendors. And you control how you upgrade, when you upgrade, and what path you follow with the software. And if you don't like Linux anymore, you can switch to any of a dozen other, compatible platforms.
With Windows, you are locked into a single, proprietary implementation and Microsoft has you by the proverbial precious body parts; there is no other vendor you can get a compatible implementation of Windows or all the Windows libraries from. Every couple of years, Microsoft completely changes their computing paradigms to ape what they perceive is a threat from some other company, and when the threat is gone, they just drop the initiative and move on to the next thing.
You can get stability buy paying a premium to a company like IBM, which is committed to providing it, or through open systems available from multiple vendors or open source, where you control your future. But building a large, long-lived infrastructure on Microsoft platforms is a costly folly--the company has proven that they will change approach every couple of years and that they will force their customers to move along.
Clustered TPC-C benchmarks are of dubious value. I've yet to encounter any business that implements an OLTP system like this and it's for a good reason. Clusters of this sort are very difficult to implement and operate. You basically end up having to physcally partition your database across dozens of servers!
.NET...it would be interesting to see how Linux compares.
a il .asp?id=102072201
What I would love to see is Linux TPC-C benchmark running on a 8 way machine. Particularly the IBM x440. It scored 91,000 + tpmCs running Windows
http://www.tpc.org/tpcc/results/tpcc_result_det
What I have to back this up is experience. I have seen alot of things, including (but not limited to) databases perform alot better after extensive tweaking, and recompilation of things like common libraries.
And although this might be fully possible to achieve with Redhat, I wouldn't exactly say that Redhat is very optimized out-of-the-box. So it would require alot more work to do the things _I_ want to do.
And btw - what's wrong with recompiling 'ls' with the '-march=i686 -mcpu=i686 -O3 -ffast-math -fstrength-reduce -fthread-jumps -fschedule-insns2 -fexpensive-optimizations -fomit-frame-pointer' flags? =P
-- Black holes are, where God is dividing by zero.
In a 2+ million dollar system software cost is a very small percentage of the total cost.
However, in reality, a lot companies have $10,000 or even $5,000 DB servers.
When you factor the cost of Windows into these systems the difference in Price/Performance is much greater than 6 percent.
As far as price/performance it was in the middle of the top ten. I think if they had used PostgreSQL (or maybe SapDB, dunno haven't used it) instead of Oracle it could have been higher in the results.
I have done an extensive study on the effect of compiler flags on performance and I can tell you that to begin with some of them have near zero effect, some others interact badly (ie they speed things up when used separately but slow them down when used together), that some of the flags you give are already enabled when you use -O2 (BTW the -Ox flags are a just convenient way to enable a miriad of "minor falgs"). That -O3 is a double edged sword since in gcc 2.x it is equivalent: to "-O2 -finline-functions" thus making the code faster on small programs (sinced there is no the overhead of function call) but bigger (and thus it could slow large programs if it makes you run of TLB entries, cache or RAM), and it also slows things down when the program (eg the Linux kernel) has already had its significant functions inlined by the programmer (if you use -O3 on it the compiler will inline functions who are small but that the programmer didn't inline because they are rarely called, so you get no benefit but the enlarged size could make that two functions who were in the same page are no longer and thus you get TLB faults). Finally due to bugs on all versions of gcc's optimizer there are significant cases where -march=i686 will be significantly slower on the 686 family (PPro, PII, PIII, Celeron) than using plain -march=i386. Ah, it is not a foregeone conclusion that the Athlon behaves better with a program compiled for the 686 than for the 386: due to differences in architecture and timetable all the 686 family runs more than 10% faster on programs compiled for the 386 than on programs compiled for the Pentium.
Now that we are at it: RedHat 7.x are compiled with -mcpu=i686 (that means: optimize for the 686 family but using only 386 instruction so it runs everywhere). Using -march=i686 like you do only gives a 2% performance increase. For programs who have parts written in assembler (kernel, glibc, ssl) RedHat provides versions for the 686 and the Athlon and these are compiled with -march=i686 and -march=k7 respectively (ie full optimization).
Nearly forgot: the effect of the minor flags not included in -02 (except for -finline-functions who is part of -O3) is quite minimal when comparing the best combo of them to plain -O2 or -O3. From a memory it was couple percent at most. defintely not worth the risk since exotic combos are not so well tested and you have a chance of gcc generating wrong code.
Life is never easy when you're posting on /. ;)
-- Black holes are, where God is dividing by zero.
> Yeah, linux has a small bit more performance (less than 1%) for a bit lower price (6%) but these aren't real shocking numbers.
Huh? The cost of the Windows software adds $150K more. That is a lot of money. Think of it this way: if I have two laptops studded with diamonds that cost 2.3 mil each (before the OS is installed), and the OS for one is free while the OS for the other is 150K - the latter seems very expensive when you look at the software budget
I think that you should break out the costs of these systems and look at the hardware and software seperately. "The systems cost the same in hardware, but there is a $150k difference in software." is a much cleaner analysis.
TPC suffer from arse-elbow syndrome:
... and three years maintenance cost.
... maintenance costs over 5 years.
http://www.tpc.org/tpcc/faq.asp :
http://www.tpc.org/tpcc/detail.asp :
THL.
Keeping
It happens every day! All you have to do to see it is take whatever M$ junk you are using and replace the software. All the PitA you see on a desktop with poor security and stability, obscure and changing protocals, propriatory data formats, unmodifiable applications, ad nauseum, translate.
We have at least one multi-processor Dell "server" where I work and the folks responsible for it are not happy. Keeping it up has been a quality of life killer. The problem is not in the hardware.
DMCA, Hollings, Palladium. What might have sounded like paranoia is now common sense.
Make it 4 then. hehe
More maintenance years would only cause the result of $/tpc-c more favour to x86 platform, no wonder why MSSQL servers almost dominate top ten of this rating.
Thanks for the info.
The Windows benchmark is running on a DL580 as well. That being said, your statement is valid. The systems are NOT identical. Click on the executive summary links at the bottom of each and you will see the detailed hardware specs for each.
Also, the clients are not only different in number, but the Linux clients are running a faster CPU as well (1400 Mhz vs. 1000 Mhz).
The only conclusion I think we can draw is that Linux plays "in the same ballpark" as Windows, performance-wise.
"This just in - Microsoft (MSFT) today promised it will inflate marketing claims by 18%-86% in Q2 of 2003, thereby shattering all hope for its competitors. After hearing the news, Sun Microsystems (SUNW) CEO Scott McNealy states that he's ready to pack it in."
OTOH, you could look at it this way: all other things being (relatively) equal, an OS developed by a bunch of guys around the world for free performed as well as an OS developed by a bunch of millionaires for $billions.
The best diplomat I know is a fully activated phaser bank.
-- Scotty.
I'm not bashing you, I just don't see your point.
If you get a car for 20000$ at dealer A and the same car for 19500$ at dealer B, will you say "Hey, it's just a small difference" and buy from dealer B?
Let's not forget, this is about database-servers here and both run the very same database (Oracle). The underlying OS is irrelevant, you don't have to run MS Office or "the Sims" on this thing.
So please tell us stupid Microsoft-bashers what is your point.
Also, migrating between databases is not *that* hard, it's all SQL after all.
But does PostgreSQL, Oracle's closest free competitor, scale to an 18 TB database under the loads they're testing?
Will I retire or break 10K?
The Register had this article yesterday where they added that the HP system is not available until next year (so how do they know the real cost) while the IBM-Microsoft system is a year old.
A new system is certainly going to cost less than an older one. Not true of cars unfortunately, but true of computers.
What would be nice is a true apples-to-apples test on the same hardware.
this is not a sig
Also, Microsoft pays no dividends (MSFT), and has a monopoly (most PCs shit with some Windows OS.
Whether or not that was a typo for shift it made me laugh.
Sorry, BSD and Stephen King are dead. Haven't you been reading Slashdot?
Oh, and that Elvis guy. He is really dead.
A better question: How fast would it run on a FreeBSD compiled just to do that particular job?
Almost. IANODBA (Oracle DBA) but I do have a very good understanding how MS SQL clustered/federated database systems work. There is a hardware difference - the server NICs - that would make a pretty large difference for MSSQL federations. The Redhat/Oracle cluster uses two 64-bit Gigabit NICs per server, and the MS/Oracle cluster uses a 64-bit dual 10/100 NIC. Both, of course, used appropriate switches. Now, to be fair, the WinOracle cluster used 1120 disks to host the data, vs 976 in the LinOracle scenario. If you're wondering, more disks/arms CAN result in better performance, and it's possible that is the case here. FWIW, we use a lot of the same hardware here - 4x2MBXeon900 DL580s and MSA1000 disk arrays. We also use 8x DL760s. All running Win2k AS / MSCS / MSSQL2k - not federated clusters, just HA.
meh.
Hardware replacement availability is rarely a factor, as a good service organization will have a way to get parts for anything they support. As an extreme example a fellow tech of mine about 4 years ago went onsite to a company that had a dead pc, it turned out it was an IBM PC XT with a dying MFM hdd 20MB I believe, the hdd had on it some customer terminal program for interfacing with an IBM mainframe back at corporate hq. He bandged the drive a couple of times on the table to get the motor to spin up and told the customer not to turn off the pc, he then put in a service request for a replacement drive, not really expecting it to be fulfilled. Low and behold our parts people found a vendor that had like 20 of the necessary drive in storage and the customer bought 2 just to make sure they had one in case the replacement died (probably in another 15 years or so).
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
That's the case for PC. For some highly propriatary hardware platform like HP's, almost every parts of the component are produced by their own. Say if there's one customer still chose to pay the maintenace cost of F-class server, which may be 10 times as much as the hardware cost, they still have to keep the production line producing those obsoleted parts. They just couldn't tell people "Sorry we don't produce this parts even though we took your money" can they? :)
Maybe due to the release schedule of RH AS 3.0?
Probably ivolved HP's kernel tweaks specific to the hardware that are not yet rolled into AS 2.1.
Not cheating... Just using the newer models which were available. It is unlikely that the Windows TPC benchmark could have used 16 - 1.4 GHz front-end clients using IIS to drive the cluster to the max. The Linux clients used Apache (with threads). It is very hard to do a 1 to 1 comparasion. The point is... RH Adavanced Server is the first version on Linux which provided this level of performance and could pass all the TPC audit requirements. This is not easy stuff to do... About 10DB below rocket science ;^)
No, if you read the report the cost of Windows adds about $14K more - and part of that is for the 8 additional client licenses since they tested the Windows system against 24 clients intead of the 16 used against Linux. You have to also consider the fact that the cost of hardware goes down over time, and that the Windows system was built 3.5 months before the Linux system - this can easily make up for the $150K, and then some.
Finally, it's not about the softare cost - it's about the bottom line: price/performance. These reports can't really be scientifically compared. One can speculate that Windows would have beat the price/performance ratio of the Linux system given an even playing field, but it's just that - speculation. The real point here is that Linux is _competing_ in the enterprise and has been taken seriously enough to actually get benchmarked.
There is no longer anything that can be done with computers that is nontrivial and clearly legal. -- Paul Phillips
The point he was trying to make was that the decision is not based on finances at this point. The performance AND cost difference is inconclusive because they are different configurations. If one completely blew the other out of the water, we could maybe claim a winner, but that simply didn't happen.
And the difference between the two system costs from an OS licence standpoint is far less (see the full report). To correct your analogy:
$20000 dealer A
$19900 dealer B
It's not based on money the initial purchase. It's based on the TCO, how it will integrate with your existing infrastructure, and whethor or not you prefer one of the other (for a variety of reasons).
There is no longer anything that can be done with computers that is nontrivial and clearly legal. -- Paul Phillips
Ok.. I may get myself in some trouble here... Please be kind.. Red Hat Advanced Server is the first version of Linux to meet all the TPC requirements and provide the performance level of this benchmark. Numerous open source performance technologies were incorporated, without which, the performance would be significantly below 100,000 tpmC. The 7x24 - 4-hour response, 3 year SW and HW maintenance is also required. The TPC audit (done by a certified TPC auditor on-site) requires the demonstration of a full system crash, as well as a single node, without losing a single committed transaction. The audit for TPC benchmarks take many days... I hope the Linux community doesn't miss the point that the TPC benchmarks are the toughest benchmarks to run and pass the audit. 138K tpmC will run 95% of the OLTP business environments today. This publication provides input to the many large corporations looking more closely at Linux in the "bet your business" application areas (databases). Although I agree that the absolute numbers of performance and price/performance may not be applicable directly to many real customer applications, it is a very fair environment (level playing field) to compare HW and SW performance and price/performance. This, IMHO, is a significant step forward for Linux in the enterprise.
You're right, that this doesn't show that Linux is faster than Windoz. What's important, however, is that some huge barriers to the adoption of Linux are coming down. There are many situations you can't sell into without a TPC benchmark. And getting one can cost over a million bucks and take several months. What's important is that HP is investing serious time, dollars, and management attention into Linux. We got a good number, and this will make it harder for IT not to take Linux seriously. (Rah, Rah!)
Also, the clients are not only different in number, but the Linux clients are running a faster CPU as well (1400 Mhz vs. 1000 Mhz).
But if you read the specs, there were 2 CPUs per Linux client, and 48 CPUs per Windows client.
--
"Outlook not so good." That magic 8-ball knows everything! I'll ask about Exchange Server next.
Is there some reason that the article mentions that the Linux solution "beats" the Windows solutions, but everywhere on the site the Windows servers (whether it be IBM's DB2 or MS-SQL) are the leaders of the pack many times over?
That article link is like something a politician would say. Do you realize that Oracle is optimized for UNIX? It never should have been ported to Windows. Such a comparision is akin to saying "Apache 1.x runs better on Linux than on Win32!" No shit, Sherlock. Anyone who looks at the top ten results will clearly see that the point this article is trying to make means jack as far as the real numbers go.
By the way, I hear the latest Linux kernel is better at running Linux programs than CygWin!
This just in: ditors at Slashdot ar no longr prmittd to us th lttr '' mor than onc in articl titls!
They did spell Benchmarks correctly at least.
So Oracle is more optimised for Linux then Windows. Why am I not suprised?
But, it means that this setup, clusters of good intel hardware running redhat, and oracle's cluster stuff is a very, very serious contender when it comes to large databases.
ANd if you look at it on a cost basis, it's a HUGE contender.
What's TCP-C? Can somebody convert it into the Pop Science(TM) units for me so I can understand?
A deep unwavering belief is a sure sign you're missing something...
Ok, so I'm no fan of Microsoft... but this test doesn't really do much for Linux's cause. This benchmark shows a system that will not be available March of 2003, while a comparable system with Windows 2000 is already available for little more money. So, while Microsoft is raking in cash from their sales and (theoretically) will have more advanced systems out by 2003, a comparable Linux server to today's offerings will not be available for months to come. And, as an added misfortune, you've got the Microsoft entrenchment to deal with. Perhaps if the Linux solution was significantly cheaper companies would invest in it. However, what you've got here is a solution that is barely cheaper and only a couple percent faster. Given the (understandable) reputation that Unix systems have for a learning curve, and the lack of available IQ points most system administrators display.... Does anyone really expect this offering to be enough to even dent Microsoft's sales?
/dev/random
The kernel has been "tweaked" extensively by some of the best kernel hackers in the world, working at Red Hat. I doubt you'd do better :)
That said, on enterprise critical systems running Oracle the less local tweaking the better. The system is certified to run Oracle and there'd better be no local tweaks if anything go wrong ("I tweaked this IO-driver a bit... performance went up 20% when I skipped this sync bit").
In the executive summary:
Disk Drives 976 18.2-GB 15K
Disk Drives 16 18.2-GB 10K
Total Storage 18054 GB
Tape Drives 1 12/24-Gigabyte DAT
Huh? How many tapes for a full backup?
And uh, 976*18.2+16*18.2=18054.4.
No RAID either?
How do transaction processing people protect their data?
At all?
--- Hindsight is 20/20, but walking backwards is not the answer.
I think `ship' was the word they were searching for. Maybe they didn't install the latest patch for DirectX, you know, the thing that controls your keyboard and mouse input but you can't uninstall?
Previous points aside, I don't like having a server OS that shits [sic] with a "safe mode".
I like Linux's "safe mode": id:3:initdefault: