Slashdot Mirror
HP Publishs First Linux TPC-C Benchmarks
The first ever official TPC-C benchmark on a Linux system has been published.
This was run on a cluster of 32 HP servers with Intel Xeon CPUs, running Redhat Linux and Oracle RDBMS. The system had over 18 terabytes of storage, and cost over 2 million US dollars. Performance was higher than a similar system running on MS Windows.
One wonders where the slowdown lies. It would be nice to find out if it lies withing the Win32 version of Oracle or within Windows. Seeing as Windows does have the top score in the Clustered TPC-C results, I'm leaning more towards Oracle being the slow one. (everyone else here would disagree no doubt)
If you look at the top results in a price/performance comparison, Windows 2000/SQL server are still winning.e sults.asp?resulttype=cluster&version=5
http://www.tpc.org/tpcc/results/tpcc_price_perf_r
It's interesting, though, that Red Hat had the cheapest Oracle implementation.
This just in: ditors at Slashdot ar no longr prmittd to us th lttr '' in articl titls!
On the other hand, the difference in performance is 17.21 vs. 18.46, i.e. approximately 6%. I think that the result is inconclusive (except that Linux can complete in this area at all).
Some more TPC-C benchmarks of big iron kit... found here
Linux cluster:
Windows cluster:
Note that the number of clients in the windows tests is higher 24 instead of 16), with smaller CPU's. Also, the server's aren't identical.
Besides from the small differences in setup, it's plain that hardware-costs greatly outnumber software costs. Yeah, linux has a small bit more performance (less than 1%) for a bit lower price (6%) but these aren't real shocking numbers. Of course, I'll get flamed for not bashing microsoft, but the difference really isn't that big.
--
If code was hard to write, it should be hard to read
I do not hazard a guess, but it is surely interesting what Linux can do best, instead of comparing it to Windows, only.
As many other people have pointed out, the system configuration was vastly different despite being 'similar' - yes, I want to see Linux out perform Windows but I would rather see it run on identical systems! When benchmarks are weighted like this, it just makes Linux look like the inferior product which is not the kind of image it should be getting for itself!
"Hey! Unless this is a nude love-in, get the hell off my property!!"
Sorry that this might be off topic and cheerleading (right place though).
GNU/Linux impresses me a great deal. I've been running it for a few years now, mainly at home and in some positions when I can get away with it and its easy to take it for granted. But its articles like this that make me step back and think about what as really been achived by a bunch of mainly unpaid developers who provide all their source code for free.
Sure there a few places where Windows is more suitable but everyday, these few places get fewer.
So to any Linux developers and supporters who read this, thank you all.
A journey of a thousand miles starts with a brutal anal raping at airport security
Of course, I'll get flamed for not bashing microsoft
:)
Don't be silly.
From this FAQ you'll see:
In general, TPC benchmarks are system-wide benchmarks, encompassing almost all cost dimensions of an entire system environment the user might purchase, including terminals, communications equipment, software (transaction monitors and database software), computer system or host, backup storage, and three years maintenance cost. Therefore, if the total system cost is $859,100 and the throughput is 1562 tpmC, the price/performance is derived by taking the price of the entire system ($859,100) divided by the performance (1562 tpmC), which equals $550 per tpmC.
Most people would focus on the hardware cost, but in reality the highlighted maintenance cost took the precedence.
Most midrange UNIX server has outragous maintenance cost. The maintenance cost of a UNIX server in the third year could be exceeding the cost of the hardware itself. It's due to the fact that older parts are difficult to find, thus make maintaining older servers more difficult. Besides, they really want to cut older production lines in favor of newer servers production.
x86 platform is known to have flat and lower maintenance cost, due to the low cost hardware and high compability with older hardware, i.e. older parts can be found easily. That's why Microsoft could easily beat the TPC pissing races.
Doomsday finally comes to Microsoft when Linux is entering the database market. Although at this moment big corps are still offering Linux maintenance with cost comparable to UNIX package, that's not surprising when Linux engineers are not as abandon as MCSE. However, it'll not be the case in the future. I think Microsoft would eventaully lose this pissing race in the long run.
There was a followup message on the lkml from hp discussing how these numbers can improved even further, you can see it at http://marc.theaimsgroup.com/?l=linux-kernel&m=103 237327405644&w=2
The Windows TPC benchmark was on last years hardware (IBM xSeries 370, released last year).
The Oracle TPC benchmark is on next years hardware (HP ProLiant DL580R - not available till May 2003).
Crowing about how performance is higher under Linux is FUD. It's not a fair comparison. Or didn't the story submitter understand that hardware always affects performance?
The reason to use Linux is all aspects of its openness and compatibility with other systems. With Linux, you aren't locked into a single vendor. You use tools and APIs that have been around for nearly two decades and are available, in multiple implementations, from dozens of vendors. And you control how you upgrade, when you upgrade, and what path you follow with the software. And if you don't like Linux anymore, you can switch to any of a dozen other, compatible platforms.
With Windows, you are locked into a single, proprietary implementation and Microsoft has you by the proverbial precious body parts; there is no other vendor you can get a compatible implementation of Windows or all the Windows libraries from. Every couple of years, Microsoft completely changes their computing paradigms to ape what they perceive is a threat from some other company, and when the threat is gone, they just drop the initiative and move on to the next thing.
You can get stability buy paying a premium to a company like IBM, which is committed to providing it, or through open systems available from multiple vendors or open source, where you control your future. But building a large, long-lived infrastructure on Microsoft platforms is a costly folly--the company has proven that they will change approach every couple of years and that they will force their customers to move along.
Clustered TPC-C benchmarks are of dubious value. I've yet to encounter any business that implements an OLTP system like this and it's for a good reason. Clusters of this sort are very difficult to implement and operate. You basically end up having to physcally partition your database across dozens of servers!
.NET...it would be interesting to see how Linux compares.
a il .asp?id=102072201
What I would love to see is Linux TPC-C benchmark running on a 8 way machine. Particularly the IBM x440. It scored 91,000 + tpmCs running Windows
http://www.tpc.org/tpcc/results/tpcc_result_det
In a 2+ million dollar system software cost is a very small percentage of the total cost.
However, in reality, a lot companies have $10,000 or even $5,000 DB servers.
When you factor the cost of Windows into these systems the difference in Price/Performance is much greater than 6 percent.
As far as price/performance it was in the middle of the top ten. I think if they had used PostgreSQL (or maybe SapDB, dunno haven't used it) instead of Oracle it could have been higher in the results.
> Yeah, linux has a small bit more performance (less than 1%) for a bit lower price (6%) but these aren't real shocking numbers.
Huh? The cost of the Windows software adds $150K more. That is a lot of money. Think of it this way: if I have two laptops studded with diamonds that cost 2.3 mil each (before the OS is installed), and the OS for one is free while the OS for the other is 150K - the latter seems very expensive when you look at the software budget
I think that you should break out the costs of these systems and look at the hardware and software seperately. "The systems cost the same in hardware, but there is a $150k difference in software." is a much cleaner analysis.
It happens every day! All you have to do to see it is take whatever M$ junk you are using and replace the software. All the PitA you see on a desktop with poor security and stability, obscure and changing protocals, propriatory data formats, unmodifiable applications, ad nauseum, translate.
We have at least one multi-processor Dell "server" where I work and the folks responsible for it are not happy. Keeping it up has been a quality of life killer. The problem is not in the hardware.
DMCA, Hollings, Palladium. What might have sounded like paranoia is now common sense.
The Windows benchmark is running on a DL580 as well. That being said, your statement is valid. The systems are NOT identical. Click on the executive summary links at the bottom of each and you will see the detailed hardware specs for each.
Also, the clients are not only different in number, but the Linux clients are running a faster CPU as well (1400 Mhz vs. 1000 Mhz).
The only conclusion I think we can draw is that Linux plays "in the same ballpark" as Windows, performance-wise.
"This just in - Microsoft (MSFT) today promised it will inflate marketing claims by 18%-86% in Q2 of 2003, thereby shattering all hope for its competitors. After hearing the news, Sun Microsystems (SUNW) CEO Scott McNealy states that he's ready to pack it in."
I'm not bashing you, I just don't see your point.
If you get a car for 20000$ at dealer A and the same car for 19500$ at dealer B, will you say "Hey, it's just a small difference" and buy from dealer B?
Let's not forget, this is about database-servers here and both run the very same database (Oracle). The underlying OS is irrelevant, you don't have to run MS Office or "the Sims" on this thing.
So please tell us stupid Microsoft-bashers what is your point.
Also, migrating between databases is not *that* hard, it's all SQL after all.
But does PostgreSQL, Oracle's closest free competitor, scale to an 18 TB database under the loads they're testing?
Will I retire or break 10K?
Sorry, BSD and Stephen King are dead. Haven't you been reading Slashdot?
Oh, and that Elvis guy. He is really dead.
A better question: How fast would it run on a FreeBSD compiled just to do that particular job?
Hardware replacement availability is rarely a factor, as a good service organization will have a way to get parts for anything they support. As an extreme example a fellow tech of mine about 4 years ago went onsite to a company that had a dead pc, it turned out it was an IBM PC XT with a dying MFM hdd 20MB I believe, the hdd had on it some customer terminal program for interfacing with an IBM mainframe back at corporate hq. He bandged the drive a couple of times on the table to get the motor to spin up and told the customer not to turn off the pc, he then put in a service request for a replacement drive, not really expecting it to be fulfilled. Low and behold our parts people found a vendor that had like 20 of the necessary drive in storage and the customer bought 2 just to make sure they had one in case the replacement died (probably in another 15 years or so).
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
No, if you read the report the cost of Windows adds about $14K more - and part of that is for the 8 additional client licenses since they tested the Windows system against 24 clients intead of the 16 used against Linux. You have to also consider the fact that the cost of hardware goes down over time, and that the Windows system was built 3.5 months before the Linux system - this can easily make up for the $150K, and then some.
Finally, it's not about the softare cost - it's about the bottom line: price/performance. These reports can't really be scientifically compared. One can speculate that Windows would have beat the price/performance ratio of the Linux system given an even playing field, but it's just that - speculation. The real point here is that Linux is _competing_ in the enterprise and has been taken seriously enough to actually get benchmarked.
There is no longer anything that can be done with computers that is nontrivial and clearly legal. -- Paul Phillips
The point he was trying to make was that the decision is not based on finances at this point. The performance AND cost difference is inconclusive because they are different configurations. If one completely blew the other out of the water, we could maybe claim a winner, but that simply didn't happen.
And the difference between the two system costs from an OS licence standpoint is far less (see the full report). To correct your analogy:
$20000 dealer A
$19900 dealer B
It's not based on money the initial purchase. It's based on the TCO, how it will integrate with your existing infrastructure, and whethor or not you prefer one of the other (for a variety of reasons).
There is no longer anything that can be done with computers that is nontrivial and clearly legal. -- Paul Phillips
Ok.. I may get myself in some trouble here... Please be kind.. Red Hat Advanced Server is the first version of Linux to meet all the TPC requirements and provide the performance level of this benchmark. Numerous open source performance technologies were incorporated, without which, the performance would be significantly below 100,000 tpmC. The 7x24 - 4-hour response, 3 year SW and HW maintenance is also required. The TPC audit (done by a certified TPC auditor on-site) requires the demonstration of a full system crash, as well as a single node, without losing a single committed transaction. The audit for TPC benchmarks take many days... I hope the Linux community doesn't miss the point that the TPC benchmarks are the toughest benchmarks to run and pass the audit. 138K tpmC will run 95% of the OLTP business environments today. This publication provides input to the many large corporations looking more closely at Linux in the "bet your business" application areas (databases). Although I agree that the absolute numbers of performance and price/performance may not be applicable directly to many real customer applications, it is a very fair environment (level playing field) to compare HW and SW performance and price/performance. This, IMHO, is a significant step forward for Linux in the enterprise.
Also, the clients are not only different in number, but the Linux clients are running a faster CPU as well (1400 Mhz vs. 1000 Mhz).
But if you read the specs, there were 2 CPUs per Linux client, and 48 CPUs per Windows client.
--
"Outlook not so good." That magic 8-ball knows everything! I'll ask about Exchange Server next.
Is there some reason that the article mentions that the Linux solution "beats" the Windows solutions, but everywhere on the site the Windows servers (whether it be IBM's DB2 or MS-SQL) are the leaders of the pack many times over?
That article link is like something a politician would say. Do you realize that Oracle is optimized for UNIX? It never should have been ported to Windows. Such a comparision is akin to saying "Apache 1.x runs better on Linux than on Win32!" No shit, Sherlock. Anyone who looks at the top ten results will clearly see that the point this article is trying to make means jack as far as the real numbers go.
By the way, I hear the latest Linux kernel is better at running Linux programs than CygWin!
This just in: ditors at Slashdot ar no longr prmittd to us th lttr '' mor than onc in articl titls!
They did spell Benchmarks correctly at least.
But, it means that this setup, clusters of good intel hardware running redhat, and oracle's cluster stuff is a very, very serious contender when it comes to large databases.
ANd if you look at it on a cost basis, it's a HUGE contender.
What's TCP-C? Can somebody convert it into the Pop Science(TM) units for me so I can understand?
A deep unwavering belief is a sure sign you're missing something...
The kernel has been "tweaked" extensively by some of the best kernel hackers in the world, working at Red Hat. I doubt you'd do better :)
That said, on enterprise critical systems running Oracle the less local tweaking the better. The system is certified to run Oracle and there'd better be no local tweaks if anything go wrong ("I tweaked this IO-driver a bit... performance went up 20% when I skipped this sync bit").
In the executive summary:
Disk Drives 976 18.2-GB 15K
Disk Drives 16 18.2-GB 10K
Total Storage 18054 GB
Tape Drives 1 12/24-Gigabyte DAT
Huh? How many tapes for a full backup?
And uh, 976*18.2+16*18.2=18054.4.
No RAID either?
How do transaction processing people protect their data?
At all?
--- Hindsight is 20/20, but walking backwards is not the answer.