Slashdot Mirror
MS Squashes SQL Benchmarks
Player To Be Traded Later writes: "Robert Cringely at Infoworld reports here on Microsoft's attempts to squash SQL Server 7 benchmarks." In short, when a testing lab came up with far better results for SQL Server 7 under Windows NT than with its much-touted successor Windows 2000, Microsoft decided they'd rather keep the touting nice and quiet.
Microsoft has made a lot of noise about how Windows 2000 is faster than Windows NT 4.0 in their efforts to sell it to businesses that don't really want to upgrade. Denying that this exists without generating some proof that it doesn't won't help them.
I used up all my sick days, so I'm calling in dead.
Hi, my name's Robert X. Cringely, and this is a completely impartial article.
This is a manual virus. Copy it to your sig and help me spread!
If they don't like the test results, they say the test was illegal and it doesn't count(They did this a while back with a few Linux vs. NT tests too).
Wow what a horrible bit of "evidence" to drudge up, since the same thing happened in reverse when some Linux benchmarks showed it performing worse than NT. The Linux crowd went berserk...
One thing I've learned over the years -- the only benchmarks that matter are ones you do yourself with real-world situations!
If you can't do them yourself, then you just have to take third party benchmarks with a mill of salt.
"And like that
I'm sure MSSQL would go pretty fast on DOS 6.2
Je t'aime Stéphanie
That's very interesting, and appears to be correct (see http://www.cybercrime.gov/ipmanual/03ipma.htm, "Statutory exceptions" section). I was under the impression that the entire EULA mess started because a clueless and/or bought judge ruled that loading a program into RAM constituted making a copy. Under that (il)logic, the EULA grants you the right to run the software which you would not otherwise have, and in exchange strips you of fair use rights. But based on 17 USC, you already have the right to run the software, so the EULA removes your fair use rights in exchange for nothing. IANAL, but I thought that a contract without "consideration" was invalid. So even if a EULA is a contract (highly questionable, given there's no communication between you and the manufacturer and no way to prove you actually read and agreed to the terms), it shouldn't be enforceable. Can anybody clear this up?
I'd like to see the EFF take up the EULA issue, of course only after they've finished sending the DMCA back to the bowels of hell from whence it came.
How to solve most of our problems: 1.Lots of nuclear plants. 2.Cure aging.
I love to see a Microsoft article go up, then observe the huge disparity in responses. Overwhelmingly, you see the two categories into which everything in the universe can be divided: 1)High school aged rabid Lunix advocates who are drooling all over themselves to get in a slashbot comment for the purposes of karma whoring, and 2)People who have jobs, earn money, and invariably have/want to work with Microsoft products. Granted, not all people who are outraged at this are making the same frothing response, but those of the former who do so are rewarded for their "cleverness" just as well as those from the latter who see that this is both typical behavior for any database vendor and wholly unsurprising to those who know squat about Windows. Of course those who are in their little "Yay Linux" world are predisposed to ignorance on the topic.
Typical Microsoft response. If someone else has a technology that could damage them, they buy the company and hide the technology to collect cobwebs. If they don't like a company, they browbeat them into the ground(prime example, Netscape). If they don't like the test results, they say the test was illegal and it doesn't count(They did this a while back with a few Linux vs. NT tests too). If the test results come back bad, MS should see it as where they came short this round, but to accept and try to fix in the future. Learn from your mistakes, don't cover them up and lie about the matter.
Keeping quiet about statistics is the best way to lie about them - This is just the same as the old toothpaste advertisments, that 'made 9 out of ten children have fewer fillings' - The results that the toothpaste company did not want seen were simply filed quietly out of sight. Companies releasing desired stats (and witholding undesirable ones) is nothing new... Its a simple fact :) There are lies, damn lies and then statistics; or in this case, Benchmarks :)
For example, compare the top result (Compaq/Win2K/SQLS2000) with the fifth result (IBM/AIX4.3.3/Oracle8.1.7). Both systems cost around $10M. The Compaq cluster scores about 2.3 times higher than the IBM system. The Compaq cluster is 24 8x700MHz PIII Xeon servers (192 processors). The IBM server is a single 24x600MHz RS64 IV (24 processors). With that sort of hardware disparity it is impossible to make any judgement on the software performance at all. The Compaq setup has much better price performance but you can't attribute that to the software. The second place DB2 cluster you mentioned is a 32 machine 4x700MHz PIII Xeon setup (128 processors) and scores much closer to the Compaq setup which also points to hardware as being the major factor.
The most interesting bit is that the software for the Compaq setup costs just over $3M, with the software for the IBM system being under $1M. Virtually all the software costs are in the DBMSs. If you have license 192 copies SQL Server is not cheap.
So you have a new OS out that you want everyone to run. Would you want a benchmark coming out that says your old stuff is better. I'd like to point out that this article talks about SQL7. SQL 2000 runs MUCH better on Win2k than it does on NT4.
SQL7 was written to take advantage of NT4, not Win2k. I can't say that the test results OR Microsoft's actions suprise me much.
A speech...
Also, it depends on how they had their Win2k box set up. Active Directory is a mess and could be slowing it down along with a bunch of other services that come with it by default that weren't part of NT.
It should be noted that for whatever reason, be it MS backing down or the company discovering they could publish it legally, the results ARE online.
In other words, MS didn't win that particular round.
News at 11, Microsoft enforces same license that every other database vendor uses. AFAICR, no big time database vendor would allow you to publish benchmark results, not just Microsoft. Now if they were going to allow the results to be published if the Win2K box beat the NT4 box, then you may have something.
Besides, they left out way too much detail to get in a fuss over. Like maybe the NT4 box was a 4 way P4, and the Win2K box was a P133 overclocked to 166 MHz and with flaky 32MB simm. They never state that the same hardware was used.
While I have never been accused of being in Microsofts corner, they are in the right on this one and we have seen darn near every major* database vendor pull the same stunt.
*For some definitions of major.
That may be true, but it's generally assumed that upgrading the HW or OS would therefore result in an increase in performance. This demonstrates that NT to Win2k isn't a vertical upgrade, it goes a little sideways too.
Which is ok. It's not something that should be hidden, and certainly not something that should be censored with the threat of lawsuits based on bogus EULAs.
Tom Swiss | the infamous tms | http://www.infamous.net/
Tom Swiss | the infamous tms | my blog
You cannot wash away blood with blood
So while W2K is a good OS (I personnaly use it for my desktop), for large enterprise work, SQL Server just doesn't appear to work.
I know it's in their license, but I have a serious ethical problem with a company being able to control "independent" reviews of their products.
I believe that it's only reasonable for a company doing product reviews to allow the vendor to respond. If Network World puts up a review saying that SQL is slower on NT5^H^H^HWin2000 than on NT4, Microsoft should not be able to kill the review. They should be able to respond, and Network World should post the response along with their review. That's called responsible journalism.
Click-through is like any other contract.
Click through is governed by copyright. As eminent proof, I can extract the usable binaries from packages I purchase for most commercial software without clicking through. I guess that makes me a thief in the US (do to DMCA). Or does it ?? Click throughs do not protect copyrights - they attempt, illegally, to extend them.
I might feel differently if they actually asked me to click through BEFORE I write a check. They ask you to agree to a license that governs something, and you cannot see the license before you make a purchase. Of course they give you the right to demand a refund, but do you remember the protests in which consumers demanded refunds for Windows ? It is a big ball of wax shined on to convince you that you do not have rights you have. Very few things about click throughs meet the standards to be called a contract.
The software has only copyright, and copyright has fair use. Evaluating and posting public commentary on that use is one of the most standard protected forms of fair use.
I find it a bit interesting that the article has no link to the websites of the testing lab or the actual benchmark result...
So where is it?
====
Codeala - Just another mindless drone
Slash should do it.
My guess would be that MSSQL7 uses some system calls that are "native" in NT4, but are some kind of backwards-compatible kludge in W2k. If that's the case, it would make perfect sense that MSSQL7 would be slower on W2k, but MSSQL2k would be comparable.
IANAL, AFAIK, yada yada, standard disclaimers and all that stuff.
The difference is that you buy a car, and then physically own it and can do whatever you want with it (within normal laws not related to the purchase of the car). You license computer software, which means that you purchase permission to use it under circumstanced stated in your license agreement. You never actually own the DB software.
It's the basic difference why you can do so many things with tangable things that you buy, as opposed to intellectual property that you licence.
Be the Ultimate Ninja! Play Billy Vs. SNAKEMAN today!
What's going to happen when a windows XP comes out? Oh I get it you have to upgrade your database server too.
War is necrophilia.
good plan. you can upgrade your OS, but you'll need to upgrade all your other software if you don't want everything to slow down horribly. Would you like the undercoating with that? These OSes will rust up on you in an instant! How about the "protection plan?"
Something is definitely fishy with their hardware if Win2K is twice as slow as NT4. I've run both servers with SQL7 intensively. You couldn't pay me to move back to NT4. 2000 isn't all that much faster, but it is much more stable and its a lot easier to use and administrate.
Want some real benchmarks? Try here. Notice a pattern? SQL Server is the fastest database server in the world. Not only that, but Win2K is in the top four slots. 2nd place is a DB2 server on Win2K. Here are real, industry standard tests performed by an independent organization, not a company with an agenda to promote or magazines to sell.
I'm not sure what the point of this article is, other than to stir up more mindless MS-bashing. Well, Timothy, maybe you should try SQL Server or another real database. Pretty much every day around noon we get the same problem because Slashdot can't handle displaying stories while lots of people are posting. A real database would do wonders to fix that.
No, Thursday's out. How about never - is never good for you?
as i recall, neither MS or Oracle are very enamored with third party benchmarks. the only time they *don't* threaten the publisher is when their product "shines".
the results are just too easy to skew, and the real-world loading is tough to accurately model.
i'm trying to learn Oracle on linux -- it's pretty cool the way I can legally install a free OS, then download oracle 8i enterprise for personal use.
"oratcl" is now on sourceforge, and php3 has gtk+ bindings for standalone applications (but works great through a browser, of course).
it's a great time to learn about databases...just don't publish those benchmarks!
Treatment, not tyranny. End the drug war and free our American POWs.
See my user info for links.
Under standard copyright law, something like benchmarks is considered fair use, and is thus not subject to asinine click through agreements. If something is not covered by copyright law, it cannot reasonably be covered by click through.
This is also the case in Europe, where, for example, you are legally allowed to resell Microsoft OS licenses. It is the only way to interpret copyright law wrt software that makes sense.
Oh, silly me. Who expected M$ to make sense ?
So what? If I go to the store, purchase a copy of the program, and install it without agreeing to the EULA, I can still legally run it (MS has been compensated; there's a provision in 17 USC that excludes the running of software from infringement) and MS can, indeed go to hell.
Of course, I would hope that they don't drag down the entire neighborhood, as I live pretty close by.
Most copyrighted material is not licensed at all, or as a condition of purchase, software included. Even the legality of a post-sale EULA is the matter of some debate. Don't assume that the things are 100% legit just because software publishers claim that they are. IIRC the case law is almost evenly split, with a slight leaning in favor of the 'EULA's don't count' side.
-- This and all my posts are in the public domain. I am a lawyer. I am not your lawyer, and this is not legal advice.