MS Squashes SQL Benchmarks

Player To Be Traded Later writes: "Robert Cringely at Infoworld reports here on Microsoft's attempts to squash SQL Server 7 benchmarks." In short, when a testing lab came up with far better results for SQL Server 7 under Windows NT than with its much-touted successor Windows 2000, Microsoft decided they'd rather keep the touting nice and quiet.

Comments

No bias here

[graniteMonkey]

an independent testing lab from publishing benchmark results that the lab ran for InfoWorld's sister publication NetworkWorld

Hi, my name's Robert X. Cringely, and this is a completely impartial article.

What about DOS?

[SpanishInquisition]

I'm sure MSSQL would go pretty fast on DOS 6.2

Re:Why do DB companies get away with this?

[bnenning]

If I go to the store, purchase a copy of the program, and install it without agreeing to the EULA, I can still legally run it (MS has been compensated; there's a provision in 17 USC that excludes the running of software from infringement)

That's very interesting, and appears to be correct (see http://www.cybercrime.gov/ipmanual/03ipma.htm, "Statutory exceptions" section). I was under the impression that the entire EULA mess started because a clueless and/or bought judge ruled that loading a program into RAM constituted making a copy. Under that (il)logic, the EULA grants you the right to run the software which you would not otherwise have, and in exchange strips you of fair use rights. But based on 17 USC, you already have the right to run the software, so the EULA removes your fair use rights in exchange for nothing. IANAL, but I thought that a contract without "consideration" was invalid. So even if a EULA is a contract (highly questionable, given there's no communication between you and the manufacturer and no way to prove you actually read and agreed to the terms), it shouldn't be enforceable. Can anybody clear this up?

I'd like to see the EFF take up the EULA issue, of course only after they've finished sending the DMCA back to the bowels of hell from whence it came.

There are two kinds of people in this world...

[Anonymous Coward]

I love to see a Microsoft article go up, then observe the huge disparity in responses. Overwhelmingly, you see the two categories into which everything in the universe can be divided: 1)High school aged rabid Lunix advocates who are drooling all over themselves to get in a slashbot comment for the purposes of karma whoring, and 2)People who have jobs, earn money, and invariably have/want to work with Microsoft products. Granted, not all people who are outraged at this are making the same frothing response, but those of the former who do so are rewarded for their "cleverness" just as well as those from the latter who see that this is both typical behavior for any database vendor and wholly unsurprising to those who know squat about Windows. Of course those who are in their little "Yay Linux" world are predisposed to ignorance on the topic.

Learn from your mistakes and admit it

[alptraum]

Typical Microsoft response. If someone else has a technology that could damage them, they buy the company and hide the technology to collect cobwebs. If they don't like a company, they browbeat them into the ground(prime example, Netscape). If they don't like the test results, they say the test was illegal and it doesn't count(They did this a while back with a few Linux vs. NT tests too). If the test results come back bad, MS should see it as where they came short this round, but to accept and try to fix in the future. Learn from your mistakes, don't cover them up and lie about the matter.

Fake benchmarks

[PepsiDman]

Keeping quiet about statistics is the best way to lie about them - This is just the same as the old toothpaste advertisments, that 'made 9 out of ten children have fewer fillings' - The results that the toothpaste company did not want seen were simply filed quietly out of sight. Companies releasing desired stats (and witholding undesirable ones) is nothing new... Its a simple fact :) There are lies, damn lies and then statistics; or in this case, Benchmarks :)

I can't say I blame them...

[km790816]

So you have a new OS out that you want everyone to run. Would you want a benchmark coming out that says your old stuff is better. I'd like to point out that this article talks about SQL7. SQL 2000 runs MUCH better on Win2k than it does on NT4.

SQL7 was written to take advantage of NT4, not Win2k. I can't say that the test results OR Microsoft's actions suprise me much.

who cares? use SQL 2000 instead

[shodson]

If they lied, yes, damn them and shame on them. However, SQL 7 was built for NT, SQL 2000 was built for Windows 2000 and is their newer product anyway.

Also, it depends on how they had their Win2k box set up. Active Directory is a mess and could be slowing it down along with a bunch of other services that come with it by default that weren't part of NT.

A note for those who didn't read the story.

[SuuSt]

It should be noted that for whatever reason, be it MS backing down or the company discovering they could publish it legally, the results ARE online.

In other words, MS didn't win that particular round.

Big Deal

[latneM]

News at 11, Microsoft enforces same license that every other database vendor uses. AFAICR, no big time database vendor would allow you to publish benchmark results, not just Microsoft. Now if they were going to allow the results to be published if the Win2K box beat the NT4 box, then you may have something.

Besides, they left out way too much detail to get in a fuss over. Like maybe the NT4 box was a 4 way P4, and the Win2K box was a P133 overclocked to 166 MHz and with flaky 32MB simm. They never state that the same hardware was used.

While I have never been accused of being in Microsofts corner, they are in the right on this one and we have seen darn near every major* database vendor pull the same stunt.

*For some definitions of major.

Where is the benchmark?

[Codeala]

I find it a bit interesting that the article has no link to the websites of the testing lab or the actual benchmark result...

NetworkWorld eventually overcame the Microsoft threat, however. The test results were posted on its site early last week.

So where is it?

====

a way around the "no benchmark" rules

[swinge]

the big DB companies all ban publishing benchmark results through terms in their license agreements. This is incredibly irritating because it undermines the free market principle of "perfect information" (markets with "asymetric information" are known to be ineffecient, something laissez-faire Randites never seem to learn). In pursuit of the public benefit of market efficiancy, wouldn't this be a way around these stupid rules:

1. put up a webserver with some CGIs that do some useful largescale things.
2. Implement the same things in several CGIs that communicate with different back-end databases.
3. Allow the public to come to the server and run and compare results (yes, you need some locking to stop them from interfering with one another).
4. If some member of the using public is a journalist, that journalist is free to publish the results because they are not party to the license.
5. You, a party to the license, are free to implement a website like this because it's just like any other website, albeit with a little extra redundancy.

Slash should do it.

Re:who cares? use SQL 2000 instead

[gregbaker]

Also, it depends on how they had their Win2k box set up. Active Directory is a mess and could be slowing it down along with a bunch of other services that come with it by default that weren't part of NT.

The article says that the lab worked with MS "for a week" to figure things out and "neither company could fix the problem". I'm no W2k expert, but I'm sure the MS tech that work with major trade publications are. I'm sure they would have thought of turning off extraneous services.

My guess would be that MSSQL7 uses some system calls that are "native" in NT4, but are some kind of backwards-compatible kludge in W2k. If that's the case, it would make perfect sense that MSSQL7 would be slower on W2k, but MSSQL2k would be comparable.

Re:Why do DB companies get away with this?

[Gerad]

IANAL, AFAIK, yada yada, standard disclaimers and all that stuff.

The difference is that you buy a car, and then physically own it and can do whatever you want with it (within normal laws not related to the purchase of the car). You license computer software, which means that you purchase permission to use it under circumstanced stated in your license agreement. You never actually own the DB software.

It's the basic difference why you can do so many things with tangable things that you buy, as opposed to intellectual property that you licence.

And the point is?

[Pinball Wizard]

Doesn't Oracle do exactly the same thing? AFAIK, you have to get permission to publish benchmarks. So what. Its in the licence.

Something is definitely fishy with their hardware if Win2K is twice as slow as NT4. I've run both servers with SQL7 intensively. You couldn't pay me to move back to NT4. 2000 isn't all that much faster, but it is much more stable and its a lot easier to use and administrate.

Want some real benchmarks? Try here. Notice a pattern? SQL Server is the fastest database server in the world. Not only that, but Win2K is in the top four slots. 2nd place is a DB2 server on Win2K. Here are real, industry standard tests performed by an independent organization, not a company with an agenda to promote or magazines to sell.

I'm not sure what the point of this article is, other than to stir up more mindless MS-bashing. Well, Timothy, maybe you should try SQL Server or another real database. Pretty much every day around noon we get the same problem because Slashdot can't handle displaying stories while lots of people are posting. A real database would do wonders to fix that.

Re:Why do DB companies get away with this?

[cpt kangarooski]

So what? If I go to the store, purchase a copy of the program, and install it without agreeing to the EULA, I can still legally run it (MS has been compensated; there's a provision in 17 USC that excludes the running of software from infringement) and MS can, indeed go to hell.

Of course, I would hope that they don't drag down the entire neighborhood, as I live pretty close by.

Most copyrighted material is not licensed at all, or as a condition of purchase, software included. Even the legality of a post-sale EULA is the matter of some debate. Don't assume that the things are 100% legit just because software publishers claim that they are. IIRC the case law is almost evenly split, with a slight leaning in favor of the 'EULA's don't count' side.