Slashdot Mirror
3D Mark 2003 Sparks Controversy
cribb writes "3DMark 2003 is out, sparking an intense debate on how trustworthy its assessment of current graphics cards is, after some harsh words by nVidia and the reply from Futuremark. THG has an analysis of the current situation definately worth reading. The article exposes some problems with the new GeforceFX previously mentioned in a slashdot article on Doom3 and John Carmack. Alas, here seems to be no end to the troubles with the new nVidia flagship." If you've run the benchmark, post your scores here, and we'll all compare.
Now, it's the video card makers slagging the benchmark makers.
Anybody remember the early 90s (93?) when Hercules got itself into hot water by hard-coding a super-fast result for the PC Magazine video benchmark? Whoo hoo, that made for some good press. Got their awards pulled and everything.
Do daemons dream of electric sleep()?
It stands to reason that a benchmark should fairly and accurately depict the widest range of common capabilities possible to determine a clear winner. Of course, this can be very hard to do. It does seem in this case though that 3DMark got caught up in the whiz-bang marketing side of things by supporting the latest and greatest(?) features and ignoring the very compatibility that would give it any real meaning.
Sorry guys, you goofed.
There are lies, damn lies, and benchmarks.
792 3DMarks.
System:
Geforce3Ti200 GFX
AthlonXP1700 CPU
256MB SDRAM
ECS K7S5A Mainboard
I don't like it. I'm gonna rely on actual game benchmarks when I compare my system's performance. Some good games to use:
Quake3 (still scales nicely)
UT2003 (the game sucks, but it's a decent CPU benchmark)
C&C: Generals (don't know how it scales, but it cripples most computers)
Doom3 (Will hopefully scale as well as Q3 when it comes out in 2 months)
Synthetic benchies just aren't that reliable anymore...
------- "From bored to fanboy in 3.8 asian girls" ----------
NVidia missed a manufacturing cycle and now it's coming back to haunt them. They really need to drop the FX and concentrate on whatever new architecture is currently being tossed around in R&D.
Originally I was planning to buy the successor to the NV30 for a great experience in Quake III and better framerates in older games. But now it looks like I'll be laying out the dough on whatever ATI brings out early next year.
So long as my opponents are rendered as a red spot on the floor, I'm happy.
Is this truly the only Earth I can live on?
AMD Athlon1400C@1550
512MB Samsung DDR, CL2@147FSB
Geforce4ti4200, clocked@260core, 520memory
a whopping: 1080 points.
Did i mention that this benchmarks makes *heavy* utilization of the otherwis in *no* game used Pixelshader 1.4? Teh exact one, that Nvidia didnt implement in its GF4Ti cards - where only 1.3 and 1.1 is in?
Guess, who has 1.4 - ATI has...
You could also call this benchmark "ATIbench2003", but that was the same in 2000, when 3dmark2000 was favoring Nvidia cards over 3dfx simply because of the lack of 32bit colordepth.
Sheeeshh...
My Apple ][+ doesn't have enough disk space to download this program. Can someone help me out?
Trolling is a art,
voltron:/home/gannoc/incoming/temp# chmod +x 3dmark2003.exep # ./3dmark2003.exe ./3dmark2003.exe: cannot execute binary file
voltron:/home/gannoc/incoming/tem
bash:
With the level of complexity in current hardware, I can't imagine anyone will come up with a benchmark that -can't- be labelled as skewed, inaccurate, or 'not giving justice'.
If I spend a million dollars developing a cool board that does zillions of sprigmorphs a second (a made up metric), and someone does a benchmark that doesn't test sprigmorph rendering, does that mean my board sucks? No, it just means the benchmark doesn't check it.
However, if Competitor B makes a board that doens't have sprigmorph rendering, but scores higher on this benchmark, which is the 'better card'?
The days of simple benchmarks, alas, are past. It used to be "how many clock cycles a second". Nowadays, whether one piece of hardware is better than another simply comes down to "Can it do what I'm doig right now any faster or cheaper than another unit?"
Event Management Solutions : http://www.stonekeep.com/
Benchmarks are generally too isolated to be of much use. They might be okay for getting a rough picture, but a high scoring 3d benchmark might not directly translate into good 3d performance.
Even so-called 'real world' benchmarks that test stuff like file opening and scrolling documents don't really get into the meat of the everyday user experience.
Using benchmarks to decide what computer to buy is like macking on the girl with the big boobs. She might look nice, but she could be horrible in bed. Also she might have crabs.
(-1, Raw and Uncut is the only way to read)
The guys at Tech Report also has an article in which they dissect parts of the benchmark and provide what both FutureMark and nVidia's comments on the matter.
IF you've run the benchmark, post your scores here, and we'll all compare.
Or you could just go directly to the futuremark forums instead.
Along with most of my geek friends, I really depend on excellent the "excellent" 3dMark scores of the latest and greatest hardware to drive down the price of the previous generation of video cards. After all, software that truly supports all of the whiz-bang features of the top-tier cards doesn't arrive until about 6-9 months after the cards appear on Best Buy's shelves.
If 3DMark isn't producing high enough scores for the new nVidia cards, where will my price breaks be?
Sorry, but this has been on major hardware sites for two? three? weeks. Www.hardocp.com had an entire article on (and mostly started the hoopla over) this entire thing. Posting 3DMark scores to slashdot is a total waste of time anyway. There is no trusted system of comparison here. Most slashdot readers aren't hardcore performance nuts anyway. (Go ahead, be a troll or a classic weenie and take that statement out of context or whatever)
I just don't think this is the right forum for this type of story. Oh well.
It seems like the 3Dmark folks decided to deliberately test DX9 features, even though there are not many cards which support them in hardware yet. Nvidia is pissed because they have not implemented any DX9 features in hardware on the FX, where ATI has them on the 9x00 whatever.
This is a valid benchmark to use to test out how your current hardware will perform in a DX9 environment. I, for one, am glad to see such a tool available so that I can take DX9 performance into account when making my next video card purchase. So my next card may be an ATI - Who knew? The last ATI product I owned was a Number 9, not exactly a 3D monster....
You think emacs is evil?! You've never used VM's XEDIT have you?!! That's evil, baby!
Read about it here. http://www.theinquirer.net/?article=7920
"An Nvidia technical marketing manager confirmed to us that Geforce FX has 4 Pipelines and 2 Texture Memory Units that can results with 8 textures per clock but only in multitexturing.
However, Nvidia did say that there were some cases where its chip can turn out 8 pixels per clock. Here is a quote:
"GeForce FX 5800 and 5800 Ultra run at 8 pixels per clock for all of the following: a) z-rendering b) stencil operations c) texture operations d) shader operations"
and
"Only color+Z rendering is done at 4 pixels per clock"
We talked with many developers and they said me that all games these days use Color + Z rendering. So all this Nvidia talk about the possibility of rendering 8 pixels in special cases becomes irrelevant.
The bottom line is that when it comes to Color + Z rendering, the GeForce FX is only half as powerful as the older Radeon 9700."
This statement is false.
Ok slashies.
3DMark 2001 measures performance for directx 7 and 8 hardware platforms.
3DMark 2003 was built from the ground up to measure performance for directx9 platforms, it is not DESIGNED to be a broad range benchmark. it isn't meant to give good scores to your computer that does what you need it to.
It's a high end performance measurement tool, which UNLESS USED IN THE PROPER CONTEXT gives you useless measurements.
Sorry for the pissiness, but jeeze. for geeks who claim to love specialized tools and hate bloat, this is the perfect tool. it does one thing specifically and doesn't throw in the kitchen sink, or support for ancestral hardware.
They aren't microsoft, they're fully supporting 3DMark 2001 for the platforms that it was designed for.
I'll hush now.
"You worthless post!"
-Shakespeare, 2 Gentlemen of Verona, 1. 1. 147
3d Mark 2003 Score: 1252
Geforce 4 Ti 4600 @ AGP 4x
800 MHz PIII
256 MB RDRAM
Intel VC 820 Motherboard
Windows XP
Games & 3d Mark ran off of 80GB WD 8MB cache Special edition hard drive, alone on a seperate IDE card on the PCI bus.
For Games:
Simcity 4- large maps and pleasing resolutions bring my comp to it's knees. Running SC4 at 1024 & higher resolution is absolutely beautiful, running it at 800 x 600- it looks like ass.
RtoCW runs fine at 1024, haven't tried it higher yet.
Delta Force: Black Hawk down runs fine at 1024, with full effects. Haven't tried it higher yet. The water effects are stunning.
UT2003 ran fine when i had a GF2 in here, haven't tried it since.
my 2 cents
Alcohol, Tobacco and Firearms should be the name of a store, not a government agency.
Am I the only one who saw "THG" in the post and thought, "The Humble Guys? They're still around? And they care about graphics??"
I had to mouseover to realize that they meant Tom's Hardware Guide and not "The Humble Guys" of 1980s BBS piracy. Hrm, I guess I'm showing my age.
Heh, for a trip down memory lane, check this out:
http://www.textfiles.com/piracy/HUMBLE/
Using your sig line to advertise for friends is lame.
I dislike benchmarks like these. It encourages video card manufacturers to design video cards that do well in benchmarks, rather than do well in actual applications.
There are tons of people who do comparisons with applications rather than benchmarking utility. Whether you're a fan Tom's Hardware (or not, I know he's had somewhat of a sorted past), there a lot of sites where people like him do testing with end user applications. Do research, find one of those sites you trust, and go with numbers based on software you use, rather than some number a benchmarking application you'll never actually run gives you.
Overrated Moderation: This posts sucks... because.
The under-issue here is that nVidia is no longer a "partner" of madonion (I know they changed their name, whoever they are now, futuremark or whatever) but ATI is (IIRC). This is helping fuel suspicion that the benchmark is designed to perform better on ATI hardware than on nVidias. You must pay a fee to be a "partnet" so there is the unspoken idea that what Futuremark is doing might be some kind of extortion.
Where the answer lies is up to you. Personally, I do think that the benchmark is unfair/not a good benchmark. For example, chaning the graphics card in your computer should have next to no effect on the CPU score, if any; yet it has a measureable effect. But all of this is mute, IMHO, since Doom III will be the new Uber benchmark trusted above all else when it comes out. Untill then, argue amongst yourselves.
Comment forecast: Bits of genius surrounded by a sea of mediocrity.
Personally, I think that a good benchmark is just doing whatever you are going to be doing and timing that.
Are you going to be playing much of the 3D-Mark benchmark ? If the answer is yes, then you should use it, otherwise it's pure masterbation. Their site claims that the purpose of the benchmark is to give you an idea of what a typical DX7-DX9 game will give you in performance. However, the 'games' they use to test it are not games you can actually play. It's basically a graphics demo. Wow.
The only benchmarks even worth considering are the Quake, Unreal, etc. benchmarks that test real games being played. And even those results should be taken with a grain of salt. They are 'real world' results, but you have to take into account many factors to actually derive useful information from them. Such as RAM, CPU, resolution that marks were run at, etc.
If you are smart, then you will buy your graphic card from a place like Fry's that will let you return it if the performance is unsatisfactory. In this day and age where the graphics card costs more then a computer, you had better get your money's worth.
Just test SimCity 4. It kicks the snot out of my P4-2.26/1GB DDR/4200.
This is not the greatest sig in the world, this is just a tribute.
Sometimes people scratch their heads about benchmarks and wonder "how did they come up with that number?" If the benchmark itself was Open Source you'd have at least a partial answer. Not to mention you'd have the eyes of many people looking over the code to make sure it was executing draws in the right and consistent manner.
So why aren't benchmarks open? What do the makers of benchmarks have to hide? Are they under NDAs from the card vendors?
He makes my head explode every time he talks video cards.
Tell him Corky here can't handle this.
What do you mean? Mine's out.
My beliefs do not require that you agree with them.
I personally dont put too much trust into any benchmark. If I see an increase in performance compared to the actual software/hardware that I run, then thats all I care about...
Either synthetic or not, you can only put so much into a benchmark. Half of the graphs for bencharks have scales which are EXTREMELY misleading. It makes a .4 fps difference look like a 400 fps difference.
--
Yes, it does matter (within reason, anyway). While your current card may do well enough at Quake 3 and the new cards may not have a huge margin over it (really, what's the difference between 150fps and 200fps except in the very rare situation where absolutely everything on the screen is blowing up or something), that's old technology. As hardware capabilities increase, software complexity also increases. That card getting you 150fps at 1024x768 in Q3 with 4x FSAA will likely barely break 30fps for Doom 3. (at that point, you tweak -- drop your resolution, turn off FSAA and anisotropic filtering, lower your detail levels, turn off unnecessary effects, etc and get up to a playable 50fps or so) The cards doing 200fps in Q3 will probably run D3 around 50-60fps. While there's little difference between 150-200fps, there's a world of difference between 30 and 60fps.
And just to head off any, "But your eye can only see 24/30/60fps anyway, who needs more?" arguments:
now nvidia are introducing a new factor in the equation: now you have to write different code for each videocard. just as there used to be 3dfx-only games.
isn't this against the idea of directx? seems very counterproductive to me, and an attempt by nvidia to monopolize the gaming industry.
Hostes alienigieni me abduxerunt. Qui annus est?
Carmack says:
It seems that the NV30 architecture requires a good deal more optimization to run shader code optimally (read: fast), while R300 deals with standard code much better. This would explain why NVIDIA is so harsh and aggressive in its criticism of the new 3DMark 2003, since the GeForce FX (NV30) seems to have a problem with non-optimized shader code, a trait that its mainstream siblings NV31 and NV34 will obviously share. If word got around - and in this community, it does - this could seriously hurt NVIDIA's sales.
To be fair, in real games this "handicap" will most likely not be nearly as pronounced as in the 3DMark test. After all, NVIDIA is very good at convincing game developers to optimize and adapt their code for their hardware.
So NVidia only runs well with optimized code huh? That's going to be a problem for them I think. It means we won't know how well it works until we get some games to benchmark it with. Sure, we could benchmark it with UT2003 or something; but that doesn't mean much. I don't care about UT2003. My current card runs that fine. I (and other people who buy these cards) care about how they will run the next gen games. We could wait until those games come out, but a lot of people don't have that patience. For those people it might be safer to get the ATI. If you go with NVidia you have to really trust that the games you want are going to be well optimized for it, though as Carmack said, they probably will be. Personally I'm still on the fence about which card I will eventually get.
Aw crap, ninjas!
I've got a 9700 pro, p2.53, sis648 and 512 DDR400. Hardly a lowend rig. My 9700 chokes on 3dmark 2k3. At several points in the demos the FPS drops below 10.
If this benchmark is supposedly so horribly biased in favor of ATI, you'd think they might at least get it to run smooth on my 9700.
I think 3dmark may be accurately pointing out that this new wiz-bang high-precision stuff may only start to be gameworthy on the NV35/R350 or even NV40/R400 generations.
roughly what he's saying is:
If you just write an application then it will run twice as fast on the ati card as the geforce fx
But if you write two applications to to the same thing and optimize one for the ati card and the other for the nvidia card then the nvidia card does better
So performance wise nvidia appear to be relying on developers to optimise their applications specificaly for the geforce fx. And they probably will get it too given their current market share.
Looks like we'll have to wait and see.
Comment forecast: Bits of genius surrounded by a sea of mediocrity.
nVidia needs to learn that you can stay alive as a company with the #2 video card, as long as you can price it competitively - hell, that's what ATI did for years. But they do need to make sure they eventually get a winner. Since FX obviously ain't it, maybe they can win one next year. And making better decisions is part of it - don't skimp on pixel shaders like 1.4 when the competition will be able to kill you with it.
They definitely need to catch back up to ATI - competition on this front is good for all of us.
-Looking for a job as a materials chemist or multivariat
There's a substantial thread on Ars Technica's forums that contains a ton of benchmark results. What it boils down to is that if you have a decent processor (Athlon XP 1600+ or better) and an NVidia GF4 Ti4600, you'll end up with something like 1500-1700 3DMarks. If you pull the GF4 out and slap in a Radeon 9700 Pro (and get the appropriate drivers installed, of course), your score would shoot up to over 4000 3DMarks.
I've got a Ti4600, and 3DMark 2003 runs like ass. Fortunately, Splinter Cell plays just fine, so I'll ignore the benchmark and get on with actually using the computer.
Next well have manufacturers making us accept a EULA before installing the drivers that will forbid benchmarking their hardware. Sound familiar?
AMD XP TBred-B 2100+ OCed to 2700+ (166fsb x 13)
2x265MB DDR400 Clocked at 333Mhz, with 2-2-4-2 Timings (Dual Channel A7N8X Deluxe)
ATI Radeon 8500 Default Clocking
My Score was a wopping 1173 3DMARKS with
Program Version 3DMark03 Revision 1 Build 3
Resolution: 1024x768@32 bit
Texture Filtering: Optimal
Pixel Processing: None
Vertex Shaders: Optimal
Uh, PS1.1 and 1.4 are part of DX9; they just happen to be available on DX8 cards too. There's no reason why you _have_ to use PS2.0 on DX9 cards if earlier versions will work just as well... and it's likely that game developers will use the earlier versions where possible, for best backward compatibility with older cards.
Just imagine if every test had required DX9: people would be whining that their DX7 and DX8 cards couldn't run anything.
This is what is scaring me about Nvidia, Espicially since just about everything Nvidia makes is running my PC right now and it would be a cold day in hell before an ATI product touches my PC.
From what I've seen so far, Nvidia is doing the exact same thing that 3dfx Did when the voodoo3 came out, and whats more disturbing is that the're following the 3dfx downward spiral so close that you could praticially mirror the two, a sort of NVfx if you will.
Making Video cards, Pushing their Rendering Format harder than ever, Bashing Benchmarks, claiming that their hardware is limited for a reason, ETC. All of this failed miserably when 3dfx did it, and it's going to fail for Nvidia as well.
It wouldn't suprise me to see a dual GeforceFX board this year, or even a quad version. It's what 3dfx did before they went under.
In Soviet Russia, Trojan exploits YOU!
When AMD's K6-2 processors were getting stomped by the Pentium II, it turned to 3DNow, leaning heavily on 3DNow-optimized Voodoo2 drivers and a 3DNow-optimized version of Quake 2. Anand's Monster 3D-2 review shows 3DNow improving a last-place 44 FPS to a competitive 76 FPS. Quake 2 played better because of the efforts of AMD and 3dfx. However, the results weren't representative, as the Turok and Forsaken benchmarks show.
I played System Shock 2 on a Voodoo3. At the time, 3dfx had Quake 3 on the brain, struggling to tweak its drivers to keep up with the GeForce. Those efforts were small consolation to me, as each new driver release would break something in System Shock, like making the weapon model sporadically disappear.
The problem with a marquis game like Quake is that it encourages short cuts. The testing is done when Quake runs (a little faster). I, for one, am glad that Quake 3 put an end to the miniGL nonsense. Give me a card with decent, reliable performance in standard APIs like OpenGL and Direct3D. Put it this way: would you buy a TV that was optimized for Friends?
3dfx started losing around the time the tnt2 came out. Or maybe a little before that. Why? Quake.
Quake3 used full on 24bit color and big (256x256) textures. The 3dfx cards, including the voodoo3 and banshee, couldn't handle these larger textures OR the higher bit depth rendering. Nvidia beat them on features, and later framerates.
I traded my Banshee card for a TNT card back in the days when Quake3 first came out. After my friend saw Quake3 on the TNT card he wanted to trade back. Yes, it WAS that much of a difference.
These days, it's more about drivers and price/performance. Visual quality is pretty much the same to the human eye on all the high end cards, and framerates like 90fps+ are about all you can see. Anything more than that is just extra horsepower for higher details or the latest and greatest game.
Nvidia won't end up like 3dfx at this point in the game. It's just the two headed monster of ATI and NVIDIA, each with their own fans and pluses/minuses.