Slashdot Mirror
Benchmark Software For Windows 7 Rollout?
tdisalvo writes "We are doing a Windows 7 rollout and I will have to compare major PC vendors. I am looking for vendor-neutral tests that will give me the data I need to present an educated opinion to my CIO. Clear, pretty charts are nice since it is for C level execs, and we need to make it understandable for nontechnical as well as technical people. More specifically, I am looking for something that will clearly show how the same processor performs (better or worse) with a particular build, motherboard, RAM, power supply, etc. My plan is to get very similar machines from major vendors and see which one's build has the highest independent benchmarks. Something with which I could test multiple computers and report on the differences in score would be ideal."
As usual, free is an advantage.
The only thing I can say is a lot of benchmarking software that offers charts and nice graphs tend to be skewed. Not all of them however. A lot of hardware companies design the parts to get somewhat abnormally high results on benchmarks, thus inflating the numbers, and providing inaccurate results. Your best bet is unfortunately more time consuming. You should have multiple software testing the machine, and then make your own chart. This is much more accurate. Try rendering a 1080p video file and record the amount of time it takes. Things like that.
It doesn't run on Windows but Phoronix Test Suite would give you a good baseline for the hardware.
Is this really necessary for a Windows 7 rollout with corporate desktops? Most machines are already overpowered for the average user using Office and what not.
I'd think the cost per machine for good 3-4 year warranties would be more important. At least, it has been in my experience.
I could see doing something like this just for developer machines, but general roll-out? I dunno. Seems like you'd just compare pricing and go with the one that makes the most budget sense.
You haven't said what you actually do with these computers. The relevant benchmarks should look like your actual workflow, otherwise you are just drag racing.
I haven't benchmarked in a while, but this new game came out recently. iD Software's Quake3. All of my hip friends use it to test their machines.
My Slot-A AMD Athlon rocks out like 75 frames a second! Try it out!
Phoronix Test Suite ( http://www.phoronix-test-suite.com/ ) supports Win7 now. It also allows comparison against OSX and Linux ( http://www.phoronix.com/scan.php?page=article&item=linux_windows_part3&num=1 ).
It's Free, it's Open Source and has a bucketload of tests already. You can combine result sets and you can even get the results uploaded for comparison at http://global.phoronix-test-suite.com/
Creating your own tests is nice and easy too.
(Full disclosure - I am one of the project members).
Questions:
A. Do have the authority to make the decision?
B. Are you tasked with giving him your "expert opinion" on the matter?
C. Are you tasked to actually educate him enough about a technical decision that he has no technical skills to currently evaluate an answer?
Answers:
A. Evaluate on the specs you know are important on the job, give him a specific brand, and say "trust me, buy these"
B. Evaluate on the specs you know are important on the job, give him a specific brand, and say "trust me, buy these"
C. You're boned.
Just go to Best Buy and have your computers optimized by Geek Squad. It'll run circles around anything else on the market.
This is pointless. Really. All the machines will test within a few percent of each other. It's not like a Dell is significantly faster than an HP (especially if the software image is the same).
If the machines have different CPU/Chipsets/Video Cards, that's a different story, but a PC -is- really just the sum of its parts.
Tell the C-level execs that the best value would be to skip the benchmark and go right to the bidding, let the vendors undercut each other for an extra month.
"Sometimes, I think Trent just needs a cup of hot chocolate and a blankie." -Tori Amos on Nine Inch Nails
Windows 7 actually uses MORE resources over XP, and slightly less than Vista. If you want to present something, make sure it's on Win7's core improvements (e.g. disk read perf with AHCI), x64 performance (if they have x64) (this is important, Win7 x64 blows the shit off Win7 x86) and app performance. I don't really know of any testing tool that can measure all of those and have a score (like 3DMark), but the built in Resource Monitor does well (although it uses percentages)
While Windows domains and management tools work fine with mixed versions, it is still the very easiest if you have all one version. Well, new systems are going to come with 7, so makes sense to go all 7 if you want to do single system.
Also it is time to start looking at an XP retirement plan for enterprises. Extended support will terminate August of 2014. So, while it isn't a crunch, it is the kind of thing to start thinking about. Better to have a plan than to wait 4 years and find out that now you have to move fast before security patches stop.
Why should C-level execs care about what model processor is used in their computers? Office users aren't looking for the absolute greatest performance, they're looking for reliability, manageability, and cost. I can guarantee that no typical* medium or large size business will make a decision on which vendor to use for office computers based on the performance benchmarks. Frankly, who gives a shit about the motherboard in a typical office user's computer. It doesn't matter, certainly not to upper management. Choose something that has a reasonable cost, a solid long term support contract, and is easy to manage in your existing environment. If anything, the support contract, expandability (adding dual monitors later, or adding more memory for heavy data analysts or future software upgrades), and the existing vendor relationships are far more important than performance benchmarks. *Assuming they're not using them to render lots of graphics or do other very specific, specialized tasks.
Every time you post an article on Slashdot, I kill a server. Think of the servers!
Performance? Really? Personally I'd want stability, reliability, and top notch support. Your average computer user loses far more productivity from downtime due to cheap hardware dying, unstable drivers, etc than to their machine starting (insert app of your choice) .2 seconds slower. I want to be able to order an exact replacement 2 years down the road if a machine dies. I want replacement parts available for the forseeable lifetime of the machines on which I standardize.
Does your company have any contracts with a vendor now? If so you may not have much choice. But I would factor in 5 year (or greater if they offer it) warranty. This way the machines are covered against hardware failure for that time. Since not replacing these machines for a while sounds like an idea, go a bit bigger to cover you for the long haul.
You didn't say what this is for. Regular office apps, cad apps, number crunching apps, etc. The intended use of the machines really effects hat you should be looking for. A 3GHz quad core with 16 GB of RAM, 1GB video card, and 500GB hard drive should be an OK start for a CAD machine. That would cover the regular office use as well. It should cover regular office use for years. It might not cover the CAD use for years. You need to factor in the intended use of the machines as well. Also desktop or laptop?
You forgot to mention what kind of workload you're needing the machines for. A machine used for CAD or FEM used in construction or achitecture will have very different requirements from a database-heavy stand-alone machine that might get used for human resources or controlling.
What business are you in?
That said, for most applications or average office users, terminal services plus thin clients are the way to go. It may look similarily expensive compared to individual PCs at the first look but if you have/are a capable Windows admin, your business is gonna save hugely on deployment and management costs.
That's the number you want to present to your CIO.
...but do include Gig ethernet and a big fat pipe to the 'net.
The sooner the employees get their porn downloaded and get back to work, the higher the productivity on that little dual core.
Orwell: "In a Time of Universal Deceit, telling the Truth is a Revolutionary Act"
The cost of you doing the benchmark and the report will probably be bigger than the money you will save by doing it.
Salut a toi EX Punk anarchiste devenu nouveau mouton conformiste...
Or a similar package is best. You get a nice number you can put into graphs and/or powerpoint presentation which such top brass is known to like. IMHO as long as your desktops are reasonable I wouldn't worry. The future is going to need good network infrastructure performance. Focus on gigabit ethernet, as someone mentioned above.
After logging in slashdot still does not take you back to the page you were on. It's been that way for 20 years.
What type of workers do you have? It makes a huge difference if you are rolling out just business desktops that do nothing other than an office suite, email, web, and DB queries, versus whether you have folks doing CAD, Engineering, Scientific or Creative applications. For the former, any modern computer will probably be more than enough, for the later some users will need the most powerful computer you are willing to pay for. For instance many 3D applications can make use of a huge number of cores for rendering, compositing, manipulating dense 3D models, or image filters, max out your ram, and put the hurt on top end dual graphics cards.
Like they will know you did?
Here's what you do.
You go to techbargains.com slickdeals.net techdealdigger.com techdeals.net etc.
CTRL+F DELL
Note specs and prices.
Do this for a week.
Then next week, jump on the first deal that meets or beats the best deal from last week.
Then order up a bunch of machines.
If the number you're ordering is an issue, just call Dell, ask for the supervisor, and then get X machines at the quoted price after agreeing to upgrade them all to the 3-year, NBD warranty.
Corporate will love the price.
Whoever manages the machines (you?) will love the NBD warranty for when a PSU fails, or a fan starts getting noisy. (When, not if.)
You won't have had to do any real work.
Everyone wins.
Especially when you can tell them that the required EULA won't allow people to publish benchmarks. You get out of this one easy but don't let them read any of the EULA or they might wonder how businesses can accept that and run Windows.
LoB
"Anyone who stands out in the middle of a road looks like roadkill to me." --Linus
Not only were the machines similar they were virtually identical. Unless you were looking at the case badge, or the PCI vendor strings, you would have been hard-pressed to tell which was which. Same intel silicon, very similar HDDs and optical drives(not that that really mattered, neither party was willing to quote anything other than a capacity, so the brands we got in the test boxes were assurances of nothing). The RAM was within spitting distance of one another and(again), the vendors would assure us of nothing other than "X capacity, verified compatible) so it wasn't as though the specifics of the test samples told us much.
We ended up going with Dell, just because they were cheaper, their driver download pages are modestly less unpleasant, and their "ImageDirect" tool is actually pretty handy.
Unless you have particular reason to believe otherwise, exhaustive benchmarking will be a waste of your, and the exec's time. The only exception that I can think of would be if you were advocating for something unusual but potentially interesting(ie. Most corporate desktops are brutally I/O bound, straining under the load of A/V, constant patches and updates, and so forth. SSDs would make them fly, comparatively. Particularly if your company actually has a lot of expensive people running around, a "number of minutes from cold boot to productivity" benchmark could be eye-opening.)
Spend the time to find a vendor who will work with you and help you through the migration. IMO, PCs with similar CPUs, FSBs, memory speeds won't vary enough in performance to justify the effort of quantifying performance differences.
This is specialized for architects but we went from a five minute "save to central" time cost with Windows XP 64Bit to two seconds on Windows 7. Our cost to deploy ROI was achieved in one week's work. "Save to Central" is an Autodesk feature for writing to a SQL Server database on a server. Talk about low hanging fruit. Management surely understands that when people are no longer standing around yapping, more money is being made; not to mention happier workers as well! Your results may vary. Of course with Windows, any new deployment is faster on the machine than a stale DLL hell one.
What you will see is marginal differences. Other aspects, like component quality, noise levels, support, etc. will be important, the slight performance differences will be completely unimportant.
I would strongly suggest that you are trying to optimize an entirely unimportant parameter and are overlooking several very important ones. Rethink what you actually want.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Here's why this guy is being asked that... Suppose Machine A is "5% faster" than Machine B at the same price point for a common task. Let's say that task is something everyone does often and is easy to measure: booting up. So, if Machine A takes 60 seconds to boot, Machine B takes (0.95*60)=57 seconds--3 seconds faster.
So, here's how the C-level execs think... Say you have 1000 employees, each saving 3 seconds/day in bootup time. 1000 employees * 3 seconds/day = 3000 man-seconds/day. 3000 man-seconds/day * (approx) 225 work days/year = 675,000 man-seconds/year = 187.5 man-hours/year saved! Just think of how much more productive we are due to that 5%!
Of course, that assumes that all your employees are robots and use every second of time productively. To add, by the time the OP gets all the machines, runs the benchmarks, and creates the pretty PowerPoint slides for the C-level execs, this little experiment probably cost the company a lot more than 187.5 hours... (Although you could probably shoehorn a 3-4-year NPV calculation showing a savings for this project...)
Windows 3.1x calc: 3.11 - 3.10 = 0.00
I've used Passmark Performance Test before to bench Windows machines:
http://www.passmark.com/products/pt.htm
Very straightforward for Windows dorks to install and use, and provides lots of simple graphs and an easy engine to make comparisons. I mostly used the demo version, but the commercial version didn't seem expensive.
Also, props to them for providing this handy reference:
http://www.videocardbenchmark.net/
Again, be sure to test in as close to the final deployed configuration as possible. I've seen pretty big differences in e.g. x86_64 vs. 32-bit Windows performance, and even with different drivers installed or different BIOS settings.
There's no business/office productivity software that requires Vista or Windows 7. In fact, I'm not aware of any software of any kind that REQUIRES Windows 7.
You can run everything on XP.
Now ask yourself: "Why are we spending -any- money on upgrades?"
Two paths from this point.
1) Slap yourself, rebuild your corporate image with a nice current minimal build and give users the option to rebuild their machines with said image dynamically, at boot time. This will produce vastly greater productivity than any attempt to upgrade.
OR
2) Continue on your current path to justify your continued employment and claim Windows 7 is necessary and the upgrade is "a must have" to remain competitive.
In no test, on the same hardware, will you see any performance increases, by any time based measurement when comparing Windows XP SP3 vs. Windows 7. Windows 7 will always be slower. Boot time, shutdown time, application launch time, or install time. All slower. And you don't have to take my word for it, break out your stopwatch, you can see it for yourself.
Might this be a possibility? Useful in VDI environments to measure performance. Might have a play during testing in standard PC. http://www.loginconsultants.com/index.php?option=com_docman&task=doc_details&gid=66&Itemid=149
For anything outside media production or CAD, there is almost no point in comparing machines of similar hardware specs these days. You will find that vendor guarantee coverage and time-spans, and response times and quality are all more important in terms of TCO, at least in my experience.
There are plenty of benchmark tests around. For C-level executives, the ones that score the machine overall performance with a number would be the simplest. Also see what customer reviews say.
The simplest way to go (and that's what I always do) - spec out a machine according to what you need + a little extra. Go shopping among the vendors (Apple, Dell, HP, ...) and see what the prices say for the machine you spec'ed out (don't forget all the little additions you need to make to make a machine complete - display, video card (missing in my recent HP buy), keyboard+mouse, warranties, support). Then go shopping for a lower and a higher level performance (with Dell the cheap stuff will include yesteryear's technology and gets very expensive with the higher clock Nehalem-architecture), look at the benchmark scores and compare them in a nice table (not too much info - 3 models & 3 benchmarks will give you a 3x3 raster). Next slide you can compare prices, warranty and support (http://www.laptopmag.com/mobile-life/tech-support-showdown-2009.aspx) for 3 vendors (another 3x3 raster).
All in all in about 5-7 slides you should tell them - this is the machine I recommend, you can also get this lower and this higher performance one. This vendor I recommend because they got A+ support ratings according to consumer reviews, you can also go with this and this, here are the prices. These options you can get extra for this price in case some of the departments need them. If necessary you can also add mobile computers and workstations if that's what your end-users need.
Custom electronics and digital signage for your business: www.evcircuits.com
Benchmarks are for proving whether particular solutions can meet requirements. But you have to start with a weighted list of requirements first, and get agreement on that list, before you benchmark anything. That list of requirements will contain a lot more than the boot-up time or whether you can open a browser window infinitesimally faster. For example, you could equip every user with solid state disks to improve boot-up performance, but could you afford it, would those SSDs provide enough capacity, and would write performance suffer "unacceptably"? It depends on your company's requirements and relative weighting. Speaking of requirements, what are the power consumption requirements (including both direct electrical consumption and indirect cooling energy requirements to combat the generated heat)? PCs vary in that dimension, too, and in how well (and how deeply) they go into power saving modes in the real-world. Energy costs are often important in the lifecycle costs of PCs.
I agree with other commenters about warranties. Self-servicing may be cheaper and better: to keep a small stock of spare PCs and swap machines if there's a failure. It depends to some extent on how physically centralized your users are, though. I also concur with the advice to err on the side of more memory (and memory expandability) versus CPU clock speed, core count, etc.
Hey, We do heaps of these for customers. It is really not about the specific performance, everything is with a few %. If you have got a specific use case, do the hard yards and run the tests yourself. If your order is a few hundred units, the vendors or your SI will help you out with demo / eval units. We had customer wanting W7 tablets, we supplied demo units from 3 vendors and we ran a large number of tests. The most important to them were form factor / weight / durability / battery as you would expect. But the performance of the WWAN 3G card was a key indicator as well. So check what you actually need, if its just "good office performance", then anything is fine. Go for the soft benefits - support online and directly to the users desk, the warranty period and the price. Also longevity of chipset, lifecycle, accessory cost etc. We have the first customers rolling out entire fleets of SSD now, they are awesome in laptops and tick the green box in desktops too. Cheers, Mike
If C level executives want to be involved, or even briefed, on what PC you're picking for a hardware refresh, sell short. If you only *think* they want such a briefing, then spend a little time polishing up your resume this week. You're about to be fired.
If you mod me down, I shall become more powerful than you could possibly imagine.
I hope you were merely tasked with finding benchmarks and that you're just a tech. If your CIO tasked you with picking the next platform and you decided to perform technical benchmarks, then you really missed the boat.
1. First, you need to be analyzing support you'll get. Don't get too hung up on it, but you need something better than a 90-day warranty. There are diminishing returns though, at some point it's not worth getting a 3 year or 4 year contract.
2. Next, you need a vendor that will help you with license management. Being able to audit your licenses for Office or Symantec or whatever quickly will help you. If you don't have volume licensing, now is the time your vendor should be helping you with it.
3. Usability matters a lot, but what matters almost as much is how cool your laptops are. When your marketing director, you know, the one that always wears cool clothes and would have to have his iPhone pried out of his cold dead hands, goes to a conference you better make sure he has the coolest laptop of any of the other marketing geeks. A lot of companies overlook this, but I guarantee you he doesn't want to be carrying around a Latitude E6510 clunker.
4. There's a nice price point right now around $1000 for decent corporate laptops and you'll get about 3 years out of them.
5. You need to be negotiating with your sales rep hard if you're making a purchase like this. Your rep isn't going to be able to make huge discounts on laptops like they can on server equipment or some software licenses, so see if you can get some killer pricing on servers while you're shopping for a big laptop package.
Skip the benchmarks, it's not worth your time. Anything you teach your boss about Core duo, i5, etc will be useless knowledge for him in six months when Intel introduces some new spec.
----- obSig
That part is I think for the .NET Framework, and it is included because it is bundled with Windows Server 2003 and later.
Performance benchmarks for typical desktop office machines are pointless. What is FAR more important is: driver stability/support and vendor support in the case of hardware malfunction. So long as your desktops have > 1gb ram they will be fine for 7, for normal office use.
We're currently a dell shop (sigh), my baseline cut-off for Dell laptops is Latitude D510 with 1GB ram for Win7 pro x86. Desktop machines - anything with similar spec to that is fine for x86.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
a) No matter what you choose, the data will be obsolete/wrong by the time the purchase orders go through
b) Pretty much everybody uses the same chipsets/CPUs/RAM anyway so it's a waste of time.
c) Any PC on the market is easily 'fast enough' for general purpose computing.
What you should really be looking for is good warranty, repair-time turnaround and overall vendor reputation.
No sig today...
What you really need to do is focus on how cool looking the machines for the "power users" are.
"Power users" being the bosses.
For extra Brownie points get them a cool desktop *and* an iPad ("for meetings...")
No sig today...
Benchmark is not most important thing for desktop. What is -reliable drivers (i.e. all hardware components have windows 7 logo as per http://www.microsoft.com/compatibility -good product support However if you really need benchmark: 1) windows 7 has built in benchmark rating - Windows 7 rating 2) for more detailed performance analysis of performance and drivers you can use a) Built in command line powercfg -energy b) Enable driver verifier on drivers (verifier.exe) to see if it cause any blue screens (don't enable on too many drivers at once, or slow system down too much) c) Use xperf.exe, part of the freely downloadable Windows Device Driver kit for detailed performance analsysi on everythign from startup/shutdown/reboots/etc d) if graphics are important there are plenty of bench marking utilities for those such as 3dmark,etc.
What about power consumption? I would imaging that even a small difference in efficiency would correspond to a fairly substantial saving in the electricity bill if it is a large deployment. I would use a kill-a-watt type power meter and factor the idle and load power consumption into the presentation, and then extrapolate that into yearly electricity cost
If these are machines that are regularly going to be utilized to a degree where they are maxing out CPU/GPU availability (high-end CAD, video encoding, rendering etc), you don't need a vendor machine test that costs thousands of dollars. You need benchmarks on the specific CPUs and GPU solutions you're considering.
The advantage there is that (particularly if your GPU is a mainstream GPU and not a CAD-specific workstation GPU), there are loads of sites who've already done the work for you, and update their benchmarks regularly. Tom's Hardware seems to have really dropped the ball, but Anandtech seems to be holding the torch still, and I'm sure someone else can provide links if there are better sites for this sort of comparison review (with pretty graphs and all) than Anandtech.
If these are office machines for people running 3rd party vendor apps, office apps, and web browsers, then none of this is worthwhile, and you'd be foolish to shop outside the triangle of price, warranty and vendor's support reputation. And by 'foolish', I'm being extremely generous.
32GB SSD will just fit OS + office. But not all apps and you still want to have room for swap / temp. Even more if you use Photoshop or other apps that need a lot ram / temp space.
32GB SSD will just fit OS + office.
What does this have to do with anything? Both systems need a magnetic hard drive. Thats 32GB SSD is effectively *free* if you go with the 2GB system instead of the 6GB system.
So what point do you have now that I had to explain the obvious to you?
"His name was James Damore."
On top of that, the only thing that truly matters is what antivirus you put on there. Anything else is overkill.
I uninstalled McAfee yesterday so that I could analyze 1 hour's worth of SP:StatementCompleted logs. Windows is terrible at i/o prioritization. Adobe Reader actually has a speed launcher that opens and closes all of the reader plugins and files, so the antivirus "marks them as safe". I don't buy that, I think the normal windows file caching is what speeds that up.
Anyway, antivirus is going to skew the results, even with 7's improved performance. Don't bother, just get faster hard drives. 10k RPM is the only solution.
Make up some fictional stats using whatever software your comfortable with. They won't know the difference.
Make the winner whichever company will give you a kickback or the cheapest that will give you a promotion.
PROFIT!!!
Before you publish the results, make sure to find out which vendor the CIO prefers so that the benchmark winner will match the vendor that the CIO picks anyway! :-)
Computers to a great extent are commodity items. Most critters will give reasonable performance as a desktop.
Look instead at other TCO factors:
1. Reliability: How often do they fail?
2. Vendor support. How fast can they get you a new part?
3. Lifecycle Longevity. Can you buy the same unit a year from now? Two? Three? Dell had their optimus program that guaranteed a life cycle of 3 years, with the only difference being CPU speed.
4. Driver support.
Third Career: Tree Farmer Second Career: Computer Geek First Career: Teacher, Outdoor Instructor, Photographer.
RAM Memory, CPU, Motherboard, PowerSupply are what constitutes the identity of the computer. You can upgrade hard drives, change the screen, add graphics cards, etc, but changing the CPU or the RAM memory after a computer has had two years, is very very difficult since it is hard to match the compatibility (a new RAM module must be compatible with both the CPU and the motherboard) and old RAM memory chips get difficult to find, and ironically, it has always been the case that the older a RAM chip is, the more expensive it gets to buy new.
New M$ operating systems always push the memory requirements to double or more.
This is the minimum RAM these OS's needed just to install (to work well it was much more, but always proportional):
Windows 3.1 -> 1Mb
Windows 95 -> 4Mb
Windows 98 -> 16Mb
Windows 2000 Pro -> 32Mb
Windows Server -> 256Mb
Windows XP -> 64Mb
Windows Vista -> 512Mb
( Take a look at http://cybernetnews.com/cybernotes-history-of-windows-system-requirements ) Bottomline:
The rest does not matter. Chose the vendor that gives you the best warranty and the best price.
:-) You have to look at the components that the computer has.
It is really really really pointless to see benchmarks or tests when buying new computers, if you want the fastest, then get the most expensive, thats it.
If you are a gamer or a computer scientist focused on algorithms, you might want to optimize the best combination of CPU, RAM, Motherboard (and GPU and HardDrive). Otherwise you just don't.
AND... never buy just looking at the brand that assembles the computer, that's stupid