Slashdot Mirror


A Look At Microsoft's 'Mini Internet' For Testing IE

MrSeb writes "With the grandiose bluster that only an aging juggernaut can pull off, Microsoft has detailed the Internet Explorer Performance Lab and its extraordinary efforts to ensure IE9 is competitive and IE10 is the fastest browser in the world. Here are a few bullet points: 128 test computers, 20,000 tests per day, over 850 metrics analyzed, 480GB of runtime data per day, and a granularity of just 100 nanoseconds. The data is reported to 11 server-class (16-core, 16GB of RAM) computers, and the data is stored on a 24-core, 64GB SQL server. The 'mini internet' has content servers, DNS servers, and network emulators (to model various different latencies, throughputs, packet loss)."

53 of 241 comments (clear)

  1. Could use the real internet eh! by Anonymous Coward · · Score: 4, Interesting

    Why not just use the real internet?

    1. Re:Could use the real internet eh! by weszz · · Score: 5, Informative

      They wanted to account for any kind of lag, so by having it all in house and disconnected from even their internal network, they have control over all variables so everything is equal.

      They did this post on their blog yesterday http://blogs.msdn.com/b/b8/

    2. Re:Could use the real internet eh! by greichert · · Score: 5, Insightful

      Because you can not have reproducable results on the real Internet. Only a fake one, where eveyrthing is controlled and reproducable, can be used for testing and making sure some settings do not make the browser slower.

    3. Re:Could use the real internet eh! by ackthpt · · Score: 3, Interesting

      They wanted to account for any kind of lag, so by having it all in house and disconnected from even their internal network, they have control over all variables so everything is equal.

      They did this post on their blog yesterday http://blogs.msdn.com/b/b8/

      They care about it so they created a genuine imitation of the real thing.

      Honestly, I'd go at some of the pages I have to each day, which are ludicrous in their use of content and scripting - web developers just pick up and drop widgets all over the place, never a look toward what impact it has on the page being interpreted or used on the receiving end. I know I've got a bad one when I hear the processor fan kick in for a stinkin' web page!!!

      --

      A feeling of having made the same mistake before: Deja Foobar
    4. Re:Could use the real internet eh! by weszz · · Score: 3, Informative

      they never do say what kind of sites, just that they are sanitized and are real world web pages... but if you are comparing IE to itself or Chrome or FF, whatever site you use should be similar in speeds, except for ones that detect clients and do different things...

      "Content servers are web servers that stand in for the millions of web hosts on the Internet. Each content server hosts real world web pages that have been captured locally. The captured pages go through a process we refer to as sanitization, where we tweak portions of the web content to ensure reproducible determinism. For example, JavaScript Date functions or Math.Random() calls will be replaced with a static value. Additionally, the dynamic URLs created by ad frameworks are locked to the URL that was first used by the framework.

      After sanitization, content is served similarly to static content through an ISAPI filter that maps a hash of the URL to the content, allowing instantaneous lookup. Each web server is a 16-core machine with 16GB of RAM to minimize variability and ensure that content is in memory (no disk access required).

      Content servers can also host dynamic web apps like Outlook Web Access or Office Web Apps. In these cases, the application server and any multi-tier dependencies are hosted on dedicated servers in the lab, just like real world environments."

    5. Re:Could use the real internet eh! by gorzek · · Score: 4, Funny

      Yes, there must be some conspiracy! Microsoft couldn't possibly want to make a good browser! They must have ulterior motives!

    6. Re:Could use the real internet eh! by Bill,+Shooter+of+Bul · · Score: 5, Insightful

      You have to figure out what the variables that you have problems with in real world usage, before you can start optimising your product to account for them.
      There has to be iterative cycles of real world, then fake internet testing to really make it work well.
      It would also help if you were able to test your competition alogn the same lines.

      I additionally wonder if they are accouting for all of the different behavious of all of the various webservers out there. If they are only testing agianst iis, well, that's not very good.

      --
      Well.. maybe. Or Maybe not. But Definitely not sort of.
    7. Re:Could use the real internet eh! by thatskinnyguy · · Score: 3, Insightful

      Because that would be somewhat unscientific. A lab setting is controllable and you would be able to trigger and know where latency is coming from and how to correct certain behaviors in software. With the real internet, it's anyone's guess where lag is coming from.

      --
      The game.
    8. Re:Could use the real internet eh! by gorzek · · Score: 4, Insightful

      So, what's going to make anyone use a Microsoft browser? They've been losing market share for ages because their browsers suck. How do they get people back? Make a good browser. Your argument might mean something if Microsoft sold IE as a standalone product, but they don't. It costs nothing (in terms of cash coming straight out of your pocket) to switch browsers, and users are notoriously not fond of switching. Since you can't count on your competitors' products to be lousy, you can only compete by making yours better. The browser market is about as Darwinistic as a software market could be.

    9. Re:Could use the real internet eh! by smelch · · Score: 2

      If real world users have a problem when their latency is too high, or too low or too medium, they use the mininet to set up that situation, find the issues, and fix them. They aren't running these tests to get statistics on IE performance in the wild for the average user, they're doing it to get statistics on how it performs in specific situations, and trying to get those stats as good as possible so each user has the best possible experience for their specific circumstances.

      --
      If I can just reach out with my words and touch a butthole, just one, it will all be worth it.
    10. Re:Could use the real internet eh! by cjeze · · Score: 2

      IE 9 is actually not bad.I am surprised how well they did it!

    11. Re:Could use the real internet eh! by Phlow · · Score: 2

      If you wonder about that, you must be wondering if their entire team that put this together is chock full of morons. Seriously, I think a little more credit is due here.

    12. Re:Could use the real internet eh! by htomc42 · · Score: 2

      So, how many of those "content servers" with captured real world web pages, are dedicated to porn? (hey, not joking- that -does- make up a huge percentage of traffic)

    13. Re:Could use the real internet eh! by gutnor · · Score: 2

      Well duh ! That is one the major reason you create that kind of lab in the first place: you found something in the real world then you craft a similar scenario in the lab and make that part of either your test case, or benchmark. Obviously part of the crafting the scenario is to make sure that it behaves the same as in the real world. Other uses include debugging, analysis of edge cases (i.e. stuff difficult to find the real world, or stuff that does not yet exist)

      Considering the renewed competition on the browser market, I don't think we can doubt that Microsoft is indeed benchmarking the other browser in its lab. You can also bet that they are running the major benchmark, and doing plenty of real tests. Just because MS is a shitty company burden by a country-level bureaucracy does not mean that they would screw up something obvious like that.

      What is really noteworthy, is how far they committed to IE. They could have dropped the ball and just created a half decent me too browser cheaply from webkit just to please granma and IT.

    14. Re:Could use the real internet eh! by smooth+wombat · · Score: 4, Insightful

      web developers just pick up and drop widgets all over the place,

      Rule #2 of IT that should never be broken: Never let a web designer design your web page.

      Giving free reign to a web designer to design a web site is like giving a two year-old a Faberge egg.

      --
      We will bankrupt ourselves in the vain search for absolute security. -- Dwight D. Eisenhower
    15. Re:Could use the real internet eh! by Bengie · · Score: 3, Insightful

      Probably an evolving tick-tock setup between real and test.

      Step1. Performance Test App against real world
      Step2. Document Real world issues
      Step3. Create test environment to run your issue cases against
      Step4. Optimize app against test environment
      Step5. Goto Step1, adding any new cases

      Steps 1 and 2 can happen independent of 3 and 4. Step5 is just to make the logic seem serial.

  2. And still... by jcreus · · Score: 4, Insightful

    Beaten by Chrome and Firefox.

    1. Re:And still... by Anonymous Coward · · Score: 5, Funny

      The only thing Firefox does fast anymore is update.

    2. Re:And still... by thedonger · · Score: 5, Insightful

      And when all we care about is the fastest browser - in nanoseconds! - will we begin to forget the truly important criteria for choosing a browser?

      Or better still, by the time IE is on par with Chrome the actual browser will be irrelevant because mobile platforms - in which IE has little share - will do to traditional computers what Cromagnons did to Neanderthals. The next generation will use integrated devices, unaware they were using a browser, and with little or no need for even a choice.

      --
      Help fight poverty: Punch a poor person.
    3. Re:And still... by gorzek · · Score: 2

      Oh snap!

      Yeah, that's why I ditched Firefox years ago for Chrome. Got sick of FF freezing/crashing all the time, as well as its performance just getting worse and worse over time.

    4. Re:And still... by mcrbids · · Score: 3, Insightful

      Firefox hasn't become any slower. What's happened is that everything else become so much faster. There used to be a "pregnant pause" when entering a domain, I remember when 30 seconds to load a page was the norm, over 56k.

      Now, I expect the results from a google search to appear as I type, interactively. This isn't just an improvement, it's a whole 'nother animal at that level of performance.

      --
      I have no problem with your religion until you decide it's reason to deprive others of the truth.
    5. Re:And still... by n5vb · · Score: 2

      The next generation will use integrated devices, unaware they were using a browser, and with little or no need for even a choice.

      And little or no understanding of how it works or how to use it as anything other than yet another few-to-many information channel they can listen to or watch, but can't talk back to in any real sense. And you're right that that's the direction it's going, but some of us aren't thrilled about that..

    6. Re:And still... by gorzek · · Score: 2

      All true. Although I have a friend who uses Opera, and she was aghast at the way Chrome renders a white screen before it starts rendering the full page. She said under Opera, the full page just shows up all at once with no weird white screen first. I'm not sure but I think Opera's renderer might be a little faster than Chrome's.

    7. Re:And still... by arkane1234 · · Score: 2

      Funny, I'm aghast at requiring everything to be downloaded and rendered before it's displayed. That gives the illusion of nothing being done, and subtly irks me.

      --
      -- This space for lease, low setup fee, inquire within!
  3. not a true real-world test by Anonymous Coward · · Score: 2, Informative

    until they add some zombied computers and malware control servers.

    oh, wait.. these are windows test systems. never mind. some kind soul probably already added them

  4. But will it run Linux? by erroneus · · Score: 4, Interesting

    I couldn't resist. But with all the work and effort and resources going into this, how is it that operations a tiny fraction of this can generate fast, reliable and standards complaint browsers better than MSIE?

    Microsoft, the problem isn't that you're not spending enough money. It's that you're not doing it right.

    1. Re:But will it run Linux? by roc97007 · · Score: 3, Insightful

      I couldn't resist. But with all the work and effort and resources going into this, how is it that operations a tiny fraction of this can generate fast, reliable and standards complaint browsers better than MSIE?

      Microsoft, the problem isn't that you're not spending enough money. It's that you're not doing it right.

      I'm not familiar with IE 8 and 9, but in the past the issue was many years and revisions of code reuse and accumulation of cruft, an insane amount of backwards compatibility and some poor initial design choices combined to make each new version bigger, slower and buggier. I imagine this combines to make developing and especially testing any new release a massive undertaking.

      To be fair, I would argue that Firefox is starting that downward spiral now. Each new version is slower and has a bigger footprint. Personally, I've switched to Chrome, but when Chrome inevitably starts lagging, I'll be on the lookout for the next completely new browser. Not merely because it's new, but because it's less likely to have years of bad decisions weighing it down.

      The problem in Microsoft's case is that they seem incapable of dumping what they have and doing a complete rewrite. There may be marketing reasons for this, granted, but if they made a clean break it'd be better for them in the long run.

      --
      Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
    2. Re:But will it run Linux? by Anonymous Coward · · Score: 2, Interesting

      yes
      http://bellard.org/jslinux/

    3. Re:But will it run Linux? by jader3rd · · Score: 2, Interesting

      It's 2012 and IE9 still doesn't have a built-in spellchecker for text areas!

      If IE had a spell checker you'd call it bloat. When it's still 2012 and browsers running on Windows 8 won't need to worry about it because it'll be built into the OS, would that relieve your frustration?

    4. Re:But will it run Linux? by Mr+44 · · Score: 2

      The problem in Microsoft's case is that they seem incapable of dumping what they have and doing a complete rewrite.

      It's really tempting to think that way, but actually doing a clean re-write is usually a complete mistake. I'd say Mac OSX is one of the only successfull examples of this (and that largely because their previous versions were so horribly outdated it was unbelievable).

      Joel has a great article on this from a while back:
      http://www.joelonsoftware.com/articles/fog0000000069.html

    5. Re:But will it run Linux? by K.+S.+Kyosuke · · Score: 2

      To be fair, I would argue that Firefox is starting that downward spiral now. Each new version is slower and has a bigger footprint.

      As opposed to Chrome, which (on my machine, at least) has an even larger footprint?

      --
      Ezekiel 23:20
  5. 1/10,000 of a millisecond by tepples · · Score: 4, Funny

    It means that each millisecond of ping is divided into over 9000 parts.

  6. Re:Granularity of 100 nanoseconds by ackthpt · · Score: 4, Funny

    Granularity of 100 nanoseconds: What does that mean?

    That's as small as they could get the bits, pounding on them with Steve's chair.

    --

    A feeling of having made the same mistake before: Deja Foobar
  7. Re:IE Crap by Krojack · · Score: 5, Funny

    And still unable to correctly implement CSS2 and HTML4

    Fixed that for ya.

  8. Congrats by layabout_guy · · Score: 2

    Whatever the result I hope microsoft do well that in turn will push competitors and we the users should hopefully benefit. Though I feel IE has a long way to go..

  9. Re:Was /. been bought or what? by weszz · · Score: 2, Interesting

    the building windows 8 blogs are actually pretty good, as long as you know that it's going to be pumping up how cool windows 8 will be, and question what is in the cool-aid you are drinking, it's worth a read now and then. They give background information on decisions they are making and why some things are the way they are as well as where they plan to take them.

  10. HHGttG by Verdatum · · Score: 3, Funny

    Wasn't this a plot-point in the Hitch-Hiker's Guide to the Galaxy series? Having an artificial universe on-site so they could go exploring but still be able to come back for long lunches...

  11. Re:IE Crap by Anonymous Coward · · Score: 5, Insightful

    Correctly according to whom? Neither CSS3 nor HTML5 are completed standards and various portions remain in flux. Of the more mature bits IE9 and IE10 implement quite a bit of it and do so quite comprehensively.

    MS is also one of very few organizations that is very actively involved with the W3C Test Suite by submitting test cases for each portion of the standard under various circumstances to demonstrate correct behaviors. What Google and Mozilla do instead is slap together a partial implementation and call it a day. More than once has their implementations been found to be not only incomplete but also incorrect.

    Stop relying on scores given by non-authoritative tests demonstrating exceedingly limited and selective interpretations of non-standardized functionality. Oh, and HTML5 Video does NOT specify a codec, in fact it was designed to handle many simultaneous codecs, including h.264, which is explicitly referenced in the draft.

  12. Re:Might try for a smaller footprint by weszz · · Score: 2

    no no... those are what COMPILE the results...

    They have a wide range of high end to low end PCs running the tests down to a 1.6ghz netbook with a Atom N270 processor.

  13. Mini Internet? by Smask · · Score: 4, Funny

    Does that mean they have only porn sites with midget porn? And a mini 4chan, populated with toddlers?

  14. Re:Was /. been bought or what? by gorzek · · Score: 4, Funny

    What we really need is another Bitcoin story!

  15. Re:IE Crap by Bill,+Shooter+of+Bul · · Score: 2

    Well, sometimes blaming tools is an excuse, other times you have no choice but to speak up and say, this isn't a pool stick, its a baseball bat.

    --
    Well.. maybe. Or Maybe not. But Definitely not sort of.
  16. Re:Was /. been bought or what? by jellomizer · · Score: 2

    Oh come on... Now that Microsoft has been pushed to #2 with Apple #1 we have to start liking Microsoft and Hating Apple. As the only true measure of intelligence is hating what is popular and mainstream.

    --
    If something is so important that you feel the need to post it on the internet... It probably isn't that important.
  17. Re:Granularity of 100 nanoseconds by X0563511 · · Score: 3, Insightful

    It means the runtime data they have collected is stored in a 100ns interval.

    --
    For large sets, this will be our guide even unto death, for the LORD will work for each type of data it is applied to...
  18. Re:Was /. been bought or what? by treeves · · Score: 2

    I was just going to comment on the abundance of nuclear-related stories that have been running on /. of late - and I'm a nuclear supporter. I wouldn't be complaining though, just making an observation.

    --
    ...the future crusty old bastards are already drinking the Kool-Aid.
  19. Firefox isn't slow at all. by sakdoctor · · Score: 4, Insightful

    While slashdot mocks the computer industry marketing for describing computers using a single metric, you seem to be quite happy with that when it comes to browser performance.

    An example: Chrome (v8 engine) has this reputation for amazing speed, but IE9 absolutely grinds Chrome into the dust when it comes to simply repositioning elements on screen; something which today's web apps spend a lot of their time doing. You can feel it too if you know what you're looking for. I don't follow IEs development as closely as Chrome or Firefox, but IE must be hardware accelerating these translations.

    I fully expect Google to focus on performance cases which help their specific apps. Again, a conflict of interest, akin to Microsoft pre-caching masses of junk, so that Office can appear to start up much faster than the competition.

    1. Re:Firefox isn't slow at all. by ifrag · · Score: 2

      so that Office can appear to start up much faster than the competition

      Ah yes, "the competition"... hmm, who exactly is that these days? Yeah, please don't say Libre/OpenOffice.

      I'm not at all saying MSOffice is good, because in a lot of ways it's terrible, but honestly thinking there is real competition now is a bit outrageous.

      --
      Fear is the mind killer.
  20. Why not? by kiwimate · · Score: 5, Insightful

    MSDN blogs are often very technically detailed, written by people who know this stuff from the inside, and if it's about a topic that's of general computing interest then it seems that's a good thing. And the blog in question is chock full of some really good detailed stuff about how they're doing performance testing, reasons why the lab is architected the way it is, detailed graphics on how they measure performance, how they analyze it...on and on.

    Frankly, this seems more akin to old Slashdot than a lot of the nonsense we see here today. (That story the other day about a girl sent home from school because her lunch wasn't healthy, and then quickly called into question over what happened? Really? What was that topic even doing on Slashdot in the first place?) Whatever you think about Microsoft, having this extremely detailed look into how one of the world's biggest software vendors (or are they the biggest now?) goes about performance testing, and how they ensure consistent results, should be really, really interesting to anyone involved in IT.

  21. Re:IE Crap by sakdoctor · · Score: 2

    IE is crap, but not for that reason. IE9 is missing some very-nice-to-have bits of HTML5 just because it was released so long ago.

    As for CSS3, yes it's deficient in some areas, but there is a trick to get around many of them: SVG
    Unless you have to support IE8 (LOL poor you), then the users will never notice the smoke and mirrors.

    With JavaScript polyfills, it's possible to take advantage of the latest and greatest in IE9 with minimal effort.

  22. Re:Granularity of 100 nanoseconds by Renegrade · · Score: 4, Informative

    No, it means that Windows has a 100ns granularity on it's timestamps.

    http://msdn.microsoft.com/en-us/library/windows/desktop/ms724284(v=vs.85).aspx

  23. Re:In b4 haters by andreicristianpetcu · · Score: 2

    the real internet has virusez ;)

  24. Re:IE Crap by Anonymous Coward · · Score: 2, Insightful

    Web standards don't work that way. Nothing becomes a completed standard until everyone (well, enough of the major browsers) has implemented it.

    Um, no. Standards word when the standardization committee defines the behaviors, argues through the issues and then rubber stamps the behavior. They don't just take the accidental similarities in draft implementation between Google and Mozilla and standardize "do whatever they did." CSS2 is final and Chrome doesn't implement all of it; just see what happens if you try to combine the first-line selector with a text-transform of lowercase. Conversely there are behaviors which all of the browsers have more or less agreed upon but which are entirely non-standard, like window.screenX.

  25. Re:Was /. been bought or what? by shutdown+-p+now · · Score: 2

    You're holding it wrong.