Microsoft Fuzzing Botnet Finds 1,800 Office Bugs

CWmike writes "Microsoft uncovered more than 1,800 bugs in Office 2010 by tapping into the unused computing horsepower of idling PCs, a company security engineer said on Wednesday. Office developers found the bugs by running millions of 'fuzzing' tests, a practice employed by both software developers and security researchers, that searches for flaws by inserting data into file format parsers to see where programs fail by crashing. 'We found and fixed about 1,800 bugs in Office 2010's code,' said Tom Gallagher, senior security test lead with Microsoft's Trustworthy Computing group, who last week co-hosted a presentation on Microsoft's fuzzing efforts at the CanSecWest security conference. 'While a large number, it's important to note that that doesn't mean we found 1,800 security issues. We also want to fix things that are not security concerns.'"

Comments

If only this was easier...

[Manip]

This is a great methodology of testing but to be honest I'm not sure it is within the scope of most software firms. While I'm sure we could all drop entirely random data into a parser and see if it fails, to REALLY conduct a test you have to do the same thing broken down by data element in the file format and then for each of those test both realistic and unrealistic test cases.

Then you throw on top of that UI and Web-Page fuzzing and you now have to somehow hook every element on a site and throw in random data which is not realistic with a large rich application.

Re:If only this was easier...

[digitig]

The solid theoretical ground would be fine if they were starting from scratch now (and looking at some of the research coming out of MS, they do seem to be trying for it -- we'll have to wait and see whether it delivers). One big problem for them, though, is maintaining compatibility with ealier versions of Office which were not written using what is now current best practice. Once you start trying to implement code with behaviour that's not properly understood, or pulling in code that's not properly understood, then that best practice is some help, but it doesn't give you the robustness you might want. The alternative would be to abandon back-compatibility, but that would throw away all their (perceived) lock-in and make it too easy for customers to jump ship, so that would probably be a bad business decision.

Re:If only this was easier...

[BitZtream]

Its only a great model for testing if you've exhausted the extensive list of known bugs that people hit every day under common circumstances.

Finding bugs in the file format is great and all, but fixing the bugs that users actually see every day is far more important and you can reset assured it will be released with a bucket load of very obvious bugs that should have been fixed rather than dicking around throwing random data at it.

I know there are potential security issues to deal with and those are important, but they still aren't as important as the users experience with the software and actually getting their own job done. Saying 'don't open word docs from someone else until this is fixed!' is a lot more practical than hearing that person say 'I'm not using Word, this retarded table layout bug is pissing me off, can we find something else to use instead of Word?'

I'm not saying they shouldn't, I'm just saying their priorities are wrong on a scale that is hardly imaginable.

As far as being realistic with a large rich application ...

Citation Needed.

Given enough processors sitting around the size of the application or its feature set becomes of little concern. It may take longer but thats no excuse to not do it.

Re:If only this was easier...

[plover]

I wouldn't knock what they're doing. As we've recently seen with Adobe, exploits in the payload format can be used to manipulate users and even launch code. And remember how we used to be all panicky about Word macro exploitations until the defaults were changed to shut them off? "Good times", indeed.

Consider that Microsoft dominates the market, and that the ".DOC" format is widely accepted across companies. Nowadays .DOC files are readily passed by email filters, web filters, etc. Office workers open them in previewers and Word without giving a second thought to security.

A buffer overrun or other fault in the handling of .DOC files could offer a hacker a way to deliver a malicious payload through channels that are today trusted worldwide. For all we know, these could already be exploited by phishing attacks.

It's definitely worth Microsoft's time and effort to execute these tests.

Re:New bugs

[GF678]

I wonder how many "new" bugs they'll create by fixing the found bugs

Yeah, just like the numerous regressions I see in the Linux kernel, WINE, Ubuntu releases etc.

Re:xkydgtufhlofhil

[troll8901]

ghulkgiplgbvihlnk luioguilgil.bjohj110-o; Huto;bn

I don't understand this Score:4 Insightful comment. Can someone explain?

Or user a sales.

[leuk_he]

It is an alternative to the monkey test: Take a sales person from across the ahlloway and let him click on your application. If it does not crash or give absurd error messages you can do the actual testing.

GIGO!

One would think that this is the case...

[WD]

What you describe is "smart" or "generational" fuzzing, where you have a detailed knowledge of the target that you are fuzzing. The thing is, dumb (mutational) fuzzing is still effective. Very effective. Check out Charlie Miller's CanSecWest presentation - An analysis of fuzzing 4 products with 5 lines of Python
http://securityevaluators.com/files/slides/cmiller_CSW_2010.ppt

In 3 weeks of (really) dumb fuzzing, 174 unique crashes in PowerPoint were discovered.

Re:Speaks to the complexity

[140Mandak262Jamuna]

So why don't you do something instead of constantly griping? Find some open source project that comes close to what you want and contribute to it. Even if you are not a developer, work on documentation, testing, bug reporting or something.

Re:wow imagine that

[Blakey Rat]

I don't know of anyone who does regular fuzzy testing. Everyone that matters does unit testing.

Just FYI, Microsoft does fuzz testing in all areas of business, not just Office. The "news" here is really that the Office fuzz testing is done with a cluster of the developers' own computers. (Although it's definitely a good story to get out to all the shitty software houses out there that don't already do fuzz testing.)

When I worked in Xbox game testing back when the Xbox 360 was shiny and new, we had a large pile of Xbox 360s that did nothing but fuzz-testing of new titles by feeding them random controller input.

Re:xkydgtufhlofhil

[jonadab]

It isn't just names, either. Apostrophes and other special characters show up all kinds of places in data where naive programmers tend to imagine they won't appear. Did you know a less-than symbol can show up in a book title? Oh, yes, and if you aren't doing entity-encoding when you build HTML from the data you will get a surprise. With experience, you eventually learn to write the code so that it will either accept those characters as part of the data and handle them as such, or in cases where that's not desirable (like, say, non-numeric characters in a year field) catch them preemptively and issue a clear error message to the user. SQL injection is a particularly easy thing to fix, because you can just use prepared statements with bound variables, but nearly every program of any size or complexity is going to run into situations where it has to do more complex data checking. User-entered data is going to have stuff in it that you didn't anticipate. Every programmer has to learn this lesson, and most have to learn it repeatedly until they eventually become near-paranoid and borderline obsessive about it.

I was gratified when users came to me complaining that they got an error message about time travel not being permitted. Ha! I *knew* it was a good idea to write a test for the end of an appointment being before the beginning of the same appointment. I don't even want to *think* about the bugs that would have ensued if that data had got into the database. The routine that checks whether a room is free at a certain time wouldn't have handled it correctly, that's for certain.

You're not just paranoid. The data really are out to get you. You have to be ready for them, ready for *anything* they may throw at your code. If you're not careful, they'll get you.