Reverse-Engineering Consoles

shpoffo writes "In relation to the recent /. article i thought i'd give people a heads-up on some more console info. For those of you with the time, interest and know-how there are a few really good resources around for reverse engineering most of the major video game consoles such as Jeff Frohwein's site for the Gameboy. There's others for the N64 here and here, Dreamcast, Playstation, TurboGraphix, Genesis, and the Nintendo."

Comments

GCC for the N64

[thogard]

You can pick up RPMs for GCC at

http://n64dev.50megs.com/

Then all you need is a transfer device to copy your code to the N64 but Nintendo has sued Bung so they have stopped selling them.

How I reverse engineered a file format

[goingware]

I don't know much about reverse engineering hardware, but I thought you might find it interesting to know how I reverse-engineered a file format.

A company wanted to allow its own movie production management product interoperate with a motion picture project management database. It was like a regular project management tool that had specialization for things of interest to the movie industry, for example there was a specific category for scheduling potted plants to be present on the set as well as manage the expenses involved with these things.

I think it was called Movie Magic Scheduling.

The publishers of movie magic were very protective about thier product and didn't want to cooperate with my clients, so they hired me to reverse engineer the product.

The initial agreement was that I would do this in a week for $1500. It took three weeks and I was working long days, but in the end I was able to take the whole production schedule for an actual full-lenth motion picture and run it through a parser I wrote that dumped it out into an intelligently interpreted text file.

I started by doing this: I created a new, blank document. Then I created a second document with one of the fields containing only the letter "A". Then I made hex dumps and used a comparison tool to compare the hex dumps.

I quickly found that there was lots of unexplained junk in the files so I wrote two tools that I would run before launching Movie Magic to create each document. This was on a Macintosh. One tool would allocate all available memory, set it to zero, then free it and quit. The other tool would create a large file set it to zero and quit.

Note that on Unix systems writing a page full of zeroes to a file is optimized to not write any data at all; however on a properly written system virtual memory pages and disk sectors are zeroed before being allocated to prevent leaking of confidential information.

Then basically I would note what change in the file and make a hypothesis as to what the cause of the change was, and test out my hypothesis by trying to predict what a further change in the file would do.

I wrote two things to document this, one a file format spec in a word processor, second a C program to parse the file. As my knowledge of the structure got more complex, I could implement the knowledge in the file parser and start dumping files through it until an error occurred. Then I could look more closely at what caused the exception.

The key to being able to do this I got from Robert Ward's book Debugging C (I think it is out of print). That is simply to use the scientific method when debugging software (or reverse engineering it).

The scientific method is simple in principle but it takes some discipline and creativity to use it effectively.

• Observe something
• Make a hypothesis as to why that was observed
• Design an experiment that will test that hypothesis. It is important that your experiment tests the hypothesis that you have made, and not something different.
• Carry out the experiment to see if it confirms your hypothesis.
• If it does, you have learned something
• If it doesn't, you have to make a new hypothesis

All the while you keep watching out of the corner of your eye for weird things to happen.

One thing that helped a lot with this is that I'd spent a lot of time working with word processor file formats while maintaining a spellchecker, and I'd designed a binary graphics format. So I knew what were the common things to do in a file format, and a lot of things could be guessed directly.

It would have been a lot harder if the format was designed to be hard to reverse engineer, but the information has to go somewhere. Even with encrypted data, with a situation like mine you can do what is called "chosen plaintext cryptanalysis".

For another example of that, during World War II we'd cracked the Japanese cipher, but in addition to the cipher the Japanese used code words. To crack the code words, when we suspected that the Japanese used a certain code (call it "FOO") to refer to midway island, naval intelligence sent a command to Midway via secure channels asking them to report that their water desalinator was broken down - through insecure channels.

After Midway sent the request for a new desalinator, we intercepted a japanese transmission saying that "FOO" had a broken desalinator. Because they had also said they were preparing a major attack on "FOO", we knew to meet them with our aircraft carriers.

The choice of hypothesis is a problem here. Robert Pirsig talks about this in Zen and the Art of Motorcycle Maintainance. For any given observation, there are a large number, possibly infinite number of hypotheses that would fit the observed facts.

Choosing which one is worth taking the trouble to test calls for scientific creativity; ultimately the choice is based on scientific quality, and one mark of a good scientist is that they have a sense of what is right - and the flexibility to recognize that their sense can be wrong.