Slashdot Mirror
Reverse-Engineering Consoles
shpoffo writes "In relation to the recent /. article i thought i'd give people a heads-up on some more console info. For those of you with the time, interest and know-how there are a few really good resources around for reverse engineering most of the major video game consoles such as Jeff Frohwein's site for the Gameboy. There's others for the N64 here and here, Dreamcast, Playstation, TurboGraphix, Genesis, and the Nintendo."
I am a professional console game programmer.
There is a major difference between creating product for a hand-held old-school console like the Gameboy Colour (GBC) and for the N64. The GBC has a very simple architecture, and by its nature has a limited range of game types that can be made for it. It is an ideal platform for hobbyist game programming (as long as you don't want to do fancy 3D graphics). In fact, I'm using it to introduce my children to assembly language programming and basic computer architecture.
It is feasible through "pure" reverse engineering to determine how to program the Gameboy - although, in fact, most information has leaked out through authorised developers who have the actual documentation.
The N64 on the other hand is a very complex system that derives much of its speed from running pre-compiled graphic processes (display lists) in parallel with the CPU. What is more, it relies on microcode to drive the graphic process. The microcode can be totally reprogrammed and this makes it very powerful. [Unfortunately Nintendo would not initially release details on how to program the microcode even to authorised developers, and so I had to reverse engineer it in order to render more complex surfaces than triangles].
IMHO almost all of the actually useful information on the N64 has been leaked and not reverse engineered. Even my reverse engineering of the microcode relied on a certain amount of social engineering with Nintendo engineers.
StrutterX
A couple of links to some interesting info.
2 90.html (view the contents of a GD-ROM)
f m?menuselection=search2&p_line=SH-4 (SH4 Tecnical Documentation)
s ega/dreamcast/technical_pages/reburningCD. shtml (GD-ROM info)
http://www.cdrom-guide.com/ubb/Forum19/HTML/005
http://marcus.mangakai.org/dc/ip0000.bin.html (Dreamcast boot file)
http://semiconductor.hitachi.com/superh/index.c
http://www.hitmen-console.org (Dreamcast Debug Handler)
http://marcus.mangakai.org/dc/serifc.html (Serial connection between DC and PC)
http://ancient.gameznet.com/files/home_console/
It's funny how my site often gets overlooked - perhaps 'cause I don't code it's not as cool.. ;)
www.gamesx.com is the inernet's largest collection of gaming hardware references bar none, as near as I can tell. If you've ever wanted a pinout, it's probably there. I'm an update slacker though, so the last thing I had up was a bit on the neato analogue buttons on the PS2 pads.
There's also a mirror of Joakim Ogren's fantastic Hardware Book.
The whole point of patents is to bring innovations into the public domain in return for granting a monopoly of limited duration.
A company that chooses to use trade secrets instead of a patent is declining the protection that patents offer, and is also withholding innovations from ultimately being in the public domain. There is no public interest in giving them protection.
I would suggest that for this reason reverse engineering is in the public interest. Either the innovation is brought to light forceably, or the company follows the proper route and brings it to light with proper documentation as is required by a patent.
-- Could you use my software consulting serv
Reverse engineering has been around for a long time but was not tested in court (I believe) until 1992 when Sega brought a suit against Accolade. Accolade allegedly violated intellectual property by figuring out how Genesis cartridges worked and producing their own.
Accolade won the case by stating that reverse engineering was legal for decades without suit (among other statements, of course). One of the stated examples was the telnet terminal application, which relies on backwards engineering of the old DEC and Tektronix terminals of the 70's.
Because of this case, a legal precedent was set (albeit from lower than Supreme courts) which said that reverse engineering was legal and did not violate trade secrets or intellectual property.
There is also an other Atari vs. Nintendo (remember the Donkey Kong cartridge tiff in the early 80's?) case which had similar results.
For more information and a better explanation, try here
You can pick up RPMs for GCC at
http://n64dev.50megs.com/
Then all you need is a transfer device to copy your code to the N64 but Nintendo has sued Bung so they have stopped selling them.
>How come if I undercut software sales by making and selling software copies, I am called a pirate?
:-) So I, like a lot of consumers, choose to avoid this possiblilty by not buying Macintosh. This leads to less market share, and less exposure for the company. Less sales. Bad. If Macintosh were still allowing clones, well, I'd have a Mac, NOW (once the clones force the prices back down to earth).
Your copy is identical. You would be breaking the law if you made a Nintendo(tm) that used the identical parts and identical shape and identical trademarked name on the front of it.
>Yet if I undercut hardware sales by making and selling software copies of the console, I am called a emulator who "reverse engineered" this console, or other words that legitimizes what I did.
Reverse engineering implies that your "copy" is similar, not the same. You certainly CAN do this with software.
For example:
When the IBM PC was released, it was $3000 (or so). Much more expensive than IBM should be charging. So Compaq decided that they could make money by selling a compatible (read: clone, copy) machine. But they were stuck: To make it compatible, they needed to use the IBM BIOS (which IBM wasn't going to license to them). But, ahhh, why not do to the BIOS what they had done to the hardware? Make it compatible, but not identical.
They hired engineers without a clue of how the IBM BIOS was programmed, stuck them in a room, and had them reverse engineer the IBM BIOS, without looking at the code. Just by trial and error, and scientific hypothesis. They then created their Compaq BIOS, which provided the same services, but using their own ideas. And, there you have it, a 100% compatible BIOS not using any IBM code. Totally legal, since NO copyrights had been broken. And, this is an example of "copying" software.
Now, if you wanted to, you could do the same with Office 2000, if you had the time and money and wish to do so. You wouldn't be breaking any laws (except, perhaps, "look and feel", which could be worked around). You would end up with Office 2000, using your own personal code set. In essence, a copy, or clone of the software. But NOT a pirated copy.
That's how a console is legally reverse engineered any copied.
>Either way, my actions hurt sales by copying someone else's product and not producing anything new.
You produced something new. You produced a new console that works like the old one, or a new software that works like the old one.
>How is that good?
Because it provides the consumer with protection should the original company go out of business, or choose not to continue the product. Imagine if there were no IBM clones, but IBM had become popular. What if, somehow, IBM suddenly went out of business? Ohoh... now there's trouble. But, now with clones abounding, the world won't come to a standstill today if IBM dives.
Also, for example, those of us who still use Nintendo (original) (like me) like the fact that clones are avaliable since Nintendo has dumped this console. See how the world benefits from this? When my Nintendo original sets on fire, I can buy a clone, and still legally use my cartridges.
And, amazingly, it benefits the original company. What happens if/when Apple goes out business? Yes, you Macintosh users, you are ROYALLY screwed. You can join the Amiga lot looking in garage sales for parts.
>Now either BOTH activities must be evil or they must BOTH be allowed
No. One is illegal, and bad. The other legal, and good.
Verbatim copying is illegal because you add nothing of value. Making _similar_, _compatible_ things is not, because you add the value of protection to the consumer. Why? Because a reverse-engineered copy proves you understand how the original works. You can therefore improve upon it. A verbatim copy shows you have no understanding of the device/software, and therefore cannot sustain the life of the device.
Oh, here's another few examples of copied, but NOT pirated software: Freedos. Linux. DR-DOS. OpenDOS.
Freedos and DR-DOS and OpenDOS are compatible with MS-DOS (reverse engineered). Would you tell me that Freedos, DR-DOS, and OpenDOS add no value to MS-DOS whatsoever?
Would you tell me that Linux adds no value to Unix whatsoever?
The fact is, the law, and morals in general, are designed to protect the citizens as a group, and not some single company. It's a free market. If your company can't stand some legal competition, then pack up and go.
If you could be told what you can see or read, then it follows that you could be told what to say or think - BoC
While there are emulators for the N64, they don't emulating the hardware, they emulate a MIPS cpu and watch for calls to known functions. Those functions are then written natively in things like open-gl. Since my PC is about 5x faster than my N64 running the same code, its in the realm of emulation but if anyone made use of the reality coprocessor (RCP) in the N64, there is no way a PC could emulate it at anywhere close to real time. It can peak out at something like .5 gigflops. Now it would be nice to get a datasheet on the RCP but its not going to happen unless it leaks out of SGI since Nintendo doens't even seem to use it in its own games.
This brings up why it won't matter if the lastest game platform has the latest and greatest unique hardware. There is no game company that is going to spend the R&D money developing a 1 platform game for a console. If they count on the hardware then they can't port it to other platforms.
You can't claim there's anything wrong with making money by creating a better product that does the same thing as something already out there. Lets say i buy a copy of Clarisworks 4.0. It does most of the same things Microsoft Word does. It reads most Microsoft Word documents. I no longer have a reason to buy Microsoft Word. Well, that's certainly a lost sale for Microsoft. Is that piracy?
...
Copying a creation is bad.
Copying a functionality is not bad.
The second implies, as the parent post said, you have created something. You have possibly created something for "immoral" [making money off others' work?] purposes, and it's possible the thing "created" is merely a creation in the form of a reimplementation of something that someone else created. But you _did_ something, at least, and almost certainly in the process added [created] some sort of functionality that was not there in the first place. [even is said functionality is just a nifty cheat mode that is nothing more than a modified version of a thing you used for debugging.]
You may notice my arguments are slightly contradictory. That's because the _situation_ is contradictory. _There Is No Right Answer Here!_ No matter how the laws are set up, there WILL be a way for someone to get screwed who does not deserve to get screwed.
Because when you buy something like a playstation, you aren't paying for a creation. You're paying for a functionality, the functionality of whatever it is the program/console does. You probably bought Clarisworks for a functionality [writing documents] as well, but that's different, because there there are other factors _besides_ functionality to consider-- implementation, interface, etc. You can't say the same thing with a PSX. These days, making a console is _selling_ an API-- yeah, yeah, you're selling hardware, sure. But kickass specialised hardware is easy to come by these days. These days the important part is constructing a good, powerful, usable API, making it available and attractive to developers and the public, and recruiting and organising and orchistrating developers and basically just building a _community_. This is where all your expenses go, not into constructing the hardware it runs on. And when you're reverse-engineering a piece of hardware, you just have to worry about the reverse engineering and the piece of hardware; the community is just, well, already there. Sure, Sony makes money off the community [developer liscences == very very lucrative] and not the actual hardware PSX itself, but at some point when the hardware becomes irrelivant because things like Connectix VGS have become so mainstreamed and the costs of emulation can be so easily absorbed by just using a faster or specialised chip.. well, you start to get to the point where you can question the point of creating the API in the first place since it will just be used, and then, well, what are the console developers going to sell?
But see, there isn't anything you can _do_ about it. You can't prevent someone from reimplementing the PSX as a different piece of hardware with the same functionality without simultaneously preventing things like Wine [which i don't think anyone would call immoral unless they sucked] because they do the same thing at the core-- it's just that one is bad because it replaces the PSX hardcore and serves no purpose beyond making money for its creators, and the other , while it is possible to use it _in place of_ windows, does not replace windows itself, and actually creates something by allowing you to do something not possible before, running windows apps run without having to use linux. There's a difference here, between compatibility and replacement, which is subtle, hard to define, and impossible to make into a law. Are you going to start saying people should be not allowed to use certain functionality no matter how they do it? Are you going to start saying nobody should be allowed to make something that does the same thing as something already existing? That's pretty foreign to current patent law, which doesn't allow you to patent an _idea_-- just a _process_. You can't patent the solution, just the steps you took to get there from the problem. You can't stop someone else from taking a different path. And you shouldn't be able to. The reverse engineering exception in the patent laws are there for a reason, to prevent people from using their control over their product to hurt people based on their dependance on what the product does, or doing things like purposefully breeding incompatibility just to hurt a competitor. Note that the patent exception states "for compatibility reasons", which neatly leaves out anything that would actually simply stealing an idea in most cases.
Any action which is designed to make compatibility more difficult will in the end remove choice, and lack of choice very rarely [and NEVER in this context] does _anything_ to consumers other than hurt them.
"Not illegal" != "moral" may be true, but so is ("not moral" != "could be illegalized" || "not moral" != "should be illegalized")
Irritable, left-wing and possibly humorous bumper stickers and t-shirts
Openboy, anyone?
I really dislike the fact that with every new generation of consoles you have to replace all your controllers, and software has to be rewritten from scratch... think about a game header including the minimum requirements (memory, colors, screen size, processor speed...) an open console like that could stay pretty much backwards compatible within the same processor family.
Of course, you'd argue that we already have computers for game development which can do everything a console can; but we all know that's not strictly true, and there's hardly a gameboy-like computer for handheld development. (Color palms? I don't think so).
There's no law against reverse-engineering software and rewriting it, just as there's no law against reverse-engineering hardware and redesigning it. It's simple. Copying hardware is just as illegal as copying software. If I were to take the back of a device, copy the PCB and populate it with the same components, it would be illegal.
It's the difference between COPYING and REVERSE-ENGINEERING.
--
It's a
-- Danny Vermin
About 10 years ago I reverse engineered the file format for Chuck Yeager's Flight Simulator (I think that was the name). It was the one where the player was a test pilot.
My technique was really fun. I would change a parameter in the file, by a little bit at first, and then by a large amount. Then, I would test fly the plane.
It was just like real life test flying! Sometimes changing values would make the plane unstable above 300 knots, and you wouldn't know that until you actually flew the plane. Or, you might get the controls crossed and to roll left you would have to push the stick right. I started making the reverse engineering a regular part of the game. My goal was to figure out what something did, without crashing the plane on any of the test flights. Sometimes that was impossible. I wish I kept the list. Heck, I wish I kept the game.
If tits were wings it'd be flying around.
A company wanted to allow its own movie production management product interoperate with a motion picture project management database. It was like a regular project management tool that had specialization for things of interest to the movie industry, for example there was a specific category for scheduling potted plants to be present on the set as well as manage the expenses involved with these things.
I think it was called Movie Magic Scheduling.
The publishers of movie magic were very protective about thier product and didn't want to cooperate with my clients, so they hired me to reverse engineer the product.
The initial agreement was that I would do this in a week for $1500. It took three weeks and I was working long days, but in the end I was able to take the whole production schedule for an actual full-lenth motion picture and run it through a parser I wrote that dumped it out into an intelligently interpreted text file.
I started by doing this: I created a new, blank document. Then I created a second document with one of the fields containing only the letter "A". Then I made hex dumps and used a comparison tool to compare the hex dumps.
I quickly found that there was lots of unexplained junk in the files so I wrote two tools that I would run before launching Movie Magic to create each document. This was on a Macintosh. One tool would allocate all available memory, set it to zero, then free it and quit. The other tool would create a large file set it to zero and quit.
Note that on Unix systems writing a page full of zeroes to a file is optimized to not write any data at all; however on a properly written system virtual memory pages and disk sectors are zeroed before being allocated to prevent leaking of confidential information.
Then basically I would note what change in the file and make a hypothesis as to what the cause of the change was, and test out my hypothesis by trying to predict what a further change in the file would do.
I wrote two things to document this, one a file format spec in a word processor, second a C program to parse the file. As my knowledge of the structure got more complex, I could implement the knowledge in the file parser and start dumping files through it until an error occurred. Then I could look more closely at what caused the exception.
The key to being able to do this I got from Robert Ward's book Debugging C (I think it is out of print). That is simply to use the scientific method when debugging software (or reverse engineering it).
The scientific method is simple in principle but it takes some discipline and creativity to use it effectively.
- Observe something
- Make a hypothesis as to why that was observed
- Design an experiment that will test that hypothesis. It is important that your experiment tests the hypothesis that you have made, and not something different.
- Carry out the experiment to see if it confirms your hypothesis.
- If it does, you have learned something
- If it doesn't, you have to make a new hypothesis
All the while you keep watching out of the corner of your eye for weird things to happen.One thing that helped a lot with this is that I'd spent a lot of time working with word processor file formats while maintaining a spellchecker, and I'd designed a binary graphics format. So I knew what were the common things to do in a file format, and a lot of things could be guessed directly.
It would have been a lot harder if the format was designed to be hard to reverse engineer, but the information has to go somewhere. Even with encrypted data, with a situation like mine you can do what is called "chosen plaintext cryptanalysis".
For another example of that, during World War II we'd cracked the Japanese cipher, but in addition to the cipher the Japanese used code words. To crack the code words, when we suspected that the Japanese used a certain code (call it "FOO") to refer to midway island, naval intelligence sent a command to Midway via secure channels asking them to report that their water desalinator was broken down - through insecure channels.
After Midway sent the request for a new desalinator, we intercepted a japanese transmission saying that "FOO" had a broken desalinator. Because they had also said they were preparing a major attack on "FOO", we knew to meet them with our aircraft carriers.
The choice of hypothesis is a problem here. Robert Pirsig talks about this in Zen and the Art of Motorcycle Maintainance. For any given observation, there are a large number, possibly infinite number of hypotheses that would fit the observed facts.
Choosing which one is worth taking the trouble to test calls for scientific creativity; ultimately the choice is based on scientific quality, and one mark of a good scientist is that they have a sense of what is right - and the flexibility to recognize that their sense can be wrong.
-- Could you use my software consulting serv