Slashdot Mirror
Update to The Magic Cauldron
Eric S. Raymond wrote
in to tell us that he has updated The Magic Cauldron
(his essay on economics and Open Source) to contain
an appendix on common arguments for keeping device drivers
closed (Pay attention Creative Labs!) He also says "The argument turns on the fact that drivers are small pieces of code, easy
to disassemble if need be. This argument would be considerably strengthened
if I could point readers at a working set of tools for disassembling Windows
drivers into recompileable source (or even just assembly) code. I would
appreciate pointers to any such tools."
Even without using something like this, disassembly of programs cannot be 100% correct. There is a lot of information needed to reconstruct a program that is thrown away by the compiler. For example you can't determine where a jump table (switch statement) or function pointer (virtual function) will lead to without actually executing the program and then it's not possible to cover all of the cases that could occur. IDA does a pretty good job of guessing though. Also IDA has hacks if it is looking at C code like looking for procedure frame headers (push ebp; mov ebp, esp) and termination code (pop ebp; ret). My solution to correct disassembly is to use emulation. This insures you never have an incorrect interpretation, but you will miss code that is never executed, and this can be important for understanding how errors and exceptions are handled.
Just because you can dissemble something doesn't mean you can understand it. It requires a huge amount of time to try to comprehend uses for unnamed memory locations and layout of data structures. In the case of a driver this is frustrated by the fact you are dealing with a lot of hidden code and data (the logic on the hardware). Not all drivers are "small pieces of code." Having looked at several vendor's Direct3d driver source code I can tell you each was extremely large and complex. Writing the driver with full knowledge of everything is hard enough, but disassembling one and understanding it would require super human willpower. There are all sort of commands that will do different things depending on that state of the card, so you have to understand what previously executed code was important to getting the card into that state.
-- Virtual Windows Project
First off, I want to make it clear that I'm not arguing against open source drivers, I'm arguing that a) open source drivers are necessary but insufficient to getting hardware running on different os's. and b) there are costs in resources and $$$ for a company in supporting an open process for drivers.
I was thinking in the context of drivers for things like new generation 3d boards, which are not reverse engineer-able in any useful sense.
You don't really "port" a 3d video driver from Windows to Linux, you write a brand new one, basically from scratch. Source code would be nice, but you still need the docs plus actual human beings to ask questions of. (Yeah, glide helps)
3Dfx has done a great job of this, but it took a some amount of liason work (==$$$) on 3dfx's part to make it happen. Every hour the 3dfx engineers spend talking to the linux driver guys is an hour a) they get paid and b) they aren't working on something else. It was probably worth it for them (hey, I bought one) but it was an "allocation of resources" problem just like in sim city.
Another reasons open sourcing drivers might cost a company : The drivers for 3d boards are areas of competition (the drivers can be as important as the chip design for producing good benchmark results)
Again, I _like_ the idea of an open driver development process. I bought and very happily use a 3dfx Voodoo3/2000 because of it. I hear nvidia is doing a great job as well.
-cks
I'm not exactly in the device driver writing business but I'd imagine that maintenence of drivers for old hardware cost a fair bit time and money. Once you make the device driver opensource and get it past the 1.0 stage, you can go on to bigger better things.
Companies like Matrox and Adaptec have, as far as I can remember, been very good about supporting even their oldest devices with drivers and jumper settings and such. As time goes on they accumulate more and more stuff that their customers expect them to support just as well as they ever did. Opensource fixes their growing legacy support problem. They don't have to worry about weather MS will include their driver in the next OS release, it will get ported to the next version of Linux almost automatically. The only way tht would not happen is if there doesn't exist anyone who is using that card anymore. If that happens, who cares if the driver dies?
I believe Microsoft's own development tools include a disassembler. I couple guys I went to school with used to use VC++ 5.0 to crack the copy protection on games.
Scott Banwart
---
Better to stay silent, and let people think
you're an idiot than to open your mouth and
This is a decent starting point for disassembly information. Lot's of texts on reverse engineering, with pointers to tools, etc...
mammon_
I found the situation of disassemblers for the X86 relatively bleak, since there is only one tool that suits my needs (offline disassembly). Every development tool contains a debugger that also does some disassembly of the current instruction stream, but they all lack the ability to define data structures.
:) ). IDA supports a whole slew of input files (DLL, VxD, NLM, COM ...) for a lot of processors (Z80, x86, Java VM, ...). It has working versions for OS/2, DOS, Win32 ...
I did find "the one true" (at least to me) disassembly tool, IDA by http://www.datarescue.com. It is an interactive disassembler that does background disassembly and lets the user identify additional code and data sections (and correct the mistakes
There are only two things about it - it is payware and no source code is available. And it is virtually impossible to find warezed copies on the net. I paid for my copy and I'm still happy with IDA (in fact, I was happy with the trial version and found it well worth the money to upgrade). The copy protection seems also to be relatively hard, at least from what I found (or better, didn't find) by looking at it.
-max
Premier argument to install Linux at the workplace - I get paid while waiting for fsck to scan the partitions.
The second story was with a 68000 monitor and debubber that I had to write a disassembler for in order to move it to a 68010 SBC. The disassembler was hand-made by me.
The third involved the original 8052 BASIC52 that Intel had produced. There were some bugs and lacking features in it, so I wrote a small assembly routine to dump the contents of the chip out its built-in serial port. The 8052 output was captured on an IBM PC, and I wrote a disassembler in MSC 4 to put it back into 8052 assembly language. I then fixed the bugs and added the features I needed, after which I programmed it back into an 8752 for further development. I sent the code back to Ciarcia, and I even got a little note back.
But the biggest tool I used was V Communications Sourcer. I got started using it to reverse IBM PC ROMS to hack the drive tables, and then started to add little features and fix little bugs. I used it commercially to reverse engineer the IBM RTIC communications card DOS drivers. I did that so we could then port the card to Novell Netware 2.15 as a VAD (IBM had no plans at the time to port it, and we needed it to provide a Bouroughs data link between a Netware server and a Bouroughs mainframe). Sourcer was, and still is, the cleanest disassembler on the market, capable of giving you back highly commented source code from binary files. I haven't updated my license lately since I prefer open systems to Windows, but if I had to do that again I would go back to Sourcer.
The point is that reverse engineering is equal parts art and science, and if you need it bad enough and know assembly well enough, you'll find and/or build the right tools for the job. ESR needs to calm down a bit.
So if a competitor can trivially disassemble drivers to reverse-engineer them (which is not the case anyhow), what is the added value in doing their reverse-engineering work for them and releasingly nicely formatted source code with symbols and #defines and everything intact, with no optimizer obfuscation, and possibly even comments? Sounds like a gift to the competition if I ever heard one.
My argument has been that it's unlikely anyone will reverse engineer your hardware from your software, but perhaps this is not the case?
I've finally had it: until slashdot gets article moderation, I am not coming back.
ESR seems to neglect the classic argument for open-sourcing the drivers---the users will make them better. From personal experience, I have noticed that device drivers (especially new ones for new products like video cards released by the company that made the product) tend to be buggy as hell and can cause frequent crashing of a Windows machine. (Alex St. John has ranted about this many times in his "Maximum PC" opinion pieces.) By opening the driver source to your users, a company could get fixes to these problems much more quickly than keeping them closed. This seems to me to be an even more convincing argument than the "some-kid-will-disassemble-them-eventually-anyway" one.
"there once was a big guy named lou
It seems that ESR's recent writing are both volumous and comprehensive, and If he is taking suggestions for after-the-fact mods, perhaps we are beginning to see an era in which we "open source" our documentation, theory, and text-books. Yeah, I know, books are open, you can read them. But I am talking about development projects, where the original author throws up a basic outline of the work, with a few sections filled in, and then manages the submissions. So I want to propose the _YALS_ project, standing for "Yet Another Love Story", we need some web space, and a few core developers, and we'll be ready to go. The basic outline will be:
1) Character_One Meets Character_Two
2) Character_One Loses Character_Two
3) Character_One Gets Character_Two We should try to flesh the plot out by August 20th, Aiming for a Version 0.1 publication by January 1st. Check The site at http://taz.eng.ua.edu:1138/crutcher/yals
-Crutcher
-- Crutcher --
#include <disclaimer.h>
Forget the open source movement for a moment. It seems that the lifetime for computer hardware is incredibly short. By keeping the source to driver software closed manufactures accomplish;
t ml
1) Decrease the life of the product, shortening support obligations.
2) Hide any shortcomings to their hardware product, saving on warranty obligations.
3) Forcing the consumer to upgrade by preventing product 'enhancments', or fixing 'problems'
4) Reveal design shortcomings to the competition.
Some wise man said that a there are two reasons someone does something. A good reason, and the _real_ reason. The arguement that it protects proprietarty technology is in most instances completely false, as many chip makers release 'reference' designs to many hardware manufactures, who in turn make only small changes to the basic design. And any valid 'proprietary' enhancements there may be are quickly undone when the next competing product hits the market. So companies are left with the real reason - less hassle and more money.
I do not think that is enough that you argue politely the benifits of open source. Consumers are going have to get tired of being on the endless upgrade mill of expensive hardware and the expensive bloated code to use it. How much power do you need to write a letter or balance a checkbook?
Maybe then we would see real innovation in the computer industry instead of creeping featurism.
The only way that you are going to see open source from a manufacture is if it is legislated by congress answering the demand of angry consumers who catch on. IBM's Aptiva MWAVE is a classic example of the above. Cut and paste the link below into your browser.
http://members.aol.com/mbs1058/mwave/classact.h
However, even this is not a real excuse - if you have a situation like this, publish as much as you can, and guard your most treasured secrets if you want to be that way.
For example, Matrox released the specs for their graphics G200 cards, except their triangle setup engine (for whatever reason) and they've also released most of the specs for their next gen G400, even before it started shipping!
http://fravia.org (which seems to be down half the time)
is all about reverse engineering (cracking). It isnt just a simple how to crack (software crack, not bad "hacker") site, he really is interested in reverse engineering. There are even pages about Linux cracking. If his site is down just search for +HCU or +ORC along with +fravia. When I was into that type of stuff, before I got into linux, I used w32dsm, which there are shareware version s of it, but just as I stopped doing that IDA became really big. IDA seems to be better, but W32dsm worked well. And for a windows debugger, you can't beat SoftICE, by Numega. No other debugger for windows even comes close.
Precisely Bruce, hardware vendors that don't divulge the source to device drivers these days are really only shooting themselves in the foot!
An awful lot of my friends and acquaintances now list Linux compatibility as not just a contributing factor, but as a deciding one in all of their hardware purchases. This big change, only in recent years, affirms to me your comment that we are indeed a market force now, a fast growing force, and one to be reckoned with.
fravia (east coast)
decompilation page
sandman
greythorne
The more interesting question is, given that most of these sites have been around so long, why dont we see more reverse-engineering of software going on? i think the availability of all this information (especially on fravia) weakens esr's argument significantly. regardless of what he says, reverse-engineering is really, really difficult, even for small pieces of code. i dont think he'll convince very many people based on that argument.
As for the appropriate tools, a while ago i found copies of wdasm and softice using ftpsearch (remember, one version of softice was a fully operational time-demo which could be cracked by itself). much of the other stuff you might need that ive seen are freeware or shareware.
unc_
I gathered that he was saying do this or your potential customers will call you a moron.
As one of those people who influences buying decisions, I can confirm that position. Venders who won't release spec ARE clueless, and I will strongly recommend against them if any reasonable alternative is available.
Chip vendors don't like this because they don't want their competitors to use their own documentation to produce drop-in replacements for their components. They use trade-secret because it's the cheapest form of protection. However, if they have to open their interfaces to get their chips designed into Linux systems, they will publish their documents and use patents and copyright as protection against their hardware competitors.
Thanks
Bruce
Bruce Perens.
exactly. i do not want to give my money to a company that does not open
their hardware specs and i would rather give it to a company that has open
source drivers. what would be useful in this context would be a web based
hardware database, where find quick info about how open a specific piece of
hardware is before buying one.
like there could be 5 categories:
1 proprietary hardware, company does not give out spec.
2 company allowed people to write binary only drivers under NDA but for free.
3 company releases binary only drivers
4 company published detailed documentation
5 company wrote open source driver or funded/helped in the development
i would only buy hardware in category 4 or 5 but often it is not easy to
find out if the person who wrote drivers got support from the company..
sure, a grep through the linux kernel helps sometimes but that is not
something the marketing departments will consider a threat to the image of
their product.
having such a database where one could directly compare the openness of
various vendors would put some market pressure on those companies..
mond.
Thanks
Bruce
Bruce Perens.