Slashdot Mirror

Windows makes use of many generic drivers. When I plug my Archos jukebox into the USB2 port in my Windows computer, it uses all generic drivers. The USB2 controller in my nForce2 south bridge is seen as a "Standard Enhanced PCI to USB Host Contoller" that uses usbehci.sys, usbhub.sys, usbport.sys, hccoin.dll, usbui.dll. On top of that, the USB root hub uses usbd.sys, usbhub.sys. Then the USB mass storage device uses usbstor.sys. At the end is the volume which uses disk.sys and partmgr.sys. All of those drivers are generic drivers written by Microsoft; there is nothing from nVidia, Archos or Hitatchi (the hardware manufacturers). Below the USB controller, a standard PCI bus from pci.sys, an ACPI bus enumerator from acpi.sys, also all generic MS drivers.

I have 6 devices that don't use all generic drivers: the ATA controller, network card, sound, AGP bridge and video card all from nVidia, and the Realtek NIC. Of those, I could use generic drivers for 3 of them, albeit with some loss of speed or functionaility.

NT has always had a layered architecture of class and port drivers; class drivers, provided my MS, implement the common functions for similar devices while port drivers handle communication with a specific device. A third-party video driver still depends on the generic video port driver. AFAIK, video cards support the VGA standard and that's about it; a standard VGA driver is the only generic video driver.

There are two ways to figure out which drivers go to which devices: the old way of loading possible drivers and asking each one if it sees any devices it supports, and plug and play which uses device IDs and .inf files. inf files match drivers to device IDs. If a new product comes out, even if it is supported by a generic driver, the .inf files won't know about it; some driver disks just provide a new .inf file. Anyways, you are right that Windows may have the binaries to support a new device but doesn't know it. That's why you can force installation of a driver over an unknown device ;)

• Am I condemned to stay with NTFS?

Windows supports FAT32 and ISO9660 out of the box. FAT32 does not provide enough error recovery to be recommendable. People using ISO9660 as a hard-drive file system are crazy masochists -- enough said. There are seemingly abandoned ports of Ext2 and ReiserFS out there. None of them are in any sense stable for production use.

Why aren't there more file systems available on Windows? The first clue is that Windows is not an open-source platform; open-source hackers tend to live on open-source platforms. The people who work on kernel-level development under Windows are likely to be pursuing commercial software from the outset.

Furthermore, Windows kernel development is something of a black art; it is hard enough that you need to have some vested interest in the platform in order to stay; you would want to live and breathe Windows kernel APIs. (APIs, incidentally, that don't seem constructed for use by humans; for example, due to the limited size of the kernel addressing space, there are several different "kinds" of memory you must carefully allocate and manage yourself. Add to this the awkwardness involved in debugging this stuff, the poor kernel-level development tools offered by Microsoft, the limited documentation, the fact that much third-party information is non-gratis, and of course that the kernel sources themselves are closed, and you have one painful hobby.) In short, you would want to become a kernel specialist.

These painfully-accrued skills are worth their weight in gold, and used to leverage careers as highly-paid consultants or highly-paid trainers, or both. And some, of course, are driver writers for hardware companies.

There's a further reason: Linux file system drivers, in my experience, are designed to be, well, Linux file system drivers. Witness the amount of effort taken by IBM and SGI to port their proprietary journaling file systems to Linux -- and this was from one Unix-like kernel to another. Windows' internal file-system driver API is completely different from Linux'. Porting one file system not only requires a lot of knowledge about the different kernel APIs, but also about the file system itself, because most likely the file-system code is not cleanly separated from the kernel-specific code; you can't just sit down and write an adapter layer. (This is actually mostly speculation, but based on casual perusal of some existing driver code.)

There will be viable, open-source file systems on Windows the day somebody takes the time and effort do implement (and maintain) one. As for myself, I bought the book and started; I gave up not because it was technically challenging, but because it was no fun, and there were more interesting knowledge out there that I wanted to store in my brain.

The reason we see so few file system drivers for windows is that it's blody hard :(

Not to mention fscking expensive. The regular DDK is free (for NT/2K) or cost of media (for the XP/2K3 DDKs), but the IFS Kit is $1000 US. And not well documented, from what I understand.

If anyone is going to try to write these for Windows, may I recommmend you check out OSR?

Disclaimer: No relation with OSR except as a satisfied client of their driver class.

If one was to argue that direct function calls were better, then one could use the Microsoft sanctioned kernel mode direct call interfaces. PCMCIA is one of the main contenders for direct call interfaces, it's used in all the sample code for the IRP minor function IRP_MN_QUERY_INTERFACE.

Simplistically, a driver publishes an interface (a GUID) and registers it. Other drivers which need this interface, do a bit of houskeeping, then send the QUERY_INTERFACE IRP to the target driver, getting a structure full of pointers to functions.

So even NT developers thought that direct call functions were a good idea.

--

This was posted by an MS guy to the OSR NTDEV listserv:

The early ACPI machines were mostly laptops. And the laptops of that generation had most of their devices either embedded in the chipset or on the ISA bus. The PCI or AGP buses were used only for video, and to connect the north bridge with the south bridge. (In Intel's chipset terms, the North bridge has all the fast gates of the chipset, including the memory controller, AGP and in that generation, the PCI bus generation logic. The south bridge contains all the slow gates, including the IDE controller, the ISA bridge, all the PC legacy stuff and probably a USB controller. Today, the south bridge probably also has audio and a few other random odds and ends.) Because the laptops of that era had all of their devices on the ISA bus, interrupt sharing worked poorly. If you bought a mid-'90s laptop from IBM or Toshiba, the serial port and possibly IR would be disabled. There would be a utility packaged with the machine that allowed you to turn on your serial or IR, but at the cost of the bi-directional parallel port, or one of the PCMCIA slots, since there just weren't enough IRQs in the machine to guarantee that all of the peripherals worked, especially if you filled both PCMCIA slots with combo cards.

I once debugged a Toshiba 750CDT in a docking station that had two PCMCIA cards plugged into the machine, two PCMCIA cards plugged into the slots in dock, two ISA cards in the dock and an extra IDE device in the dock, too. This meant that the total demand on the machine was 20 IRQs, when only 16 were actually available.

(As an aside, I've been trying to convince Intel to put APIC interrupt controllers, which would allow many more IRQs, in their laptop chipsets since 1997. My predecessor had been trying since '94. They may actually manage it soon.)

Along comes ACPI. When you turn on ACPI in a machine, it suddenly switches all the power management logic in the machine from delivering its interrupts as BIOS-visible, non-vectored System Management Interrupts over to OS-visible, vectored interrupts. And that interrupt is delivered level-triggered, active-low, which means that it can be shared with a PCI interrupt.

Now consider that these early ACPI machines were already over-committed in terms of interrupts. There was no way to make them work with PCI devices spread out on lots of IRQs. So I just made the code collapse all the sharable devices onto the ACPI interrupt, which was fixed in the chipset by Intel at IRQ 9. By doing it this way, I could hide the fact that ACPI had just created a demand for one more IRQ. (If you use a non-Intel chipset that has ACPI coming in on some other IRQ, you'll see all the PCI devices in Win2K go to that IRQ, not 9.)

Further complicating this story was that I was trying to get ACPI machines to work back in 1997, when the people working on Plug and Play in Win2K hadn't yet gotten their stuff going yet. At time, it wasn't possible to move a device from one set of resources to another after it had been started. This meant that any IRQ solution that I came up with had to work from the first try, so it had to be conservative.

The everything-on-IRQ-9 solution worked. It got the machines to run, as long as none of the device drivers mis-handled their ISRs. (Later, this turned out to be a huge debugging problem, since when you chain eight or nine devices, you'll get somebody who fails.) The solution wasn't optimal, but it did work. I meant to go back and change it later, before we shipped Windows 2000.

A couple of years passed. I had been working on multi-processor problems and on other aspects of ACPI. It got close to the time to ship Windows 2000 and somebody brought up the old question of IRQ stacking. I worked up a more-elegant solution, one that spread out interrupts on most machines. By that time, Plug and Play had been mostly completed, and that wasn't a bottleneck any more. But the test team told me that they wouldn't let me put it into the product, since they didn't have time to re-test the thousands of machines that had already been tested with the old algorithm.

At the time, I thought that this was somewhat ridiculous. I thought that my code would work just fine. I thought that their fears were un-justified. But I was overruled, and I just put the code into what became Windows XP, letting Windows 2000 ship with the simple, safe, yet frustrating stacking.

This is a good point in the story to explain that, in ACPI machines, the IRQ steering is accomplished by interpreting BIOS-supplied P-code called ASL. The IRQ routers are completely abstracted by the BIOS. The OS doesn't need to know about the actual hardware. The old IRQ steering code in Win9x, which was dropped into the non-ACPI HAL in Win2K, had to have code specific to each chipset, which meant that it didn't work when new chipsets were shipped. It was also written in a way that it assumed that there were exactly four IRQs coming from PCI. ACPI machines sometimes have many more. (This is the reason that you don't see the IRQ steering tab in ACPI machines. It just wasn't flexible enough and we didn't have time to re-do it.)

What we discovered with Windows XP was that all of those ACPI machines that had been tested with their IRQs stacked on IRQ 9 tended to fail when you spread the IRQs out. A typical example of a failure would work like this: WinXP doesn't need the IRQ for the parallel port unless you're using one of the extended modes. So the parallel driver releases its IRQ until it's needed. The IRQ choosing logic (called an IRQ "arbiter") would move a PCI device onto the parallel IRQ. This action depends on re-programming the chipset so that the parallel port isn't actually triggering the IRQ. This is supposed to happen by interpreting even more BIOS P-code that manipulates the chipset, since there is no standard for parallel port configuration.

If your chipset comes from Intel, this probably works, since the mere act of setting a PCI device to an IRQ also disconnects that IRQ from the ISA bus. But if your chipset comes from VIA or ALi, there is another step involved. The problem is that nearly all of the BIOS P-code out there is copied from old Intel example code. So they are almost all missing the extra step necessary in VIA and ALi machines.

If the BIOS fails to stop the IRQ coming from the parallel port, the machine hangs, since the parallel port, which sends its IRQs active-high, edge-triggered, will ground the interrupt signal in the passive state. And grounding an interrupt which is enabled active-low, level-triggered will cause an endless stream of interrupts. The parallel port is just an example. Pick any device that is in the legacy SuperIO chip and the story repeats itself.

In Windows XP, I made a bunch of changes. In machines without cardbus controllers, (which don't have the IRQ problems created by PCMCIA,) it will try to keep the PCI devices on the IRQs that the BIOS used during boot. If the BIOS didn't set the device up, then any IRQ may be chosen. But if your machine has a VIA chipset, or if it has a BIOS that we know to be broken, then we fall back to the Win2K-style stacking behavior. The unfortunate truth is that you guys on this list mostly build your own machines, rather than buying them from reputable manufacturers, which means that you guys own the machines with broken BIOSes and VIA chipsets. So even with WindowsXP, you'll see the same old stacking behavior.

One notable addendum is that any machine with an APIC interrupt controller, and thus more than 16 IRQs, will spread interrupts out, even in Win2K. In the past, this was mostly limited to SMP machines. But any desktop machine shipping today that gets the Windows logo has to have an APIC. (This was another reason that I hadn't gone back to re-write this code earlier. Intel had promised that all machines would have APICs by 1998. If this had materialized, then none of you would have had any complaints by now.) I'm actually currently working on software for some future NT that will let an administrator configure the
machine in any way he or she desires.

Stop trivializing the problem, Shill-boy. Of course I expect MSFT to have people use the APIs. MSFT programmers have to use them to do things like write the POSIX subsystem, write the login system.

Of course, if what you shill I mean say is true, then MSFT is keeping the native API under wraps because it's so crappy. That's not true, of course: there's some things you can do in the native API that you can't do in Win32. You can't clone a processes address space in Win32, so you can't emulate the Unix fork() system call in Win32. The POSIX subsystem does emulate fork(), so MSFT does use the native API.

But go on, tell me how to emulate fork() using Win32 calls. Tell me how to do things like write my own login service. Tell me how to cancel an outstanding asynchronous I/O request in Win32. Tell me how to do disk defragmentation using Win32 calls. Tell me how to write an IFS using Win32: I want to put my Solaris UFS disks on my W2K box so I can get rid of this expensive Sun hardware.

Really, you should read the URLs I put in my article. You don't have to believe me, you can believe Open Systems Resources, you can believe Mark Rossinovich. Read the references I put in my last article before shilling further, and please, back away from the crack pipe.

The "secret APIs" are not a rumor. Notice the dates on these references, the secret APIs have been in NT all along.

• Using the NT API for file I/O
  
• Inside the Native API
  
• Do you need source? - go down the page about a third of the way: The conclusion was that Vogels's group used source code only as documentation (there is no other documentation for NT), examples, and to understand the behavior of NT. It turned out to be useful for debugging, and it led to the discovery of interesting APIs that are not documented or available in Win32.
  
• Inside Windows NT Disk Defragmenting - MSFT gave one company access to the defragmenting APIs, and never bothered to document them to anyone else.
  

MSFT hasn't hesitated to use the secret APIs either. From the July 10 InternetWeek: Microsoft has historically achieved market dominance by controlling APIs and forcing competitors to write software to Microsoft's APIs, then changing the APIs. "Instead of satisfying their own customers' demand, competitors are busy catching up with Microsoft," said IDC analyst Dan Kusnetzky.

From the October 8, 1998 NY Times: And Microsoft, the people added, did what it has always denied it does -- used access to its technology as a powerful lever in business negotiations, by offering Netscape preferential access to the Windows "application program interfaces," or A.P.I.'s, the links that enable other companies' programs to run smoothly on the Windows operating system. By turning down the deal, Netscape, they say, would not have that preferred access to Microsoft technology -- a threat that Microsoft fiercely denies making.

Think about it - can you, using only Win32, write all of the stuff that MSFT provides with NT/W2k? No. Clearly, MSFT keeps APIs to themselves. MSFT wants to allow itself the latitude to write faster, more functional programs than the ordinary developers can write. MSFT has proven time and time again that it will use secret APIs to its own advantage, or to the advantage of selected partners (Executive Software, for example). This practice is certainly bad for the consumer. Secret APIs raise the cost of entry into the NT system software market, which will keep out competitors, raise prices, and reduce choice.

on a side note, why the hell can't M$ include the drivers and what not needed for maintaining and reading a non FAT/FAT32/NTFS partition.

Microsoft doesn't ship them because Microsoft wants to lock you into its proprietary NTFS. Third parties don't ship them because Microsoft's IFS kit is incomplete and prohibitively expensive; at US$1,000, who can afford to port ext2 or reiserfs to NT? On the other hand, OSR is selling a filesystem development kit, among other toolkits, but I could not find any pricing information (other than "e-mail sales@osr.com", which doesn't help in the fast pace of Slashdot discussion).

on a side note, why the hell can't M$ include the drivers and what not needed for maintaining and reading a non FAT/FAT32/NTFS partition.

Microsoft doesn't ship them because Microsoft wants to lock you into its proprietary NTFS. Third parties don't ship them because Microsoft's IFS kit is incomplete and prohibitively expensive; at US$1,000, who can afford to port ext2 or reiserfs to NT? On the other hand, OSR is selling a filesystem development kit, among other toolkits, but I could not find any pricing information (other than "e-mail sales@osr.com", which doesn't help in the fast pace of Slashdot discussion).