Slashdot Mirror

I can't comment much on the tradeoffs except to say that I think it solves the problem of Cygwin's fork() being terrible. (SUA also provides a route to get multiple files with the same case-folded name but different case-sensitive names, which I don't think you can do with Cygwin since it goes through the Win32 API.)

Yep, fork() on Interix (SUA) works much more efficiently. The NT kernel has supported what's essentially fork() since at least NT 4.0. The problem until Interix - and the reason why Cygwin's fork() sucks - is that the Win32 DLLs don't react well to being fork()ed. kernel32.dll gets confused, and simple things like console output stop working. Interix doesn't use the Win32 API, instead using a custom POSIX API and the NT API directly. The NT API has been updated to work in the event of a fork().

The NT API function NtCreateProcess spawns a new process. The SectionHandle parameter takes a handle to the image section (IE, CreateFileMapping with SEC_IMAGE) representing the EXE you want the new process to run. If you pass NULL for SectionHandle, you will instead be creating a copy of the parent process's address space, the main part of fork().

I'm pretty sure the NTAPI was largely documents in the 3.51 days, with the docs accompanying the 3.51 DDK.

It appears that NtCreatFile wasn't documented until the 2000 DDK. Given that's a rather critical function of an OS API, I'm not sure if you're correct.

It has never really been a secret, though of course people who write Windows drivers for a living benefit from mangement thinking it is. ...

Internally, there's a huge divide at Micrsoft between the apps guys, who use Visual Studio to do their work, and the kernel guys, who use SlickEdit and C and view everyone else as inferior. Not only is the kernel stuff undocumented in Visual Studio, it's entirely the wrong toolkit.

Even if it wasn't a secret, per se, it does sound like efforts were made to obfuscate the information. Considering most Visual Studio developers are of the VB kind, I can understand why documentation would focus on using the Win32 API. Still, it seems a bit extreme to outright exclude such information. But, then, having a whole other toolkit just for kernel stuff seems a bit extreme as well.

The Windows DDK is (or at least was, a few years ago) far closer to the style of work Unix guys are used to.

While that's good to know, why would I ever think to look in the *driver* development kit to develop a console or gui application? Even if you're correct that as early as the 3.51 DDK mostly documented the NTAPI (I'm not entirely sure how much of NTinternals.net really covers [previously] undocumented NTAPI stuff), there was enough obfuscation to make some developers become Window development superstars of a sort because core parts of the OS are unknown to the general development population. Overall, that really doesn't seem a net positive.

I'm pretty sure the NTAPI was largely documents in the 3.51 days, with the docs accompanying the 3.51 DDK.

It appears that NtCreatFile wasn't documented until the 2000 DDK. Given that's a rather critical function of an OS API, I'm not sure if you're correct.

It has never really been a secret, though of course people who write Windows drivers for a living benefit from mangement thinking it is. ...

Internally, there's a huge divide at Micrsoft between the apps guys, who use Visual Studio to do their work, and the kernel guys, who use SlickEdit and C and view everyone else as inferior. Not only is the kernel stuff undocumented in Visual Studio, it's entirely the wrong toolkit.

Even if it wasn't a secret, per se, it does sound like efforts were made to obfuscate the information. Considering most Visual Studio developers are of the VB kind, I can understand why documentation would focus on using the Win32 API. Still, it seems a bit extreme to outright exclude such information. But, then, having a whole other toolkit just for kernel stuff seems a bit extreme as well.

The Windows DDK is (or at least was, a few years ago) far closer to the style of work Unix guys are used to.

While that's good to know, why would I ever think to look in the *driver* development kit to develop a console or gui application? Even if you're correct that as early as the 3.51 DDK mostly documented the NTAPI (I'm not entirely sure how much of NTinternals.net really covers [previously] undocumented NTAPI stuff), there was enough obfuscation to make some developers become Window development superstars of a sort because core parts of the OS are unknown to the general development population. Overall, that really doesn't seem a net positive.

I think you mean "Win32", as that's the library most user-mode systems guys program against.

"Win32" hasn't been the official name for a while, and it's now called the Windows API. (After all, it also works on 64-bit builds of Windows.) See the second sentence on Wikipedia.

I don't know why people keep imagining all these "undocumented" interfaces. Microsoft lost all those lawsuits long ago, and everything is documented (though sometimes crapily, as with anything).

I'm not as convinced I'm right, but I'm still seeing some evidence. For instance, there are several functions on an old Sysinternals page on the Native API that I can't find documentation of (either under the Nt* name, Zw* name, or Ke* name), such as NtCreateSemaphore. undocumented.ntinternals.com has a bunch that I can't find on MSDN. Still, I'm willing to concede I might be wrong on this point, and merely say that until relatively recently (even in terms of the history of NT), it was undocumented.

The NT kernel has a root namespace for everything in the system (from local filesystems to network drives to sockets to synchronization objects like mutexes), and in fact treats everything as a file (just like Unix) underneath.

Using the Native (NT Executive) API you can read or set the ACL on any object in the namespace, assuming you have the appropriate user rights and you own the object (or the ACL allows you to modify the permissions). NT kernel objects can also be case-sensitive (though that can confuse some Win32 programs). Often, you can delete, move, etc files that are locked by the Win32 subsystem, which can be useful in certain situations (though in Vista they made the IO system capable of cancelling outstanding IOs on its own so the zombie process bug that ends up locking files doesn't happen anymore. Its unfortunate Vista is so DRM-laden, or I'd try upgrading.)

The APIs are NtQuerySecurityObject and NtSetSecurityObject and I believe the devices are in \Device\Tcp, \Device\Ip, \Device\RawIp, \Device\Udp, etc. Check out http://undocumented.ntinternals.net/ for more details on what is in the native API (ntdll). This API provides everything necessary to implement a full POSIX layer, which is exactly what Services for Unix does, installing itself as a new runtime subsystem right next to the Win32 subsystem. (With Server 2003 R2 SP2 they shipped it as an available component as part of the install; I've even got setuid support and GCC installed as part of the package.)

Even Microsoft documented at least one! Here's a comment in the Win2K source code that got leaked onto the net a few years ago:

private\mvdm\wow32\wcntl32.c: // These undocumented messages are used by Excel 5.0

You can see "documentation" for many of them on the Sysinternals site.

Not that they're likely to do it, but what if one of these had a worm come along exploiting it? The quick and obvious fix would be to simply remove it.

Is this site a lie?

http://undocumented.ntinternals.net/

private\mvdm\wow32\wcntl32.c: // These undocumented messages are used by Excel 5.0

While I agree with you that the current Office developers are simply good and talented coders and aren't simply leeching off of some undocumented API for their spiffy graphics, it's long been alleged that Microsoft has used undocumented APIs for Office. While I can't find the cite, I believe this was a key part of the anti-trust lawsuit.

You can see "documentation" for many of them on the Sysinternals site. One thing I'd warn against is actually using these calls in production code. Undocumented means unsupported -- MS could decide tomorrow to yank these in their next XP hotfix, and your code would be left hanging high'n'dry. Not that they're likely to do it, but what if one of these had a worm come along exploiting it? The quick and obvious fix would be to simply remove it.

Same with the NT kernel. For example, the dlls of a process are stored using exactly the method described in this patent (notice the three List_entrys).

Process memory allocation != heap allocation. The smallest amount of memory the kernel can allocate to a process is one page, which is 4096 bytes on the i386 architecture. Linux operates exactly the same way. To request more memory be committed and mapped in a process, the NtAllocateVirtualMemory syscall is used. 4096 bytes is much too large for most general purpose allocations, so a heap structure is used to further divide up the memory size. The standard heap code (used by the Win32 heap functions and Microsoft's C runtime) resides in ntdll.dll, and are executed entirely in user mode. When the heap code needs more pages to expand the heap, it surely does call NtAllocateVirtualMemory, but most allocations are done on a much smaller scale.

XP introduced (and had backported to 2000) the Low Fragmentation Heap option that uses presized buckets to reduce long term fragmentation. That's the only thing I can think of that they could be talking about. It's part of the standard heap code in ntdll running in user mode.

Microsoft certainly didn't rewrite "large parts of the kernel." If they were going to do something on that scale, you'd think they would ditch (or at least marginalize) the awful Win32 subsystem, but they aren't. This is one of the worst attempted technical articles I've seen in a while. Besides, the kernel never was the weak point in Windows NT's (yes, Vista is still NT) security model, or as an OS in general.

Process memory allocation != heap allocation. The smallest amount of memory the kernel can allocate to a process is one page, which is 4096 bytes on the i386 architecture. Linux operates exactly the same way. To request more memory be committed and mapped in a process, the NtAllocateVirtualMemory syscall is used. 4096 bytes is much too large for most general purpose allocations, so a heap structure is used to further divide up the memory size. The standard heap code (used by the Win32 heap functions and Microsoft's C runtime) resides in ntdll.dll, and are executed entirely in user mode. When the heap code needs more pages to expand the heap, it surely does call NtAllocateVirtualMemory, but most allocations are done on a much smaller scale.

XP introduced (and had backported to 2000) the Low Fragmentation Heap option that uses presized buckets to reduce long term fragmentation. That's the only thing I can think of that they could be talking about. It's part of the standard heap code in ntdll running in user mode.

Microsoft certainly didn't rewrite "large parts of the kernel." If they were going to do something on that scale, you'd think they would ditch (or at least marginalize) the awful Win32 subsystem, but they aren't. This is one of the worst attempted technical articles I've seen in a while. Besides, the kernel never was the weak point in Windows NT's (yes, Vista is still NT) security model, or as an OS in general.

CreateProcess is not a syscall; it's a Win32 function exported by a Win32 client library, kernel32.dll. The internal API handled by the kernel is the native API. All of the syscalls start with Nt* and are exported by ntdll. The related syscall is NtCreateProcess, a function that creates a process object, loads ntdll.dll, maps an image section (for the program binary) and nothing else. It does not have a STARTUPINFO parameter.

CreateProcess from Win32 does call NtCreateProcess, but it also does a lot of other things, like create a startup thread and report the new process to the Win32 subsystem server (csrss), where the GUI parameters are used. The GUI is a core part of Win32, but Win32 is not a core part of the kernel. Almost all Windows software depends on Win32 (and so the GUI), but the kernel is happy to without win32: see the recovery console or first stage setup program. Full kernel, no Win32.

Before NT4, the Win32 subsystem was implemented entirely in user mode (mainly winsrv.dll hosted by csrss.exe for the server, kernel32.dll, user32.dll, gdi32.dll, advapi32.dll for client libraries used in programs that depend on Win32). Since NT4, much of winsrv's code got moved into kernel mode in win32k.sys. The kernel mode part of Win32 is all contained in win32k.sys; it's not part of the kernel, it just runs in kernel mode.

I tried getting a grasp on those 1024-characters-long API call names from MicroSoft, and I was really glad that *n?xes provided me simple _system calls_ (which MS does _not_ provide, since the developers can use only the APIs).

I agree that the names can get a little long, but at least they are descriptive. If you had never heard of freopen before, could you tell what it does without looking it up (from just the name)? How about SetStdHandle?
No, the system call interface isn't documented by MS. Yes, that sucks. There are alternative sources that do document the native API (the system call interface).

Anyway, I find MS libraries too chaotic. But when we speak about kernels, the win32 kernel and the event-driven idea behind everything windows-like aren't really good, nor new. Why, else, people would use *nix kernels for critical applications, such as servers? The only thing interesting about the win32 approach is the HAL. Everything else is just (bad, imho) habit: you think that using already prebuilt APIs would speed your work, but for me it is only laziness.

Chaotic? The win32 api maintains source-level compatibility (that's all POSIX offers) with win16 since Windows 1.0, and binary compatibility with win16 binaries by use of a virtual machine. Could I hope to run a binary created in a Linux 1.0 era environment on the newest stable environment? How about a kernel module? You can load drivers from NT3.1 (first release of NT) in WS2003. Windows developers have gone to great lengths to maintain stability and compatibility in their APIs. You are lucky if the next release of the GNU C runtime doesn't have breaking changes requiring code modifications.
The reason behind this is that customers get really mad and tend to not buy upgrades when the next version of your product breaks their existing apps; custom business apps that have long lost the source code and developer support.

What do you have against the concept of asyncronous IO? You wouldn't want all the hardware busses in your computer to behave syncronously.
I see it as a question of balance between a generic solution and a specific solution. When UNIX uses a filesystem to access almost everything and treats as many things as files, that is a generic solution. Generic solutions are more consistent and usually simpler, but don't handle variations and unique situations well.
Anyways, if an application would be best served by handling something in a special way and the operating system is willing to participate, where is the harm in letting them do it? Asyncronous IO doesn't work for everything but when it does, syncronous IO really looks ugly.

Keeping some APIs static for 10 yrs means also keeping backwards compatibility with some old or "slow" functions. And more code you write more bugs you can potentially generate. So I don't really see the point of having a lot of APIs you'll never use instead of a hundred syscalls and then you use another library (for example, STL). If you want a thousand functions, pick up a library like, who knows, GTK+. It's in userland and fully extendible. Don't you like it? Change it.

Or you could keep the old APIs and get people to use new ones that work better. I'm confused; you said Microsoft APIs were too chaotic, but now they're static..?
In Windows, the system call interface is the native API, exported in ntdll.dll (look for the Nt* functions). Win32 sits on top of that, as a big runtime library that provides compatibility, graphics and user interface controls. Many win32 functions are thin wrappers to native functions: CreateThread (from win32) just calls NtCreateThread (a syscall). XP sp1 has 285 syscalls. Linux 2.6.7 appears to have 268.

Change it, as in create (yet another, I'm sure) fork of whatever project,

First, the win32 functions look a lot longer because you made a new line for each parameter, and the standard headers use typedefs in all caps. Second, use MapViewOfFileEx if you want to specify a start address.
mmap has 6 parameters. CreateFileMapping has 7 parameters, and MapViewOfFileEx has 5. Minus the extra parameters each for offset in win32(as dumb as it is) posix takes 6 and win32 takes 10 parameters.
mmap:fd = CreateFileMapping:hFile
mmap:start = MapViewOfFileEx:lpBaseAddress
mmap:prot = CreateFileMapping:flProtect
mmap:length = MapViewOfFileEx:dwNumberOfBytesToMap
mmap:offset = MapViewOfFileEx:FileOffsetHigh and Low, as two seperate values to create a 64 bit offset
mmap:return value = MapViewOfFile:return value
CreateFileMapping:dwMaximumSizeHigh and Low have no equvalent in mmap; they can be used to expand the size of the file
CreateFileMapping:lpAttributes has no equivalent in mmap; CreateFileMapping creates a section object in the kernel: it has its own ACL. mmap always uses the file's permissions. This also controls if child processes inherit the section object.
CreateFileMapping:lpName(optional) has no exact equivalent in mmap since the original file(if there was one) had a name. This section of shared memory does not. This specifies a name for the section that will be placed in the object manager namespace, if you want one.

In Windows shared and mapped memory is accomplished through a section object. UNIX considers shared memory another type of file. Since in Windows a section != a file, an extra step is needed to create a section from a file. Both can create a memory section that is not connected to a specific file.
So UNIX is a little more convient since it can do two steps in one. If that was a common task in a program you could easily write a function to combine the two steps into one.
As for the asthetic state of win32, I think that the all caps typedefs and variable type prefixes (dw, h, lp) are hideous; but they don't have much of an impact on the api's actual usability.

Overall, this isn't a very good example of POSIX's superiority; win32 provides more functionaility in this case and the extra parameters can be out of the way using one extra function.

Sources:
mmap
CreateFileMapping
MapViewOfFileEx
MapViewOfFile
See also:
NtCreateSection
NtMapViewOfSection

First, the win32 functions look a lot longer because you made a new line for each parameter, and the standard headers use typedefs in all caps. Second, use MapViewOfFileEx if you want to specify a start address.
mmap has 6 parameters. CreateFileMapping has 7 parameters, and MapViewOfFileEx has 5. Minus the extra parameters each for offset in win32(as dumb as it is) posix takes 6 and win32 takes 10 parameters.
mmap:fd = CreateFileMapping:hFile
mmap:start = MapViewOfFileEx:lpBaseAddress
mmap:prot = CreateFileMapping:flProtect
mmap:length = MapViewOfFileEx:dwNumberOfBytesToMap
mmap:offset = MapViewOfFileEx:FileOffsetHigh and Low, as two seperate values to create a 64 bit offset
mmap:return value = MapViewOfFile:return value
CreateFileMapping:dwMaximumSizeHigh and Low have no equvalent in mmap; they can be used to expand the size of the file
CreateFileMapping:lpAttributes has no equivalent in mmap; CreateFileMapping creates a section object in the kernel: it has its own ACL. mmap always uses the file's permissions. This also controls if child processes inherit the section object.
CreateFileMapping:lpName(optional) has no exact equivalent in mmap since the original file(if there was one) had a name. This section of shared memory does not. This specifies a name for the section that will be placed in the object manager namespace, if you want one.

In Windows shared and mapped memory is accomplished through a section object. UNIX considers shared memory another type of file. Since in Windows a section != a file, an extra step is needed to create a section from a file. Both can create a memory section that is not connected to a specific file.
So UNIX is a little more convient since it can do two steps in one. If that was a common task in a program you could easily write a function to combine the two steps into one.
As for the asthetic state of win32, I think that the all caps typedefs and variable type prefixes (dw, h, lp) are hideous; but they don't have much of an impact on the api's actual usability.

Overall, this isn't a very good example of POSIX's superiority; win32 provides more functionaility in this case and the extra parameters can be out of the way using one extra function.

Sources:
mmap
CreateFileMapping
MapViewOfFileEx
MapViewOfFile
See also:
NtCreateSection
NtMapViewOfSection