Mark Russinovich on Windows Kernel Security

An anonymous reader writes to mention that in the final part of his three part series, Mark Russinovich wraps up his look at changes made in the Windows Vista Kernel by exploring advancements in reliability, recovery, and security. "Applications written for Windows Vista can, with very little effort, gain automatic error recovery capabilities by using the new transactional support in NTFS and the registry with the Kernel Transaction Manager. When an application wants to make a number of related changes, it can either create a Distributed Transaction Coordinator (DTC) transaction and a KTM transaction handle, or create a KTM handle directly and associate the modifications of the files and registry keys with the transaction. If all the changes succeed, the application commits the transaction and the changes are applied, but at any time up to that point the application can roll back the transaction and the changes are then discarded."

Comments

Message to Microsoft

[bonefry]

Just leave my applications alone !

Re:Not dupe, but almost

[Bonker]

MS made a good move in hiring Russovitch. We can hope that he has more positive influence over kernel changes to XP and Vista than MS has bad influence over things that he does and does not get to say and software (sysinterals) he does or does not get to release.

doesn't belong in the kernel

[nanosquid]

There is little reason to put these kinds of transactional services into the kernel: they don't involve security or user permissions and they must be efficiently implementable in user code anyway (otherwise, most databases wouldn't work well on NT). So, I'd classify this as "kernel bloat".

Re:doesn't belong in the kernel

[Bacon Bits]

Yet PS3 portability was *absolutely vital*.

Re:doesn't belong in the kernel

[Lally Singh]

They also involve atomic I/O to multiple systems simultaneously. Userland can't do this. Databases work on one system, their own data files, and have full control over these files.

Userland apps don't have that kind of control over the registry. Hell they may not be sure to have that kind of control over the files they're manipulating.

Besides, I'd rather have this code once in a DLL than 10 times in 10 different apps. That's real bloat.

Re:doesn't belong in the kernel

[Anonymous Coward]

Because, unlike a DB's transactional system, Vista's is fully pluggable meaning that anything can build volatile transaction management into their application or driver and automagickally gain the benefits of a single logging mechanism and full support for distributed transactions.

And yes, this is a really big deal. Other than maybe OS/400 I know of no system where file system operations can be done atomically with database operations. I can poll data, write a file and update that data all in one atomic operation. Either the data has been exported and marked as such, or not. No in-betweens. No incomplete files. No complete files despite a failure to mark the data in the database. No attempt to manually compensate for errors. It's all there, one operation, atomic and automatically so. This is absolutely massive for my world where financial and business transactions are moved constantly in such a fashion.

Yes, Microsoft does innovate sometimes. This is one of those occasions.

Re:doesn't belong in the kernel

[Anonymous Coward]

Think outside your small box. These features can provide significantly more than current user-mode transaction management that databases (or OLTP monitors) provide today.

For instance, how does a user-mode controlled file-system transactions rollback changes on reboot? Consider what happens when the change involves system files. NTFS handles this cleanly in Vista but any attempt to do this with a user-mode service would fail because the system files would already be in-use by the time the service started. Further, any transactions in-doubt would remain in-doubt until the service started (if you could get that far).

Similarly, what about settings in the registry? Consider if the transaction spanned installing a driver and updating its settings. What happens if the system powers down midway through that update? Without kernel-level transactions that are available at boot-time you cannot easily recover before inconsistent settings are used with the driver.

Beyond the boot-level support; another exciting aspect about the KTM is that this is a transaction-manager that provides transactions that can be shared across multiple processes. Try that with a user-mode transaction manager like XA or DTC and see your per transaction performance. By managing the transaction in the kernel, the 2PC performance is significantly improved.

In the end, NTFS and the registry live in the kernel -- so their tramsaction manager also has to live in the kernel. That is just the way it works.

Another exciting aspect covered is the ability to coordinate DTC transactions with KTM transactions. Meaning, you can coordinate your SQL Server (or Oracle) updates with your file-system updates. This is cool! No longer do you have to worry about finding orphan-files in workflow applications or other applications that manage both files and relational data.

Lastly, they are talking about full ACID properties with NTFS's transactions. Think about it -- you could update every file on your web server, in place, while its wild with activity. All your changes are isolated from the users until you say "commit" in your application.

Re:doesn't belong in the kernel

[EvanED]

C is a language, not a program.

Yes, I realize that.

And the preprocessor directives are part of that language. Or perhaps you missed sections 5.1.1 and 6.10 of the description of that language? (Assuming the numbering stayed the same from draft to final.)

Re:doesn't belong in the kernel

[nigelo]

> Yes, Microsoft does innovate sometimes. This is one of those occasions.

Well, DEC VMS had this capability decades ago, so is it really innovation?

http://h71000.www7.hp.com/commercial/decdtm/index. html

Re:doesn't belong in the kernel

[swillden]

They also involve atomic I/O to multiple systems simultaneously. Userland can't do this.

Of course it can. As long as the OS provides atomic I/O operations on individual files, the logic for implementing two-phase commit is the same in userland or in kernel space. For that matter, atomic I/O can be implemented entirely in userspace, as long as the OS provides a mechanism for userspace to find out when the data has actually been written. It's more convenient to put single-file atomic I/O in the kernel, though (or at least in the file system).

Besides, I'd rather have this code once in a DLL than 10 times in 10 different apps.

That's not an argument for putting it in the kernel.

Re:doesn't belong in the kernel

[dgatwood]

Of course it can be done in user space. If a user space app can't do it, neither can the kernel. And it isn't atomic I/O. There's no such thing as atomic I/O. I/O operations are reordered, split, combined, etc. by everything from the OS to the controller to the hard drive, and for network volumes, it is even worse. There's no practical way to guarantee atomicity, so you have two choices: have filesystems (including remote filesystems) with rollback capabilities (which still don't completely guarantee anything) or design a file structure that achieves the same thing (which still doesn't guarantee anything). The former is nice for a lot of reasons (e.g. so that every developer doesn't have to reinvent the wheel), but isn't essential by any means. It also would greatly increase the complexity of the VFS layer and filesystems written for it, so if that is the only purpose for doing transactions, it makes a LOT more sense to implement them in a user space library instead of in the kernel.

If your sole purpose is to be able to do multi-file rollbacks, user-space transactional support is as easy as designing your file format and/or layout around it. There are two easy ways to do this: files with built-in history and using swappable directories.

Files with built-in history:

For the initial modification pass, modify each file by appending what amounts to a diff footer. If an error occurs, you can undo all of the changes by truncating the files prior to the latest diff footer. Once these modifications are complete, you no longer need to worry about rolling anything back (except for cleaning up temp files if something fails in the second pass) because the data is safely on disk. (Note: this does require that the kernel and all devices and/or network disks reliably flush data to disk upon request. Don't get me started on buggy ATA drives.)

In the second (optional) pass, you coalesce the diff into a new copy of the file and swap the compressed version in place of the original file. This is generally an atomic operation in most operating systems. If anything occurs during the second stage, it is a recoverable failure, so there is no need to roll anything back. Heck, Microsoft's file formats pretty much do this anyway. (Notice the 500 megabyte single page MS Word document that occurs when you make lots of changes and always "save" rather than "save as".)

Swappable directories

The easiest way (tm) to handle system configuration files in an atomic fashion is to modify config files in the same way you would perform a firmware update: you have an active configuration directory and an inactive configuration directory. You read the active one, make changes, and write to the inactive one. Then you trip a magic switch (tm) that says that the previously inactive directory is now active, and vice versa. Assuming you don't have out-of-order writes going on (which the kernel can't really guarantee any better than user space, sadly), this is a very easy, effective way to perform an atomic commit. And if you have an "exchange in place" operation in which the data for two files or directories in the same directory are swapped in a single atomic operation, that's a really lightweight way to implement an atomic commit/rollback mechanism without most of the complexity.

Considering how easy this is to deal with in user space, the only legitimate reason I can think of to do it in the kernel is so that you can take it out of application control entirely (e.g. to make it easier to sandbox an untrusted application). Otherwise, it makes a lot more sense to do this in a library. Now if it had snapshotting where you could roll back the entire filesystem to arbitrary points in time, that might be interesting (for different reasons)... but basic transactional support in a filesystem is much less so, IMHO, unless your purpose is to be able to sandbox an application. If so, then all this other stuff basically comes for free. In that context, doing this in the filesystem layer makes sense. However, if that is not their purpose for doing this in Vista, then kernel bloat definitely strikes me as an accurate depiction.

Just my $0.02.

Re:doesn't belong in the kernel

[dgatwood]

IO commands can be reordered as much as you want, but commands can be tagged for flush or immediate execution, and if a device doesn't comply with those, it is either badly writen or it has incorporated enough safeguards so it doesn't matter.

I'm laughing. No, really. A lot of devices lie in one way or another. It's not just a few badly written devices. You'd be shocked. My favorite are the drives that completely lie and say that the cache has been flushed when it really hasn't. Most of those drives, you can issue a second flush request and it will block, but there are some that always instantly return success for a flush request. It's not even all that uncommon, ESPECIALLY with external drives because of bridge firmware bugs.

I've heard some true horror stories about the number of problems companies find when qualifying off-the-shelf ATA drives for server use. If you heard them, you would cry. Truly, all you can assume is that the operating system has made a best attempt at a guarantee. Anything more than that is just kidding yourself... and you can get that same level of guarantee just as easily from user space (at least in UN*X) by executing two sync system calls in a row.

In addition, how do you expect a user mode component to be able to implement transactional services for kernel components?

WHAT!?!?!?!?!?! Why in you-know-where would anything in the OS kernel need filesystem transaction support? Stuff in the kernel shouldn't even be READING files, much less writing them! Yes, I'm serious. The kernel should provide only the most critical, basic system-wide OS services. Anything that requires such heavy lifting is NOT providing such a critical, low-level service, and thus, does NOT belong in the kernel. It's that sort of thinking that has led to such abominations as khttpd....

I'm going to go hide now. I'm very scared....

Re:Not dupe, but almost

[suv4x4]

On topic now, I don't like the way Russinovich is blowing Vista's horn now. I liked him more when he was more critical and analytical on what could be improved, instead of what has already been done.

He's working at Microsoft now, you do realize that "what could be improved" he's actually doing right now, so he can call it "already been done".

You don't really prefer pointless critique with no improvement, do you.

Kernel enhancements? Are you sure?

[Applekid]

According to the bottom of just one function of the KTM reference:

"Requires Ktmw32.dll."

Why would a kernel add-on require a .dll, exactly? Did I miss some Windows fundamental about it's kernel? And if it's not really a result of a kernel enhancement, is this yet another potentially useful technology specificly excluded from earlier versions of Windows entirely for business purposes instead of technological limitations?

Re:Kernel enhancements? Are you sure?

[Cyberax]

Because this DLL is just an interface to kernel features.

Windows NT was initially designed to use single kernel for different subsytems (OS/2 subsystem, POSIX subsystems, etc.). Subsystems are implemented as dynamic modules talking with the kernel through LPC (Local Procedure Call, see http://en.wikipedia.org/wiki/Local_Procedure_Call ). So in this case ktmw32.dll just wraps LPC calls in a nice API. That's actually a rather good design.

Re:Kernel enhancements? Are you sure?

[EvanED]

Windows NT was initially designed to use single kernel for different subsytems (OS/2 subsystem, POSIX subsystems, etc.)

Not just initially designed, it DOES use a single kernel for different subsystems. You can't get the OS/2 one any more, but the POSIX subsystem morphed into (part of) the Services for Unix which has become the Subsystem for Unix-based applications.

On 32-bit Windows, 16-bit Windows applications are handled by the "Windows on Windows" subsystem. On 64-bit Windows, 32-bit Windows applications are also handled by a "Windows on Windows" subsystem, though a different one than WOW16.

What is the registry in Vista?

[mosel-saar-ruwer]

For years, the "Registry" was some weird mish-mash of binary files, many of which represented Jet databases.

Has Jet been completed abandoned in Vista?

If so, did they switch to the slimmed down SQLServer [that was supposed to be part of WinFS]?

Re:What is the registry in Vista?

[Foolhardy]

The registry is a single root hierarchical database with registry hive files mounted at the second level (below \REGISTRY\MACHINE and \REGISTRY\USER for the computer's config and user config, respectively). The registry engine is implemented in kernel mode as an executive subsystem (inside ntoskrnl.exe), where it is known as the Configuration Manager. Registry hives use a transaction journal (like many filesystems do) to avoid corruption during a power failure or crash. Standard system hives are located in %SYSTEMROOT%\System32\Config and include SAM for local user accounts, SECURITY for various secrets held by the computer, SYSTEM for core system configuration early during boot, and SOFTWARE that stores all other config associated with the computer in the registry. Every user profile has its own registry hive for user-specific configuration. Everything above is still the same in Vista as it was in NT 3.1.

There are two database engines that have been known as Microsoft "Jet", known as Jet Red and Jet Blue. Jet Red is also known as the Access database engine. It is a fairly featureful SQL database. Jet Blue is now officially the Extensible Storage Engine (ESE), and has been a system component since Windows 2000, backing WMI data, Active Directory, Exchange, and others. It is an ISAM database and is optimized for large sparse tables and also supports a transaction journal. Both are 100% user-mode and were not a part of the initial release of Windows NT. Microsoft has said that Jet Red is depreciated, and that future versions of the Access database engine will be integrated with Access and not have a public interface. Jet Blue's interface is well documented and will continue to see use for some time to come. Both being user-mode, dependent on Win32 and the wrong type of database (relational vs hierarchical), the Jet engines would not be suitable replacements for the registry.

SQL Server is a high-end SQL database engine. It was rumored that WinFS would use SQL Server Express and that Microsoft eventually plans to move some of the services that use Jet Blue to SQL Server (such as Active Directory). In any case, SQL Server is an even less possible replacement for the registry.

Microsoft has not gotten rid of the Registry in Vista. In fact, the new boot manager uses a registry hive to store boot configuration, replacing the old boot.ini.

Re:Not dupe, but almost

[dedazo]

On topic now, I don't like the way Russinovich is blowing Vista's horn now. I liked him more when he was more critical and analytical on what could be improved, instead of what has already been done.

Fresh blood always brings fresh perspectives. Mark has been on the outside looking in for long enough that he probably has a fairly asymetrical view of the NT kernel to that of the core MS developers, and in software development (especially at that level) that tends to be extremely useful.

Hmmm... Upgrade to Ultimate Vista perhaps?

[pegr]

"If all the changes succeed, the application commits the transaction and the changes are applied, but at any time up to that point the application can roll back the transaction and the changes are then discarded."
 
What, was my credit card declined for my upgrade to Vista Ultimate Edition?

Not exactly "error recovery"

[Ancient_Hacker]

Recovery? The Windows Registry is the exact opposite of recoverable:

• "Regedit" has no Undo command. It applies changes immediately.
• The Registry file is in a proprietary and undocumented format.
• One scrozzled byte in the Registry can make it completely broken and make the system unbootable.

Not exactly most people's idea of robust and recoverable.

Re:Not exactly "error recovery"

[bmajik]

The registry can be backed up and restored, in whole or in subsections.

The format of the registry is largely irrelevant, but it is described to some extent in the "Inside Windows xxx" series of books (which Russinovich co-authored)

Which specific byte would you "scrozzle" in the registry to render a machine unbootable? How and why would you do it?

Extra credit:
Which files under /boot, /etc, /sbin, and so on would you be willing to stake your career on being "safe to corrupt by 1 byte" and still guarantee a bootable system?

There are things to like or dislike about either a registry based approach (opaque data storage with a defined interface) vs a flat text based approach (clear data storage with an undefined interface). I don't think you make a compelling "anti-registry" argument with the points you list, however.

Re:Not exactly "error recovery"

[Doctor Crumb]

You are confusing a windowing system (X11) with an OS (Linux). While you may have to "screw around on the command line" to get X working again, everything else will continue to work just fine (filesystem, webserver, internet, etc), all of which you can use either from a virtual console or a remote connection. If explorer.exe won't start, how exactly do you fix that without sitting down with a recovery CD?

Re:Not exactly "error recovery"

[fabs64]

Difference being you're not supposed to modify things in /boot and /sbin for all your settings, and /etc is text and therefore much harder to screw up. (you could put an EOF as the first byte in the file, but the system will still probably at least give you a "file x is empty" error message).

What you've said is correct, the gp's gripe is really about using a binary configuration file.. a fairly stupid decision but that is only my opinion.

My argument for flat text (or xml or whatever) over binary, is the same as every argument about the two for the past 5 years, binary is too system specific, reading/writing it is almost always more painful (faster though), reverse-engineering the format is always a much bigger job.

The "defined interfaces" argument doesn't really wash, you can define an interface just as easily for a text file as you can for binary, however with text someone may be able to edit it by hand or write their own interface, with binary this is much more difficult.

I think what it really comes down to is this; If you decided to write a new OS today from scratch and wanted to have a central configuration database (a good idea, as shown by /etc), would YOU come up with the windows registry?

Re:Not exactly "error recovery"

[Penguinisto]

What is that magic, dangerous byte? If it ever existed, wouldn't be trivially easy to write a virus and scrozzle it? Where are these viruses?

You can't pump V1@grA spam or DDoS packets out of a dead machine, and malware writers are definitely in it for the cash nowadays. (IIRC, a large percentage of malware specifically hides in the registry...)

/P

Re:Not exactly "error recovery"

[Talchas]

/boot - kernels/initrds I'm not using (which should be tossed out). configs. I don't know what would happen if an insignificant byte in grub.conf changed, but I generally wouldn't bet on being safe. /sbin - oddly enough, lots. fdisk/cfdisk. fsck + friends would probably boot somewhat. e2image, e2label, ctrlaltdel,shutdown,mkfs*,installkernel. /etc - much of it, just to boot. Boot and have everything work, far less of it - but at least I could usually boot and I would usually know what was broken.

But yes, I understand your point. However - a broken byte in an unbacked up (yeah a bad idea) registry that causes complete failure is far far harder to find than one in /etc, as the one in /etc would generally be in a single plain-text file, and you'd often be able to guess which file from error messages. /sbin and /boot are binary files that don't contain custom configuration data - if something breaks, you can easily replace it without thinking hard. If you lost all of /etc or the registry, you would have to reconfigure your system, which may be far more work.

Re:Not exactly "error recovery"

[Beryllium Sphere(tm)]

I've been known to put my /etc into cvs.

Re:Not exactly "error recovery"

[drsmithy]

Difference being you're not supposed to modify things in /boot and /sbin for all your settings, and /etc is text and therefore much harder to screw up. (you could put an EOF as the first byte in the file, but the system will still probably at least give you a "file x is empty" error message).

Users aren't supposed to modify the Registry, either.

What you've said is correct, the gp's gripe is really about using a binary configuration file.. a fairly stupid decision but that is only my opinion.

It was designed in the late '80s (well, arguably the late '70s) when a 20Mhz 386 with 4MB of RAM was "bleeding edge". Text-parsing is expensive.

I think what it really comes down to is this; If you decided to write a new OS today from scratch and wanted to have a central configuration database (a good idea, as shown by /etc), would YOU come up with the windows registry?

Windows NT wasn't designed today, it was designed in the late 80s.

Re:Not exactly "error recovery"

[drsmithy]

However - a broken byte in an unbacked up (yeah a bad idea) registry [...]

The Registry is automatically backed up at the completion of a successful system boot. This has been true since at least Windows 2000, and probably longer.

Re:Not exactly "error recovery"

[bmajik]

Are you seriously asking?

what is the comment convention?
What is the whitespace convention?
Does this version of this flavor use spaces or tabs in /etc/fstab (or is it /etc/vfstab on today's OS?)
What are the file locking semantics?
Which set(s) of files are logically related to the same peice of functionality?
What character encodings are supported?
What _line termination_ is supported?

nfs export your /etc directory r/w some time. Let someone edit your /etc/fstab in TextEdit on a Mac. Make sure they cut-and-paste from a sample they saw on a website, or that they got via an email. Are you willing to bet your uptime on that working correctly? Even if you control the email or webpage they're pasting from.. are you _sure_ it would work? That there'd be no conversion problems?

Have you ever done a *nix upgrade on a machine? What happens to most of the files in /etc? The answer is NOT "your changes are seamlessly rolled forward"

Re:Not exactly "error recovery"

[larry bagina]

I've been known to put my /penis into vagina.

Re:Not exactly "error recovery"

[weicco]

If explorer.exe doesn't start, user gets (at least in XP) blue screen (not that BSOD, but just a blue screen) in front of him. Then user can press CTRL+ALT+DEL, execute Task Manager and use that to start other applications. You can test this by terminating all explorer.exe processes with Task manager. But you must be quick, since XP will try to automatically start shell process if it sees it has been terminated. Btw. there's a registry entry which tells which shell should be started: HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\WinLogon Shell=Explorer.exe

Maybe you are confusing windowing system to NT kernel :)

'Protected Processes' and PC games

[JanusFury]

A fairly common trend these days in PC games (mostly multiplayer ones) is the use of a kernel-mode windows driver (effectively a rootkit in most cases) to 'protect' the game from hacking. Many eastern (korean, taiwanese, etc.) game development companies opt to use this mechanism to secure their games instead of writing secure client and server code - for example, GunBound, Maple Story, Ragnarok Online, Rakion, etc... pretty much any MMO you see an ad for these days that isn't from a US or European studio uses this stuff for security. The basic mechanism it uses is that it hooks all the low level operations you can do on your system (file access, process access, etc.) and prevents you from touching anything related to the game. The end result is that you can't even so much as end-task a misbehaving game 'protected' by this driver.

With the huge amount of popularity this approach seems to have (I personally suspect it's a result of some very, very aggressive marketing on the part of the driver's developers), I wouldn't be suprised to see many games start demanding that users run them on Windows Vista, so that the 'protected process' mechanism can be used to fully 'protect' the games from users' interference. While you'd at least be able to end-task them, I can't say I see this as an improvement. It's saddening that many companies believe the solution to security is a series of hacks, workarounds, and black boxes - the only real solution is careful, methodical design and engineering. It seems very likely to me that within a few years, many PC games will refuse to run on anything except a Vista system with nothing but signed drivers loaded, and that's saddening. I dislike the notion that I am denied even basic rights to investigate what an application is doing on my machine simply for the sake of 'security', when it's trivial to set up a second machine to inspect and modify a game's network packets and cheat all I want.

Re:'Protected Processes' and PC games

[mpapet]

the solution to security is a series of hacks, workarounds, and black boxes

Outside of some very specific gov't spooks contract work, no one is willing to adopt the other because it's more expensive and requires more thought and risk than the average PHB can handle.

Case in point: The number of cashless casino gaming systems that are centrally managed. The average casino manager understands their casino systems are a single, massive, point of failure. Just wait till they go to cashless floors and someone engineers a jackpot for themselves.

Security programming is hard. Really hard. The developers at these companies may _want_ to do the right thing, but they don't because of the complexity and limited resources.

The big-digital-bank-robbery just hasn't happened yet.

Re:'Protected Processes' and PC games

[admactanium]

> Just wait till they go to cashless floors and someone engineers a jackpot for themselves. That would be a neat trick to do with the end-user UI of a slot machine. Physical security is a pretty important first step.

unless i'm misunderstanding you, slot machines already work this way. you can play a slot machine with cash, or you can use a bar-coded receipt from other slot machines. i've sat down and played a few slot machines with no cash and, even after winning some money, stood up and walked away with my winnings without any additional cash. at some point i did have to put cash into the system (the first slot machine) but you can interact with slot machines, execute transactions (plays) and be paid all without cash.

Re:Not dupe, but almost

[EllynGeek]

Sysinternals was one the very few companies with the expertise to dissect and analyze Windowses truthfully. It's no accident that after the Sony rootkit fiasco they were quietly acquired by microsoft. Just one more assimilation.

Re:I've seen that before...

[EvanED]

It seems to be: every database more complex than a flat file processed by a pair of simple perl scripts has support for transactions like this.

Yes, DBs do. But traditionally file systems don't. The only other system that provides this that I know of that isn't a full-fledged database is VMS.

Sysinternals Utilities

[NearlyHeadless]

I just noticed today that Russinovich's utilities are available in a single-file download: http://www.microsoft.com/technet/sysinternals/Util ities/SysinternalsSuite.mspx

This is...

[Organic Brain Damage]

Windows Kernel. This is Windows Kernel on ACID. Any questions?

A file system is still needed

[gr8dude]

How can that be possible? The DBMS itself sits on top of a file system; you can hide this detail from the user using a layer of abstraction, but the file system is still there.