Actually, it seems to be selective mirroring at the file level, so it's like RAID1 for the files and directories that you ask to have mirrored. I don't know how may pieces of WS2003 are included, but you may be able to build a software RAID5 dynamic disk set manually.
The target market is for people who don't install the OS themselves-- they buy a computer from an OEM with the hardware and OS preinstalled. The OEM will make sure that the drives are hot-swappable and such. Adding new drives and merging them into a RAID is done automatically and silently upon plugging the drive in. They'll be happy to make sure you buy only their specially branded "WHS compatible" hot swappable drives for that purpose. I would be surprised if a retail version is even available on shelves.
Single instance storage has existed since at least Windows 2000. The "recover older versions" thing is connected to the "older versions" shell extension, also connected to volume shadow copies, giving you the option to restore from both known backups on file and local VSS copies of a file. Automated remote backups aren't hard to implement with file sharing, and I would think that WHS comes with some simple UI to set it up, with which workstations to backup and corresponding credentials on each.
Remote control uses RDP. I think it will be configured to accept only connections on the local network, and will certainly require some kind of authentication.
It's because Notepad supports both UCS-2 and ASCII encodings, but without any metadata or conventional file headers to tell it which format is actually being used. Unless you tell it otherwise, Notepad uses the Win32 function IsTextUnicode() to try and guess the format of the text. IsTextUnicode() offers only a best-effort attempt to detect what format a blob of text is in. See the description of IsTextUnicode() for the various fuzzy detection methods it offers. It doesn't always work, especially if the sample size is small.
Notepad tries to "just work" with both formats of text based only on the contents of the text. There is no strict solution, given the lack of metadata. Most text editors require you to explicitly state the format of text (or always assume some default), leaving no possibility of accidental failure.
There is a certain class of security vulnerability where malformed data passed to a library in the same process can cause code execution. From the library's point of view, since the library is in the same process as the caller, they're both at the same trust level, so calling a function does not cross a security boundary and no secure validity checking need be performed. The worst that could happen is that an app causes a library to execute code in its own process, a non-issue. The only parties involved are the application and the user library. This was the picture in 1993 when the first version of Windows NT (3.1) was released. It was largely still the case in 1995 with the release of Windows 95. This is the era where this and other vulns (like the GDI metafile escape one) are from.
Problems start when the app passes along data from some outside untrusted source without understanding its content or validating it, like when a web browser passes an.ANI to user32.dll. Back when user32.dll was written for NT 3.1, the devs never conceived of an app implicitly loading a malicious.ANI (without validating it) from a third party. At the same time, the app would much rather treat those things as opaque blobs to pass on to libraries, implicitly expecting them to do the validation too. The libraries see the security boundary as being on the other side of the application, expecting the application to validate any data before processing. Both expect the other to validate data, but in some cases (like this one), neither do.
Right now, the trust assumption of many user libraries hasn't been fixed because there is a lot of code in that position and it would be a lot of work to go through it all. Managers hate fixing code issues like that because it takes a lot of time and money but doesn't result in anything tangible like pretty features. Applications already suffer enough code bloat without having to implement validation for all the data they come in contact with that gets passed right to support libraries-- managers don't want to spend time and money on validating things that should be someone else's responsibility. Microsoft has had this class of vulnerability on low priority for a long time, and it's been the source of A LOT of issues.
I'm not excusing Microsoft's behavior, just trying to explain it somewhat. Someone sure dropped the ball in not finding finding problems similar to the 2005 issue though.
This by reason of Microsoft's own self-stated bug hunting and code modification procedures.
Microsoft is a big company. Not every department is following the Security Development Lifecycle, as much as marketing may like to imply it. The main two examples that do are SQL Server 2005 and IIS6, both of which are doing very well. I haven't heard Microsoft say that all of Windows or the Win32-GUI core team were using SDL.
On top of all of that, this is yet another (of about three instances I have found so far), where it's clear that Vista is not "all new code" as MS likes to maintain it is.
Vista contains copious amounts of new code, but very little of it replaced old code. The sound system (the mixer mainly) was largely rewritten, the backup program got replaced with a POS from scratch, the logon GUI arch (i.e. msgina.dll replacements) got replaced... and I can't think of anything else that is new code to replace old code. I'd say that at least 75% of the Windows NT3.1 code base is still present in Vista.
When you ask the NT kernel to open any object, including a file, the default is to be case-sensitive with the name lookup. There is a flag you can specify (OBJ_CASE_INSENSITIVE) to make it insensitive. NTFS supports it both ways. Win32 always specifies this flag, unless you pass FILE_FLAG_POSIX_SEMANTICS to CreateFile. Most Win32 programs don't do that, so end up asking the filesystem for case-insensitive handling. Since the topic was how case-insensitivity was hurting Java, the question is why isn't the Java runtime passing that flag to get the case-sensitivity it wants?
SFU doesn't use Win32, and never uses OBJ_CASE_INSENSITIVE AFAIK. It's true that a lot of Win32 apps will choke at the sight of two filenames differing only in case, but then there are a lot of things that most of them don't do properly, like LUA. I agree that Win32 is an ugly mess, that plenty of software built on it is as bad, and that it should all go away... but one of Microsoft's biggest claims to fame is that mass of software built on that interface. They're not about to give it up. Just don't confuse Win32 with the kernel or NTFS.
Yep, that's one bang up set of tools. Not even SSH. Telnet. Yep, keeping up with the times. Real UNIX integration there.
It's too bad Microsoft doesn't include more standard tools, but that's what interopsystems.com is for. Most UNIXy software will compile directly or with minor modifications, and they offer binaries for SSH, Apache, bash, and lots of others.
But who runs this?
I do, FWIW. There aren't a ton of users, but the number may grow more now that SFU is being included in 2003 R2 and business versions of Vista. It's targeted towards users with old apps running on an ancient commercial UNIX more than modern Linux and *BSD users.
Either way, that article is virtually irrelevant to today's systems other than to point out the pThread issue, and that MySQL does/did have issues on OSX.
Lock contention will always be a problem for hardware multithreading, regardless of architecture. I'm not saying that it's a problem that can't be fixed. Windows NT had some awful scaling problems as recently as NT4, when it used to struggle to scale past 4 CPUs executing heavy IO. Linux used the global-kernel-lock arch in 2.0, and wasn't preemptible until 2.6. I'm just saying that last I checked, OSX isn't ready for >4 heavy SMP yet.
For comparison, it would be hard to test OSX with a lot of CPUs because the most you can get with Apple hardware is what, 4? Why doesn't Apple have a high-end server model with a lot more?
There's a ton of Java developers out there. Their code does not run on MS OSes in general in production. Their tools are generally OS agnostic as well. In fact, in general, their tools run better on non-MS OSes. (Something about case-sensitive file systems)
NTFS is perfectly case sensitive. The Win32 interface to it generally isn't, but can be if you ask for it. AFAICT, Java is the one that decides to use filesystems in a case-insensitive manner, because that's what it asks for when it calls functions like CreateFile.
There's the additional advantage that it's a *nix subsystem, which happens to mesh nicely with our targeted deploy environments.
I'd love to see a 16 or 32 core Mac Pro in the near future - imagine the processing ability of such a system.:)
It's too bad that OSX has such kernel scaling problems, what with very coarse locking (somewhat improved in Tiger) and the necessity to use slow BSD user threads (as opposed to Mach kernel threads). Those things are going to need to be fixed before 16 or 32 cores are worthwhile, and I hope they don't have to break too much compatibility to do it.
You're confusing OS security design with user interfaces, install defaults, user habits and suspected company policies.
The Windows NT (which all Windows versions since XP are derived from) security model is comprehensive, powerful and granular. The biggest flaw it could be said to have is that it is too much of the above, and so too complicated. Every possibly sensitive object has a security descriptor which includes an ACL. Every process has a token which identifies its authority. I invite you to find a flaw in the actual OS security model.
Vista does indeed have an annoying policy of asking users to confirm each privileged action, thereby allowing them to be a privileged user without giving carte blanche to their processes. This is mostly a UI change, not a deep system change: it's a way for an unprivileged process to ask the user to bless a new process with greater privileges when the original process didn't have enough to do what it wanted to. The standard UNIX model of unprivileged users + full privilege admins has always existed on NT; UAC is simply a more convenient interface to sudo.
The past install and OEM default of creating only a single, full privilege user to be used for normal work does indeed suck for security. It is not, however, a required mode in any way. It has always been possible to log on as a safe unprivileged user for normal work. If the user and the OEM doesn't know better, and software developers have no concept of least privilege, then its their fault, not the OS's. Would you blame Linux or Ubuntu if Dell decided on a fork of Ubuntu that logged the user on as root silently by default and shipped software with the system that required root when it shouldn't?
About Microsoft's alleged priority of patching of WMP DRM failures faster than zero-day attacks, I'd like to see some evidence of that. The biggest worms all had patches released for them long before they infected computers. Besides, WMP and the core OS developers are in entirely different groups; you can't just transfer resources from one department to another in a big company like MS. IOW, it's an artifact of corporate bureaucracy, (not malicious intent or negligence) at best.
Yes, and Windows is still broadcasting (!) open ports [...]
What does mean, exactly? Have you read one too many "Your computer is broadcasting an IP address!!!!11" ads?
For one thing, since Win32 doesn't have setuid, it uses a privileged local service for such programs instead. A lot of the services are local only. Something like a device support service would naturally be local only. Of the remotely reachable services that start automatically by default in XP SP2, they either can't be turned off or are indeed on to be convenient.
Windows NT was designed for LANs in which a central authority can control all the computers and ask them for information, so the Local Security Authority service always listens for authorized network requests. The only way to make it unreachable from the network is using a firewall. LSA's design prevents it from being used by unprivileged users, but implementation flaws in it have fueled worms in the past.
In NT4, RPC became a necessary service because certain internal components started using it for local RPC. It's possible to disable the remote transports via the registry. RPC has been the source of multiple vulnerabilities, both on Windows and various UNIXes.
The computer browser is enabled by default to be convenient, and can be helpful any time you're connected to a network. The Server service enables file and printer sharing, and most users expect it to be on. Remote registry goes with the central admin design, but should probably be off. Some sites recommend turning all of these off. They haven't been vulnerable in the past AFAIK.
Lots of services are set to manual and only get started on demand. For example, if you set the Help and Support Center service to manual, it'll only be started when you open the Help and Support Center. Each service has a security descriptor which controls the users that are allowed to start and stop a service: most allow any interactive user to start them.
This site has a good overview of services and lists a conservative configuration. It isn't all that far from the Microsoft default config for the automatic starting of network services. Unfortunately, the worst offenders can't really be disabled and most malware gets in via privileged users running arbitrary binaries, not network worms.
A critical section makes use of a synchronization mechanism like a semaphore at entry and exit time to ensure that only one thread of execution (whether that be a process or a thread) is running in the critical section at any one time.
On Windows at least, a critical section requires no kernel object, and only a few instructions with no syscall to acquire and release as long as there is no contention on the object. If, while entering the section marked as already owned, a kernel notification event is created for waiters to sleep on. A kernel mutex OTOH, always requires a kernel object and a syscall for both acquire and release. Syscalls are quite expensive, making critical sections much faster in most cases. A design involving a large number of small lockable objects with rare contention would benefit from being able to use them in particular. I know that Solaris also has lightweight mutexes that can't be shared between processes, and I assume they avoid syscalls in most cases as well.
So what is it? YAMIHDE [Yet Another Microsoft In-House Database Engine]?
Yes, and a fairly old one. The NT registry format was created at the same time as NTFS. Herearesomepages about it. There are at least two versions of the format-- the newest one was introduced in XP. XP also made loading registry hives more efficient, and allows much larger hives to be created and loaded.
The Jets, the registry, NTFS, FoxPro (which they bought), and SQL Server are all the Microsoft multi-purpose binary database engines that I can think of. Jet Blue, the registry and NTFS are the only ones that the OS uses for itself today.
I could have sworn that I read a few years ago that they were ditching the existing registry engine, and were going with a new engine for Longhorn/Vista.
There was something about Cairo being a directory, registry, filesystem, etc. (and everything else under the sun). This blogger remembers it too.
I know that some Microsoft teams have been using the registry less, e.g. IIS6 now uses a new XML database for config instead, but not all of MS's many developers are moving in the same direction. The registry is required very early in the boot process to determine which drivers are necessary to load to access the boot volume and filesystem. Unless that changes, there would be little reason to replace the entire registry as it is with something else.
The BCD store is implemented as a registry hive, kept in either EFI or \Boot\Bcd. It's not mounted most of the time though; the boot loader mounts it long enough to read it and bcdedit mounts it just long enough to make an edit.
On the topic of one byte corruption of binary databases, what about the filesystem database? How many FSes could break boot with a single corrupted byte, e.g. in a filename record?
Like I said, you can mount any registry hive on a running Windows system, including the %SYSTEMDRIVE%\Boot\Bcd hive. bcdedit.exe does exactly that. A windows-based rescue disc can just as easily mount the hive for editing. It will make editing boot config from a different operating system's rescue disc a lot harder, though. More about bcdedit.exe and the BCD hive
To mount a registry hive manually from regedit, select either HKEY_LOCAL_MACHINE key or HKEY_USERS and select File->Load Hive. Note that you can also save and restore any key as a hive by selecting "Registry Hive Files" as the file type during export or import. See also reg.exe's load, unload, save and restore functions.
The registry is a single root hierarchical database with registry hive files mounted at the second level (below \REGISTRY\MACHINE and \REGISTRY\USER for the computer's config and user config, respectively). The registry engine is implemented in kernel mode as an executive subsystem (inside ntoskrnl.exe), where it is known as the Configuration Manager. Registry hives use a transaction journal (like many filesystems do) to avoid corruption during a power failure or crash. Standard system hives are located in %SYSTEMROOT%\System32\Config and include SAM for local user accounts, SECURITY for various secrets held by the computer, SYSTEM for core system configuration early during boot, and SOFTWARE that stores all other config associated with the computer in the registry. Every user profile has its own registry hive for user-specific configuration. Everything above is still the same in Vista as it was in NT 3.1.
There are two database engines that have been known as Microsoft "Jet", known as Jet Red and Jet Blue. Jet Red is also known as the Access database engine. It is a fairly featureful SQL database. Jet Blue is now officially the Extensible Storage Engine (ESE), and has been a system component since Windows 2000, backing WMI data, Active Directory, Exchange, and others. It is an ISAM database and is optimized for large sparse tables and also supports a transaction journal. Both are 100% user-mode and were not a part of the initial release of Windows NT. Microsoft has said that Jet Red is depreciated, and that future versions of the Access database engine will be integrated with Access and not have a public interface. Jet Blue's interface is well documented and will continue to see use for some time to come. Both being user-mode, dependent on Win32 and the wrong type of database (relational vs hierarchical), the Jet engines would not be suitable replacements for the registry.
SQL Server is a high-end SQL database engine. It was rumored that WinFS would use SQL Server Express and that Microsoft eventually plans to move some of the services that use Jet Blue to SQL Server (such as Active Directory). In any case, SQL Server is an even less possible replacement for the registry.
Microsoft has not gotten rid of the Registry in Vista. In fact, the new boot manager uses a registry hive to store boot configuration, replacing the old boot.ini.
Even if you were to get ReactOS running on (for instance) a G4 Mac mini, you're still not going to get the Windows version of Office running on it unless you have some sort of x86 emulation thrown in.
What they need is an OSX environment subsystem (like what Wine is for Win32 on Unix) for PPC Reactos to run Microsoft's PPC-Mac port of Office:)
Seriously, you're right that the compatibility angle would be largely defeated by running ReactOS on non x86 compatible hardware, but I think many people would also run ReactOS because they prefer a NT-style system to the more popular (on non-x86) UNIX variants. Besides, there are still many portable open-source apps and drivers that can and will be ported to non-x86 ReactOS.
NT is designed to be portable, and Microsoft has ported it to MIPS, Alpha, PPC, x86, Itanium and AMD64 each at one time or another. The ReactOS developers are retaining NT's portability, but running anything other than x86 is a very low priority right now.
My view is that if there does turn out to be a god (or gods), and the only reason I burn is hell is that I saw no evidence for their existance and thus didn't believe, then this is not a deity worthy of belief; let alone worship.
I attended a Christadelphian service today and the topic of the exhortation was how the change in style from direct statements, commands and such in the old testament to the Jesus's ambiguous parables in the new is an intentional measure to weed out superficial believers that only believe in things that are handed to them with tons of evidence. The idea is that only the true faithful will have the perseverance to study the Bible, work to interpret it correctly and so distance themselves from the fair weather believers. Apparently, Yahweh got sick of people not believing deeply enough around the book of Judges, having to keep cleaning up the recurring sinning and such, and wanted to change strategies to better separate the true faithful from those who aren't (including those who would require clear doctrine and evidence). Also, that if Jesus had been honest about who he was, the public might have liked him too much to crucify him (which would have defeated Yahweh's purpose of human sacrifice to enable forgiveness). Finally, that some people are called to be faithful whilst the others will be deaf to the truth, not understand and not believe.
I found it a bit silly that such an all-powerful god would have to resort to such deceitful means to further its ends. Still, it's
one explanation from believers for the lack of evidence for Christianity, the vagaries of Jesus's parables and for the myriad conflicting Christian doctrines. Perhaps the people who believe they see truth in certain religious writings are merely genetically predisposed to such belief?
Disclaimer: I'm a weak atheist (i.e. that I require all propositions (including "this god exists") to meet their own burden of proof and all the religious claims I've seen as yet have failed to meet that burden enough to follow).
That's just it: the window manager isn't broken. Unrelated windows are unaffected. Only the browser's own windows are unresponsive, because the browser itself is having a problem. No OS/window manager components are broken. I don't find it at all surprising that a process can break its own windows. Isn't this true of any windowing system?
That is the browser, its plugin model and Adobe's plugin's fault. The browser calls the plugin in the main thread that does everything, including window control, and the Adobe plugin blocks while trying to communicate with the Acrobat Reader process. In particular, it blocks until the Acrobat process is fully started. The browser's main thread (now blocked inside the Adobe pdf plugin) isn't processing window messages in the meantime, and so can't redraw, resize, etc. requests. If you click the close button, CSRSS realizes that the WM_CLOSE message isn't getting processed and offers to 1. let you wait it out, or 2. kill the process. Sometimes if you kill the Acrobat process, the Adobe pdf plugin will realize this and cancel gracefully. Sometimes not. This is an application issue that will have the same failings on any OS.
SignalObjectAndWait. A mutex is considered signaled when it is unowned; using a mutex as the object to signal will release the mutex (and then wait for the second object to become signaled atomically).
First, it appears that you are right about SuperFetch actually bringing back pages from the page file preemptively. I should have read the article you linked before posting.
I was a bit disappointed in part one of that series: they stated that all IO requests were limited to 64k by the IO manager, a fact I can demonstrate to be false in my own filesystem driver. Either the limitation exists in some other part of the OS or they were thinking of something else like the almost-64MB MDL size limit.
More priority control over IO and memory is great; I just hope that the user mode apps will use it properly. I'm afraid there's a long history of abusing/not using the kernel's features when they're needed most. CopyFile (used to at least) not set the caching options correctly on the files it opened, defaulting to full random access caching. The result when copying a large file (larger than the installed RAM) was that all your memory is wasted on two cached copies of the end of the file (one of the source copy, one of the destination copy) at the expense of other cached files and the standby list. If they'd only used no buffering (with multiple async requests) or set the sequential access caching flags... I just hope that the new priorities aren't ignored in the same way.
I've always been impressed with the kernel team's work on the OS, and pulling my hair out at the decisions made by 'higher level' teams like the shell. The problem of CSR minimizing a process's working set upon window minimize you mentioned is one example. The kernel team adding synchronous cancel functions to help ameliorate the shell team's inability to understand the concept of separating GUI and blocking IO threads is another. The new backup program's general awfulness... My favorite part about OSS is that you generally have lots of alternatives to crappy OS components, or at least the option to tweak their behavior yourself. I have high hopes for ReactOS, but they always seem to be two steps behind.
Eh, I guess I'm feeling some sour grapes syndrome over new kernel features that I can't have unless I totally switch to Vista along with its new annoyances. Is it so much to ask to have my cake and eat it too?:)
I've always been a bot confused by the ubiquitous amounts of "free memory" that could be put to better use; is it just that task manager doesn't report that memory as being used?
One point of confusion about memory usage in NT is how standby memory is reported. A page on the standby list used to belong to a process, but has been removed to reduce that process's working set. A copy of the page exists both in the page file and in memory, so no disk access is required to either give it back to the original process (a soft fault) OR to use the memory for something else, since the data is already in both places. The standby list tends to be quite large (since the memory manager trims process working sets aggressively) and is double reported by task manager as both available memory and system cache. NT4's task manager showed only file cache instead of the system cache (file cache + standby). Available memory includes both standby and free memory. I guess the rationale is that standby pages are available for other uses without touching the disk. The standby list system is used in every version of NT. The kd standard extension !memusage will report free, standby, active, etc. memory (which really will add up to your installed RAM).
NT does not bring pages back from the page file unless they're faulted back (or possibly from read-ahead of sequential faults). This is one reason that VM's page back so horribly: there's one thread accessing memory randomly (so the kernel doesn't think to read ahead) waiting for each page to be read back one at a time. This also means that you'll be left with a lot of free memory after a large process (like a game) quits. I'd be happy if there was some way to bring back pages preemptively when the memory becomes free like the Linux mod you mentioned. I'm pretty sure that Vista's superfetch works only with the file cache (pre-caching files that are expected to be read), not with process working sets. Fun fact: the NT file cache works by memory mapping cached ranges of a file into system space (with the same functions as user mode mmaping) and then using memcpy to complete the request.
Note that this is because the backup application uses the shadow copy service to take a snapshot of the system
Actually, sdclt (Vista Backup) does NOT seem to use VSS, or else it'd be able to back up registry hives directly. Instead, it skips them because they're locked. In any case, sdclt should be able to revert to non-vss mode like ntbackup does. Sdclt is also missing many other ntbackup functions like backing up EFS files in their encrypted form, and actually being able to select the individual files and directories you want to back up. There was a thread about it on Shell:Revealed, but it's been deleted.
Also note that you can download the old backup program for Vista Business / Ultimate, IIRC.
I can't find any such download. I wish there was one. The closest is this
ntbackup archive restore only program, and a possible way to install the old ntbackup that may be unsafe because it requires a different (incompatible) version of vssapi.dll than the OS uses. Either that or use the NT4 version of ntbackup and give up modern features like VSS... which is still better than sdclt.
Unfortunately, not all the help is good. The new backup program (sdclt.exe) includes two silly and obvious limitations (that the old ntbackup.exe didn't), namely the requirement to have admin privileges to backup your own files, and the inability to put the archive just anywhere (e.g. the same hard drive or a USB flash drive). The backup program has hyperlink style help buttons for exactly those questions, but they both redirect to a general FAQ about the backup program that doesn't actually include those questions or answers to them.
Microsoft is made up of many developers and departments, each of varying responsibilities, skills, and understanding of the OS. Accordingly, parts of the software range from excellent quality with great documentation and system integration to awful, mysterious pieces that barely work. This has been the norm for Microsoft software (and many other large projects) for a long time.
Here's an exhaustive list of Windows NT syscalls in every service pack since NT4 SP3. NT 3.1 (not listed) has 180, NT 4.0 has 248, XP has 284, Vista has 394 (the greatest increase in a single version), mostly for transaction support, a new IPC mechanism and configuring the new boot loader. I'm not familiar with most of Vista's new functions, but I know that all the functions in XP are necessary. BTW, Linux 2.6.20 has 319 syscalls (according to arch/i386/kernel/syscall_table.S). Several of the Linux syscalls have become placeholders, obselete. Show me a single obsolete/compatibility driven NT kernel syscall.
Applications interface with Win32, not the syscall API (also known as the NTAPI or the native API). Win32 is where all the compatibility hacks are, and it is indeed more ugly because of it. NTAPI is insulated from apps and contains no compatibility hacks or baggage. Even so, the NTAPI is very stable; I'm not aware of a single function that was implemented and has changed or become obsolete or depreciated.
All the mess of Vista development is in user-mode, especially in Win32 and the shell. All of the features planned for the kernel in Vista have shipped and were ready long before release, AFAICT. It's most of the user-mode stuff that's been scrapped or scaled down and is a mess.
If they're showing syscalls, then what are all the lines connecting them? Syscalls don't call each other; they're an array of functions called from user to kernel mode, in that direction only. How does one show relationships between syscalls exactly? It's awfully convenient that the graphs are too blurry to actually read the bubble text or we might be able divine what they're talking about.
Alternate file streams have a lot in common with extended attributes. NTFS supports both, and the idea has its uses. The biggest difference between them is that alternate file streams can store much more data effectively, since they support random access data operations.
Normal APIs don't support extra streams. Getting fopen() to work with streams is a hack, to put it mildly.
The full format for an NTFS attribute is <path>\<file>[:<stream>[:<attribute>]] (the angle brackets denote data fields, the square brackets denote optional fields and other characters are literal) If you don't name any stream, you get the default, null name stream and if you don't name any attribute you get the $DATA attribute (as oppsed to others like $EA, $SECURITY_DESCRIPTOR and $FILE_NAME; everything about a file is stored in a named attribute), i.e. <path>\<file>::$DATA.
Any function that can pass colons in the filename to the kernel can open alternate file streams, no hacks required.
The notation used on Windows is... interesting. If you are in D:\ with a file called C, does C:foo refer to a stream on D:\C or to a file called foo in the current directory of the C drive?
Whenever Win32 sees a path starting with a letter and a colon, it's always interpreted as a drive letter. This is a Win32 issue since drive letters are a Win32 (not kernel) idiom.
On a Linux or MacOS system, all characters except '/' and '\0' are valid in filenames, so we have nothing to spare. No, you can't steal the ':'.
On NTFS, : " / \ | are reserved. Note that the null character IS NOT reserved (although Win32 doesn't support it).
Today I can copy a file with the dd command. I can copy a file using the cat command and shell redirection. Multi-forked files would lose data.
There ARE some special versions of cp and related that properly support streams and extended attributes, although most Windows programs don't.
Do these extra streams get permission bits?
No. The security descriptor belongs to the file, not a specific stream. These are pieces of data that are meant to be about the file, with the same sensitivity as the main data.
Can a stream have a stream?
No. There is only one level of indirection from the file itself to a stream. An entire hierarchy of metadata would be unnecessary; a hierarchical naming convention (like the reverse-DNS scheme Apple recommends for extended attributes) would be much more useful. It wouldn't be good for extended attributes to have subordinate EAs either.
Can I move a stream from one file to another? Can I move a stream to be just a regular file? Can I move a file into another file, to become an extra stream?
A data stream can only be renamed within a file. In other words, a rename operation cannot cause a data stream to be moved to a different file.
A stream on a directory cannot be renamed to the default data stream.
"Renaming" the default data stream is allowed, but this is not a true rename, because it leaves behind a zero-length default data stream.
These are attributes about a file, not files in themselves. The most they have in common with files is that you can use the standard read and write functions on their data.
Why should everything become more complex (buggy, slow, insecure, confusing, etc.) for this barely-useful feature?
I don't think they're slowing things down, and insecurity by obscurity is a high-level interface problem, not a filesystem problem. Extended attributes and alternate st
Slashdot Mirror
Actually, it seems to be selective mirroring at the file level, so it's like RAID1 for the files and directories that you ask to have mirrored. I don't know how may pieces of WS2003 are included, but you may be able to build a software RAID5 dynamic disk set manually.
The target market is for people who don't install the OS themselves-- they buy a computer from an OEM with the hardware and OS preinstalled. The OEM will make sure that the drives are hot-swappable and such. Adding new drives and merging them into a RAID is done automatically and silently upon plugging the drive in. They'll be happy to make sure you buy only their specially branded "WHS compatible" hot swappable drives for that purpose. I would be surprised if a retail version is even available on shelves.
Single instance storage has existed since at least Windows 2000. The "recover older versions" thing is connected to the "older versions" shell extension, also connected to volume shadow copies, giving you the option to restore from both known backups on file and local VSS copies of a file. Automated remote backups aren't hard to implement with file sharing, and I would think that WHS comes with some simple UI to set it up, with which workstations to backup and corresponding credentials on each.
Remote control uses RDP. I think it will be configured to accept only connections on the local network, and will certainly require some kind of authentication.
See also:
Windows Home Server on Wikipedia
A demo video from MS
an OEM offering from HP
It's because Notepad supports both UCS-2 and ASCII encodings, but without any metadata or conventional file headers to tell it which format is actually being used. Unless you tell it otherwise, Notepad uses the Win32 function IsTextUnicode() to try and guess the format of the text. IsTextUnicode() offers only a best-effort attempt to detect what format a blob of text is in. See the description of IsTextUnicode() for the various fuzzy detection methods it offers. It doesn't always work, especially if the sample size is small.
Notepad tries to "just work" with both formats of text based only on the contents of the text. There is no strict solution, given the lack of metadata. Most text editors require you to explicitly state the format of text (or always assume some default), leaving no possibility of accidental failure.
Problems start when the app passes along data from some outside untrusted source without understanding its content or validating it, like when a web browser passes an
Right now, the trust assumption of many user libraries hasn't been fixed because there is a lot of code in that position and it would be a lot of work to go through it all. Managers hate fixing code issues like that because it takes a lot of time and money but doesn't result in anything tangible like pretty features. Applications already suffer enough code bloat without having to implement validation for all the data they come in contact with that gets passed right to support libraries-- managers don't want to spend time and money on validating things that should be someone else's responsibility. Microsoft has had this class of vulnerability on low priority for a long time, and it's been the source of A LOT of issues.
I'm not excusing Microsoft's behavior, just trying to explain it somewhat. Someone sure dropped the ball in not finding finding problems similar to the 2005 issue though.Microsoft is a big company. Not every department is following the Security Development Lifecycle, as much as marketing may like to imply it. The main two examples that do are SQL Server 2005 and IIS6, both of which are doing very well. I haven't heard Microsoft say that all of Windows or the Win32-GUI core team were using SDL.Vista contains copious amounts of new code, but very little of it replaced old code. The sound system (the mixer mainly) was largely rewritten, the backup program got replaced with a POS from scratch, the logon GUI arch (i.e. msgina.dll replacements) got replaced... and I can't think of anything else that is new code to replace old code. I'd say that at least 75% of the Windows NT3.1 code base is still present in Vista.
SFU doesn't use Win32, and never uses OBJ_CASE_INSENSITIVE AFAIK. It's true that a lot of Win32 apps will choke at the sight of two filenames differing only in case, but then there are a lot of things that most of them don't do properly, like LUA. I agree that Win32 is an ugly mess, that plenty of software built on it is as bad, and that it should all go away... but one of Microsoft's biggest claims to fame is that mass of software built on that interface. They're not about to give it up. Just don't confuse Win32 with the kernel or NTFS.It's too bad Microsoft doesn't include more standard tools, but that's what interopsystems.com is for. Most UNIXy software will compile directly or with minor modifications, and they offer binaries for SSH, Apache, bash, and lots of others.I do, FWIW. There aren't a ton of users, but the number may grow more now that SFU is being included in 2003 R2 and business versions of Vista. It's targeted towards users with old apps running on an ancient commercial UNIX more than modern Linux and *BSD users.Lock contention will always be a problem for hardware multithreading, regardless of architecture. I'm not saying that it's a problem that can't be fixed. Windows NT had some awful scaling problems as recently as NT4, when it used to struggle to scale past 4 CPUs executing heavy IO. Linux used the global-kernel-lock arch in 2.0, and wasn't preemptible until 2.6. I'm just saying that last I checked, OSX isn't ready for >4 heavy SMP yet.
For comparison, it would be hard to test OSX with a lot of CPUs because the most you can get with Apple hardware is what, 4? Why doesn't Apple have a high-end server model with a lot more?
The Windows NT (which all Windows versions since XP are derived from) security model is comprehensive, powerful and granular. The biggest flaw it could be said to have is that it is too much of the above, and so too complicated. Every possibly sensitive object has a security descriptor which includes an ACL. Every process has a token which identifies its authority. I invite you to find a flaw in the actual OS security model.
Vista does indeed have an annoying policy of asking users to confirm each privileged action, thereby allowing them to be a privileged user without giving carte blanche to their processes. This is mostly a UI change, not a deep system change: it's a way for an unprivileged process to ask the user to bless a new process with greater privileges when the original process didn't have enough to do what it wanted to. The standard UNIX model of unprivileged users + full privilege admins has always existed on NT; UAC is simply a more convenient interface to sudo.
The past install and OEM default of creating only a single, full privilege user to be used for normal work does indeed suck for security. It is not, however, a required mode in any way. It has always been possible to log on as a safe unprivileged user for normal work. If the user and the OEM doesn't know better, and software developers have no concept of least privilege, then its their fault, not the OS's. Would you blame Linux or Ubuntu if Dell decided on a fork of Ubuntu that logged the user on as root silently by default and shipped software with the system that required root when it shouldn't?
About Microsoft's alleged priority of patching of WMP DRM failures faster than zero-day attacks, I'd like to see some evidence of that. The biggest worms all had patches released for them long before they infected computers. Besides, WMP and the core OS developers are in entirely different groups; you can't just transfer resources from one department to another in a big company like MS. IOW, it's an artifact of corporate bureaucracy, (not malicious intent or negligence) at best.What does mean, exactly? Have you read one too many "Your computer is broadcasting an IP address!!!!11" ads?How, exactly?
For one thing, since Win32 doesn't have setuid, it uses a privileged local service for such programs instead. A lot of the services are local only. Something like a device support service would naturally be local only. Of the remotely reachable services that start automatically by default in XP SP2, they either can't be turned off or are indeed on to be convenient.
Windows NT was designed for LANs in which a central authority can control all the computers and ask them for information, so the Local Security Authority service always listens for authorized network requests. The only way to make it unreachable from the network is using a firewall. LSA's design prevents it from being used by unprivileged users, but implementation flaws in it have fueled worms in the past.
In NT4, RPC became a necessary service because certain internal components started using it for local RPC. It's possible to disable the remote transports via the registry. RPC has been the source of multiple vulnerabilities, both on Windows and various UNIXes.
The computer browser is enabled by default to be convenient, and can be helpful any time you're connected to a network. The Server service enables file and printer sharing, and most users expect it to be on. Remote registry goes with the central admin design, but should probably be off. Some sites recommend turning all of these off. They haven't been vulnerable in the past AFAIK.
Lots of services are set to manual and only get started on demand. For example, if you set the Help and Support Center service to manual, it'll only be started when you open the Help and Support Center. Each service has a security descriptor which controls the users that are allowed to start and stop a service: most allow any interactive user to start them. This site has a good overview of services and lists a conservative configuration. It isn't all that far from the Microsoft default config for the automatic starting of network services. Unfortunately, the worst offenders can't really be disabled and most malware gets in via privileged users running arbitrary binaries, not network worms.
The Jets, the registry, NTFS, FoxPro (which they bought), and SQL Server are all the Microsoft multi-purpose binary database engines that I can think of. Jet Blue, the registry and NTFS are the only ones that the OS uses for itself today.There was something about Cairo being a directory, registry, filesystem, etc. (and everything else under the sun). This blogger remembers it too.
I know that some Microsoft teams have been using the registry less, e.g. IIS6 now uses a new XML database for config instead, but not all of MS's many developers are moving in the same direction. The registry is required very early in the boot process to determine which drivers are necessary to load to access the boot volume and filesystem. Unless that changes, there would be little reason to replace the entire registry as it is with something else.
The BCD store is implemented as a registry hive, kept in either EFI or \Boot\Bcd. It's not mounted most of the time though; the boot loader mounts it long enough to read it and bcdedit mounts it just long enough to make an edit.
On the topic of one byte corruption of binary databases, what about the filesystem database? How many FSes could break boot with a single corrupted byte, e.g. in a filename record?
Like I said, you can mount any registry hive on a running Windows system, including the %SYSTEMDRIVE%\Boot\Bcd hive. bcdedit.exe does exactly that. A windows-based rescue disc can just as easily mount the hive for editing. It will make editing boot config from a different operating system's rescue disc a lot harder, though. More about bcdedit.exe and the BCD hive
To mount a registry hive manually from regedit, select either HKEY_LOCAL_MACHINE key or HKEY_USERS and select File->Load Hive. Note that you can also save and restore any key as a hive by selecting "Registry Hive Files" as the file type during export or import. See also reg.exe's load, unload, save and restore functions.
The registry is a single root hierarchical database with registry hive files mounted at the second level (below \REGISTRY\MACHINE and \REGISTRY\USER for the computer's config and user config, respectively). The registry engine is implemented in kernel mode as an executive subsystem (inside ntoskrnl.exe), where it is known as the Configuration Manager. Registry hives use a transaction journal (like many filesystems do) to avoid corruption during a power failure or crash. Standard system hives are located in %SYSTEMROOT%\System32\Config and include SAM for local user accounts, SECURITY for various secrets held by the computer, SYSTEM for core system configuration early during boot, and SOFTWARE that stores all other config associated with the computer in the registry. Every user profile has its own registry hive for user-specific configuration. Everything above is still the same in Vista as it was in NT 3.1.
There are two database engines that have been known as Microsoft "Jet", known as Jet Red and Jet Blue. Jet Red is also known as the Access database engine. It is a fairly featureful SQL database. Jet Blue is now officially the Extensible Storage Engine (ESE), and has been a system component since Windows 2000, backing WMI data, Active Directory, Exchange, and others. It is an ISAM database and is optimized for large sparse tables and also supports a transaction journal. Both are 100% user-mode and were not a part of the initial release of Windows NT. Microsoft has said that Jet Red is depreciated, and that future versions of the Access database engine will be integrated with Access and not have a public interface. Jet Blue's interface is well documented and will continue to see use for some time to come. Both being user-mode, dependent on Win32 and the wrong type of database (relational vs hierarchical), the Jet engines would not be suitable replacements for the registry.
SQL Server is a high-end SQL database engine. It was rumored that WinFS would use SQL Server Express and that Microsoft eventually plans to move some of the services that use Jet Blue to SQL Server (such as Active Directory). In any case, SQL Server is an even less possible replacement for the registry.
Microsoft has not gotten rid of the Registry in Vista. In fact, the new boot manager uses a registry hive to store boot configuration, replacing the old boot.ini.
Seriously, you're right that the compatibility angle would be largely defeated by running ReactOS on non x86 compatible hardware, but I think many people would also run ReactOS because they prefer a NT-style system to the more popular (on non-x86) UNIX variants. Besides, there are still many portable open-source apps and drivers that can and will be ported to non-x86 ReactOS.
NT is designed to be portable, and Microsoft has ported it to MIPS, Alpha, PPC, x86, Itanium and AMD64 each at one time or another. The ReactOS developers are retaining NT's portability, but running anything other than x86 is a very low priority right now.
I found it a bit silly that such an all-powerful god would have to resort to such deceitful means to further its ends. Still, it's one explanation from believers for the lack of evidence for Christianity, the vagaries of Jesus's parables and for the myriad conflicting Christian doctrines. Perhaps the people who believe they see truth in certain religious writings are merely genetically predisposed to such belief?
Disclaimer: I'm a weak atheist (i.e. that I require all propositions (including "this god exists") to meet their own burden of proof and all the religious claims I've seen as yet have failed to meet that burden enough to follow).
That's just it: the window manager isn't broken. Unrelated windows are unaffected. Only the browser's own windows are unresponsive, because the browser itself is having a problem. No OS/window manager components are broken. I don't find it at all surprising that a process can break its own windows. Isn't this true of any windowing system?
That is the browser, its plugin model and Adobe's plugin's fault. The browser calls the plugin in the main thread that does everything, including window control, and the Adobe plugin blocks while trying to communicate with the Acrobat Reader process. In particular, it blocks until the Acrobat process is fully started. The browser's main thread (now blocked inside the Adobe pdf plugin) isn't processing window messages in the meantime, and so can't redraw, resize, etc. requests. If you click the close button, CSRSS realizes that the WM_CLOSE message isn't getting processed and offers to 1. let you wait it out, or 2. kill the process. Sometimes if you kill the Acrobat process, the Adobe pdf plugin will realize this and cancel gracefully. Sometimes not. This is an application issue that will have the same failings on any OS.
SignalObjectAndWait . A mutex is considered signaled when it is unowned; using a mutex as the object to signal will release the mutex (and then wait for the second object to become signaled atomically).
First, it appears that you are right about SuperFetch actually bringing back pages from the page file preemptively. I should have read the article you linked before posting.
:)
I was a bit disappointed in part one of that series: they stated that all IO requests were limited to 64k by the IO manager, a fact I can demonstrate to be false in my own filesystem driver. Either the limitation exists in some other part of the OS or they were thinking of something else like the almost-64MB MDL size limit.
More priority control over IO and memory is great; I just hope that the user mode apps will use it properly. I'm afraid there's a long history of abusing/not using the kernel's features when they're needed most. CopyFile (used to at least) not set the caching options correctly on the files it opened, defaulting to full random access caching. The result when copying a large file (larger than the installed RAM) was that all your memory is wasted on two cached copies of the end of the file (one of the source copy, one of the destination copy) at the expense of other cached files and the standby list. If they'd only used no buffering (with multiple async requests) or set the sequential access caching flags... I just hope that the new priorities aren't ignored in the same way.
I've always been impressed with the kernel team's work on the OS, and pulling my hair out at the decisions made by 'higher level' teams like the shell. The problem of CSR minimizing a process's working set upon window minimize you mentioned is one example. The kernel team adding synchronous cancel functions to help ameliorate the shell team's inability to understand the concept of separating GUI and blocking IO threads is another. The new backup program's general awfulness... My favorite part about OSS is that you generally have lots of alternatives to crappy OS components, or at least the option to tweak their behavior yourself. I have high hopes for ReactOS, but they always seem to be two steps behind.
Eh, I guess I'm feeling some sour grapes syndrome over new kernel features that I can't have unless I totally switch to Vista along with its new annoyances. Is it so much to ask to have my cake and eat it too?
NT does not bring pages back from the page file unless they're faulted back (or possibly from read-ahead of sequential faults). This is one reason that VM's page back so horribly: there's one thread accessing memory randomly (so the kernel doesn't think to read ahead) waiting for each page to be read back one at a time. This also means that you'll be left with a lot of free memory after a large process (like a game) quits. I'd be happy if there was some way to bring back pages preemptively when the memory becomes free like the Linux mod you mentioned. I'm pretty sure that Vista's superfetch works only with the file cache (pre-caching files that are expected to be read), not with process working sets. Fun fact: the NT file cache works by memory mapping cached ranges of a file into system space (with the same functions as user mode mmaping) and then using memcpy to complete the request.
Unfortunately, not all the help is good. The new backup program (sdclt.exe) includes two silly and obvious limitations (that the old ntbackup.exe didn't), namely the requirement to have admin privileges to backup your own files, and the inability to put the archive just anywhere (e.g. the same hard drive or a USB flash drive). The backup program has hyperlink style help buttons for exactly those questions, but they both redirect to a general FAQ about the backup program that doesn't actually include those questions or answers to them.
Microsoft is made up of many developers and departments, each of varying responsibilities, skills, and understanding of the OS. Accordingly, parts of the software range from excellent quality with great documentation and system integration to awful, mysterious pieces that barely work. This has been the norm for Microsoft software (and many other large projects) for a long time.
Here's an exhaustive list of Windows NT syscalls in every service pack since NT4 SP3. NT 3.1 (not listed) has 180, NT 4.0 has 248, XP has 284, Vista has 394 (the greatest increase in a single version), mostly for transaction support, a new IPC mechanism and configuring the new boot loader. I'm not familiar with most of Vista's new functions, but I know that all the functions in XP are necessary. BTW, Linux 2.6.20 has 319 syscalls (according to arch/i386/kernel/syscall_table.S). Several of the Linux syscalls have become placeholders, obselete. Show me a single obsolete/compatibility driven NT kernel syscall.
Applications interface with Win32, not the syscall API (also known as the NTAPI or the native API). Win32 is where all the compatibility hacks are, and it is indeed more ugly because of it. NTAPI is insulated from apps and contains no compatibility hacks or baggage. Even so, the NTAPI is very stable; I'm not aware of a single function that was implemented and has changed or become obsolete or depreciated.
All the mess of Vista development is in user-mode, especially in Win32 and the shell. All of the features planned for the kernel in Vista have shipped and were ready long before release, AFAICT. It's most of the user-mode stuff that's been scrapped or scaled down and is a mess.
If they're showing syscalls, then what are all the lines connecting them? Syscalls don't call each other; they're an array of functions called from user to kernel mode, in that direction only. How does one show relationships between syscalls exactly? It's awfully convenient that the graphs are too blurry to actually read the bubble text or we might be able divine what they're talking about.
The full format for an NTFS attribute is <path>\<file>[:<stream>[:<attribute>]] (the angle brackets denote data fields, the square brackets denote optional fields and other characters are literal) If you don't name any stream, you get the default, null name stream and if you don't name any attribute you get the $DATA attribute (as oppsed to others like $EA, $SECURITY_DESCRIPTOR and $FILE_NAME; everything about a file is stored in a named attribute), i.e. <path>\<file>::$DATA.
Any function that can pass colons in the filename to the kernel can open alternate file streams, no hacks required.
Whenever Win32 sees a path starting with a letter and a colon, it's always interpreted as a drive letter. This is a Win32 issue since drive letters are a Win32 (not kernel) idiom.
On NTFS, : " / \ | are reserved. Note that the null character IS NOT reserved (although Win32 doesn't support it).
There ARE some special versions of cp and related that properly support streams and extended attributes, although most Windows programs don't.
No. The security descriptor belongs to the file, not a specific stream. These are pieces of data that are meant to be about the file, with the same sensitivity as the main data.
No. There is only one level of indirection from the file itself to a stream. An entire hierarchy of metadata would be unnecessary; a hierarchical naming convention (like the reverse-DNS scheme Apple recommends for extended attributes) would be much more useful. It wouldn't be good for extended attributes to have subordinate EAs either.
No, no, no. From FILE_RENAME_OPERATION:
These are attributes about a file, not files in themselves. The most they have in common with files is that you can use the standard read and write functions on their data.
I don't think they're slowing things down, and insecurity by obscurity is a high-level interface problem, not a filesystem problem. Extended attributes and alternate st