wake up masher sounds like you've drinking a bit too much of that constitutional kool-aid all "democracies" have been screwing over their citizens from day 1. this just didn't happen under Obama's watch and blaming him is weak cop out.
"So, for example, if you go back a century ago, right after the U.S. invasion of the Philippines — a brutal invasion that killed a couple hundred thousand people — there was a problem for the U.S. of pacification afterwards. What do you do to control the population to prevent another nationalist uprising? There’s a very good study of this by Alfred McCoy, a Philippines scholar at University of Wisconsin, and what he shows is that the U.S. used the most sophisticated technology of the day to develop a massive system of survelliance, control, disruption to undermine any potential opposition and to impose very tight controls on the population which lasted for a long time and in many ways the Philippines is still suffering from this. But he also points out the technology was immediately transferred home. Woodrow Wilson’s administration used it in their “Red Scare” a couple years later. The British used it, too."
pick any period in history and these power hungry cunts have always been trying to stay in power no matter what. I haven't got any answers just know that what we call democracy is not working because it always allows the people in power to get away with shit.
Poor article I don't think roblimo really went along with an open mind so he seems to have spent his tme nitpicking e.g. he states in his article that you need Internet Explorer to use virtual earth but it works fine in FireFox 2.0 .
You can fix this but at your own risk, the extensions will load but they might not be compatible with the changes in that version of firefox. All extensions have a file called install.rdf. There is a section called maxVersion that Firefox checks to see if it should enable or disable the extension. If maxVersion is lower than the current version, then Firefox automatically disables the extension because it considers it to be incompatible.
To modify install.rdf do the following
1. Close Firefox 2. Open %APPDATA%\Mozilla\Firefox\Profiles\ 3. Delete extensions.rdf 4. Go to the extensions folder. 5. Now you'll have to go to every folder there and edit its install.rdf file with a texteditor such as notepad. 6. You will see something like this: CODE
I'm fairly certain that if Open Office significantly eats into Office's market share that Microsoft will release a cut down version that is significantly cheaper or rmaybe even free. They've already done this with their cut down version of XP
A Short History of Nearly Everything by Bill Bryson was the book I enjoyed the most this year. He did make a few mistakes and he does gloss things over but it's an excellent read for anyone that wants to know about most of the major scientific advances in the last 300 years and the people that have made them. For me the real strength of the book is the way he brings these people to life with his anecdotes and the fact that he makes the very important point of how incredibly little we know.
so why did they register AppleUniversal.com a few days ago?
Domain Name: APPLEUNIVERSAL.COM
Registrar: BULKREGISTER.COM, INC.
Whois Server: whois.bulkregister.com
Referral URL: http://www.bulkregister.com
Name Server: NSERVER2.APPLE.COM
Name Server: NSERVER.APPLE.COM
Status: ACTIVE
Updated Date: 11-apr-2003
Creation Date: 11-apr-2003
Expiration Date: 11-apr-2004 NOTICE: The expiration date displayed in this record is the date the registrar's sponsorship of the domain name registration in the registry is currently set to expire. This date does not necessarily reflect the expiration date of the domain name registrant's agreement with the sponsoring registrar. Users may consult the sponsoring registrar's Whois database to view the registrar's reported date of expiration for this registration.
I would have thought it would be a bold and intelligent move to ask the Chinese to make a contribution. Wouldn't you rather have the Chinese to be working with the ISS partners rather than competing against them.
I agree that Chinese spies on ISS are simply unacceptable but hasn't China signed all the various non-proliferation agreements and surely the international prestige such a participation would bring them would stop the Chinese doing anything stupid.
Their human rights record does bother me and it needs to improve but that has never stopped the US getting into bed with countries with similar or worse records.
Although China has
announced that it's planning a permanently manned space station this seems like a waste of time, effort and money. I think it would make more sense to let China either take Russia's place or just let them join the ISS program. But I guess relations between the US and China need to improve before this could happen.
Hmm don't they insure these satellites against any resulting financial loss? Surely this would negate the need need for any reduncies. But on the other hand their premiums are probably going to shoot through the roof so maybe....
Unfortunately a lot of people don't actually read the EULA. They just click through until the software is installed. Even if you do read it it's full of dense obscure legal language that mostly doesn't apply to you. Advertising software if implemented correctly can allow developers to make money from their software without requiring the end user to pay.
The problem is it's often not done properly. There are spyware apps like
aureate that operate in stealth mode by passing themselves off as Windows system processes and making sure that they don't even show up the task list or binding themselves to winsock so that you delete or uninstall them your Internet connection stops working. Microsoft should be made to fix these holes in IE but I think some pressure should also be applied to the people that write these programs.
The hypotheses competing with panspermia is abiogenesis. Abiogeneis theorises that life can arise spontaneously from non-life molecules under proper conditions. I tend to favour abiogeneis slightly over panspermia simply because we know that there is life on Earth, but we don't know if there is any elsewhere in the Universe.
The four steps to necessary for Abiogenesis are:
Inorganic Molecules to Organic Monomers
Organic Monomers to Organic Polymers
Formation of membranes from the polymers
Acquisition of a means of reproduction
Maybe the asteroids instead of seeding the earth provided the energy required for the first step.
This approach would only work in the short term because there are already file hashing applications like sig2dat that help p2p users share exact copies of files that they have verified as good.
These are the main lessons that we can extract from the Hotmail conversion.
1) We need a consistent, comprehensive, thoughtful approach to integrated management of a set of servers. This does not necessarily mean that we should slavishly follow the UNIX model of iterating through a list of machines with an/rsh/command, or pushing configuration files to a list of machines. The fundamental goal is to be able to manage machines as an aggregate; doing this through a GUI is not necessarily evil, so long as it can be done remotely, and once. The point applies to application distribution as well as to system tuning.
2) NLBS is at an economic disadvantage, due to its association with Advanced Server, and Hotmail operations staff were sufficiently satisfied with the existing solution that they did not feel the need to investigate NLBS?s operational advantages.
3) The metabase needs to be ripped out and replaced with something that is much easier for an administrator to see and understand, and be confident that there are no hidden surprises. The IIS6 planners have heard this opinion.
4) It should be easier to tune and lock down a single system, and have the changes propagated to all systems in a given class.
5) Windows is too complex to understand at first, particularly during a conversion from UNIX. There are just too many things about it for a planner in a startup to understand. Typically there is little time to attend training. The problem is most Computer Science graduates come to their startups already understanding enough about UNIX to be confident that they can use it effectively. We do need to be careful to balance the complexity and transparency carefully.
6) The basic need for an Internet site, converting from UNIX to Windows, is to be able to quickly replace their application and operational methodology with something at least equally good. Improvements that come for free are good, but implementing new technologies and programming methods will need to take a back seat so long as they delay the main purpose, which is to keep a site online and competitive. Anything else is a cost that needs to bring a clear benefit.
Helping UNIX system administrators with the transition to Windows is an experience in itself, and much was learned. Again, this is data from a single corporate experience, but we suspect it is fairly typical. Here, then, is the human engineering overview.
Initially, the plan to convert from FreeBSD to Windows was met with responses ranging from skepticism to hostility, in a way that should be familiar to those who share the attitudes of the various UNIX communities to Microsoft software.
We engaged with the operations staff by asking them to define what their everyday tasks are, in all areas of operating system and application maintenance. Instead of a set of tasks, we were handed a set of the UNIX commands and features that were used to carry out those tasks. While this did not directly meet the need, it gave us an opportunity to address all of the features directly, and show that Windows has an exact equivalent in the core system, or in the Resource Kit, or easily provided with a script. There were very few cases where no satisfactory alternative could be found. Essentially, this was throwaway work, as the eventual solution solved the problems in a more Windows-like way, but it was an excellent opportunity to gain the confidence of the operations staff.
It was clear from the responses that some people from the UNIX side of the house cannot distinguish our different systems that are marketed under the Windows brand; there was an inbuilt assumption that Windows 2000 shares the features and faults of Windows 95. Those who were somewhat familiar with Windows NT were not aware of the range of the non-GUI offerings (to be fair, neither were we); the set of commands in the product and the Resource Kit is fairly broad although, as we have seen, there are gaps and they lack stylistic consistency.
Other staff members, not members of the regular operations team, carried out the conversion. When deployment came near, the Operations staff had to learn the new tools and paradigms. Their existence proved enough; the main interest of Operations staff is, after all, to run and administer the system, and once they found that there were tools, whether custom-built or standard, that did the job well enough, they were able to take control and gain a sense of ownership. Some standard one-day courses were also given to the staff, to prepare them for handling system debugging, hotfix application and so on. By this time, the staff had become convinced that Windows is, after all, a real operating system with surprising richness.
It is naturally a requirement that a web-based service operate continually, without customer-visible degradations of service. This is not just a matter of pride; even a loss of availability for a few minutes every month can produce too much degradation in the perceptions and (assuming we publish uptime numbers) the availability measurement.
It?s a solution, but a weak one, to put servers behind load-balancing equipment and take them out of service when required for upgrade or other maintenance. The challenge is to keep each server running continuously as much as possible. Except for operating system upgrades, a system based on FreeBSD and Apache can keep operating while the application is upgraded, and Windows should be able to do the same.
Application updates at Hotmail are of two kinds: content and code. Content updates change only data files, generally those that directly determine what the customer sees on the screen, and they are carried out on their own schedule. Apache can handle both content and code updates without stopping the service. Updates can be rolled out directly, when the data is updated in place. They can also be timed, when the updates are put on the servers in a staging location together with an update batch job that will be triggered at the desired moment. The timed update is used when it is important for the application?s integrity that the entire site be updated simultaneously, something that is impossible to achieve when updating several thousand servers across a single network.
Application update techniques
Apache running under UNIX supports both kinds of updates very simply. A CGI application can be replaced, even while the old file is being executed, and the next execution will use the new file. The same is true of content. If Apache?s own configuration files must be updated, there is a procedure to signal the server to reset itself and reread its configuration, and that takes around a second.
Unfortunately, IIS 5.0 does not support either kind of update well. When IIS accesses content directly, it locks the folders. Fortunately, this doesn?t apply to most of the Hotmail upgrades. The bigger issue is updating the ISAPI filters, which must be done while the IIS server is stopped. The entire process can take a minute or so.
The Hotmail staff has invented a technique that uses a thin ISAPI filter (the ?shim?). It loads the application as a separate DLL and passes on all the ISAPI requests. It also watches for updates to the application DLL in a predetermined place, and when it is notified of an update it maps the new DLL, sends it all new requests, and allows the old requests to terminate before removing the old DLL. This technique has been made available to the IIS team.
Intellimirror
The team investigated, but decided not to use, Intellimirror-based update. First, Intellimorror requires AD to be implemented. Second, Intellimirror (working with the Installer) only makes updates to applications when a user logs in or when the system is rebooted. Since user login is an irrelevant activity in this context, and the whole idea is to prevent a reboot, Intellimirror-based update does not meet the need.
Distribution mechanism and format
The UNIX implementation packaged new code as a compressed file using the UNIX/tar/format, and distributed it (and the necessary installation code) using the UNIX/rdist/utility.
The team investigated use of MS Installer technology for a packaging format. Although it would probably have met the requirements (including the ability to unpack versioned files into specific locations, make registry changes, and run arbitrary code during installation) it proved too difficult to learn, despite the availability of a few decent authoring packages. The team stayed with the zipfile method of packaging.
The UNIX/rdist/ mechanism is also well suited to installation and updates on a large number of identical machines. From a central location, the administrator can iterate over a list of servers and push packages to them. The/rdist/daemon (service) running on the remote systems will extract files from the packages into their specified locations and run arbitrary commands before and after installation. This is approximately equivalent to MS Installer features, with the additional ability to push distributions over a list of machines. The Hotmail team implemented a version of the/rdist/ daemon to run on Windows.
Monitoring and Logging
Network Operations Center
The Hotmail infrastructure is monitored remotely, in an operations center located with the development staff in the Sunnyvale campus. There are many tools in place to monitor the performance of the server farm. Some of these measure the systems by their external behavior, and they did not need modification. Others use information gathered by the servers themselves (performance counters, disk statistics and so on). It proved to be relatively simple to write scripts that would extract the desired information from the Windows performance counters and send them to the Operations consoles.
Autonomous monitoring
Some of the self-test and monitoring features of the servers are performed by customized operations (usually scripts) executed at predetermined intervals. These intervals are anything between a minute and a week.
Using FreeBSD, such tasks are scheduled by the/cron/service. Jobs are scheduled by being listed in a file, one line per job. Changing the file is easy to accomplish using the command line (or/rdist/), and replacing the entire file is a good way to ensure that each server has exactly the schedule of jobs that the administrator intended. Jobs can be scheduled to execute once, or at intervals down to one minute.
Although the Windows Task Scheduler service is fundamentally able to look after such jobs, the interfaces provided in Windows does not measure up to the task.
The usual interface is the GUI, which is appropriate for setting up jobs on a machine at a time, is labor-intensive and error-prone.
The command/at/is deprecated, is not able to schedule repeated jobs at a frequency of less than one day.
The command/jt/was offered by the Task Scheduler team, but it is unsupported and awkward to use (it was intended for testing).
None of the three interfaces offers an easy way to replace the current task schedule entirely.
The team met the need by running the/cron/service provided in Services for UNIX. As described earlier, relying on Services for UNIX (or any other package subject to extra license costs) provides a bad model for other customer deployments. We have provided input to the Whistler command line team for an improved interface to Task Scheduler.
Logging
There was a minor issue concerning the UNIX integrated logging feature (/syslog/). The kernel, standard services, and application code can write lines of text to/syslog/, and a single configuration file is used to determine the destination of the text lines. Thus an important alert can result in a console message and email, while an informational message can be written to a log file. The administrator can change the destinations without code having to be recompiled.
An application like Hotmail often uses the application access to/syslog/to write statistical data of business interest (such as creation of a new account or sending of an email message). Administrators can use other tools to analyze the logs, archive them, or simply count occurrences and throw the logs away. Typical usage is at the order of one event per second; the high performance associated with the kernel log is not required.
There are no features in Windows 2000 that provide the same combination of convenience and configurability, although the kernel event log comes close. For convenience, and also to avoid recoding, the team elected to use the/syslog/feature from the Interix subsystem, introducing the issues about notional cost that have already been discussed.
Whistler introduces the Enterprise Event Log, a lightweight WMI feature, which seems to provide the desired functionality. A closer examination of the kernel logging may show that it too can meet the need, Any replacement should involve trivial change to the existing application code (perhaps even using a macro); it would be desirable not to have to recode calls to/syslog/ in order to keep down the amount of source code conversion.
Ad-hoc Maintenance
There are occasions when, after deployment, the administrators need to make a configuration change consistently across the entire farm. The/rdist/mechanism can be used for configuration changes; if the change is simple then/rsh/ can be used. The key fact about UNIX that makes this work is, again, that all system administration tasks can be done from the command line.
Windows should provide the same functionality, given some means of aggregating a group of servers and some way of performing an operation consistently across all the servers. Single commands, pipelines, or scripts (command scripts or COM-based scripts) would be appropriate actors; however, scripts need to be downloaded, executed (and, if necessary, cleaned up). There should be the ability to defer the activity until a specific time, presumably using the improved Task Scheduler. In other words, Windows must support old-fashioned batch processing.
One specific example of a feature that is not accessible to the batch model is Network Interface Card customization; for example, there have been requirements to change the card?s speed from 10 Mbps to 100Mbps (at a specific time) or to change the MTU setting. The configuration model of an Ethernet NIC varies between manufacturers, and the standard GUI is driven by a schema that is found in the registry. Such a GUI is not at all adaptable to the batch model. It is possible to make the required changes to the registry, but that would require a subsequent reboot, which is not acceptable. A brief period off the network, while the card resets itself, is the most downtime that can be accepted.
The Hotmail team, with help from a network engineer, developed a rudimentary application that would put a specific value in the registry and (using an undocumented interface) reset the card in a way that will make it pick up the new value. We strongly urge that the feature be put into the shipping system.
Each of several thousand systems must be converted to the new operating system and application suite, and this process must be carried out while the service is operating, and within a short timespan. Required are a mechanism for packaging the image and a method for delivering it. Among the special requirements:
Each server already has a name and static IP address; to fit in with existing operating practices and configurations, they should retain the same name and IP address. Using a static address, compared with DHCP, makes system administration simpler and more transparent. A machine?s name relates to its physical position within a cluster.
It should be possible to convert a machine without physical access.
It should be possible to revert systems quickly to FreeBSD in case of serious problems with the Windows conversion.
Downtime for reboots and service restarts should be minimized.
Several technologies were investigated and rejected. In most cases, there were blocking issues that were seemingly small, but without guarantee of resolution the team had to adopt a method that they could control. Some of the issues were:
RIS can be used for automatically installing an image from a server when a machine is initially booted. Drawbacks include: physical access is required to the machine (to force a network boot), and the system requires that an IP address be supplied with DHCP (DHCP is not used at Hotmail, because of the requirement for static IP addresses). It was impossible to control the name of the new server as required. In addition RIS was not supported for installing Server, although it was known to work.
AppCenter is intended for this kind of application. However, the initial release of AppCenter is targeted for small installations. It also lacks some features needed by application installation and update.
Unattended setup performs a standard installation across the network; because of all the file copying and calculation involved, it is too slow.
The team opted to extend an existing technology, ?kickstart?. This uses the OS existing on the machine to bootstrap an image, prepared using sysprep, and then run scripts to perform the remaining configuration tasks that need to be carried out after the install. The image copy is sufficiently fast, and the post-install steps are minimal.
IIS configuration
It proves to be difficult to configure IIS in a precisely controlled way. The metabase is obscure and poorly documented, and produced too many surprises. Furthermore, a system created using sysprep does not produce a ready-to-run metabase.
Consequently, it was necessary to construct the metabase by using scripts. The scripts were a mixture of command files that repeatedly call the/mdutil/utility, and some special-purpose pieces of scripting code (VBScript in this case, although any language that supports COM would work). The scripts are run as part of the mini-setup step that follows construction of the operating system on the target computer.
Figuring out the metabase structure, which elements needed to be set, and how to suppress the unwanted elements (for example, the trees defining the default and administration site) was the most complex and error-prone part of the entire setup design. Considerable reverse engineering was necessary. Major improvement is needed in the way the metabase is described to users, and the way that administrators can script the commonest tasks.
Tuning and hardening the system
The task was to tune the system for the best combination of throughput and performance, and also to harden it against attack from outside. This required attention in several areas:
System configuration, in removing all unnecessary system services and making sure the remaining services are configured as effectively as possible.
Registry settings for performance and security.
Metabase settings for performance and security.
The team was unable to find a comprehensive set of published settings that covered the above areas, perhaps because there are so many sets of demands on system configuration in general. However, we feel that configuring a system to be a locked down web server will be a common enough task that it would be useful to establish and publish a set of recommended actions and settings.
Use of Active Directory
Active Directory (AD) is a key addition in Windows 2000, yet it has been difficult to justify its use in the web server farm context.
Users in AD
AD is generally used to manage populations of users and machines. At Hotmail, it is not interesting to use AD to manage customers. User privileges and restrictions are already handled by the Hotmail application code, and there is no concept of granting or restricting access to customers within the Hotmail infrastructure. Furthermore, there is a constantly changing population of many usernames (over 100 million in July, 2000), a size that may be beyond the capabilities of Windows 2000.
The site has users in another sense: administrator accounts that are used to manage the machines by hand or by script. However, all administrators are fully trusted in the system (once they are inside the firewall), and it is normal to allow them to log in with full administrative privileges. This is the equivalent to the UNIX/root/account. It is useful to allow single sign-in, to allow an administrator to move from one machine to the next, and also to add new users at a central point; however, these needs are easily met by NT4?s NTLM.
Computer systems in AD
There is a stronger argument for entering the servers in AD. This will provide integration with DNS, and holds out the potential for administrators to classify machines in whatever ways they find useful operationally.
The Hotmail server farm is organized as a series of clusters, each containing several hundred servers. These machines must be named systematically. In practice, server names are duplicated between clusters, as they are identified uniquely by the fully qualified domain name (each cluster is a subdomain). This presents a problem for AD, which (apparently because of NetBIOS compatibility) does not permit duplicate short names anywhere within a set of subdomains. Getting rid of the NetBIOS legacy will be a great boon for Microsoft.
This apparently trivial restriction was enough to postpone the idea of constructing an AD, which in any case is additional work without obvious benefit. It was necessary to maintain the names of systems through the upgrade, because of legacy monitoring and administrative tools. Existing administrative mechanisms were adapted and did not need the benefits of AD. It is expected that, later, administrative staff will be able to develop tools that can make use of AD (for example, the ability to query on servers with a particular characteristic may be useful) but for now there is no need to break into the circle.
The Windows DNS service, operating without AD, proved perfectly capable of handling the load, and was able to take up the data from a UNIX BIND server easily. Windows DNS is used at the site for both internal and external name resolution.
Hotmail has a large investment in Cisco Local Director every web access goes to an LD, which redistributes the load among real servers. Hotmail chose to continue with LD, rather than use the Windows load balancing technology, because the infrastructure was in place and did not need to be reconfigured (reducing the learning curve). Also, LD fits the Hotmail model well; it is possible to place up to 400 servers behind the virtual address, and each Hotmail cluster can have over 300 identically configured servers.
Another major issue is the potential cost. Although Hotmail uses Microsoft software without license fees, we must consider this project as a model for real customers. Use of WLBS requires Advanced Server, but Server provides all the other features used by Hotmail. Using list prices, the cost comparison for a farm of 3500 servers is:
Using WLBS (hence Advanced Server): $15M+
Using LD and Server: $6M+
This does not take into account any extra PCs necessary to handle WLBS overhead (administrative, as well as the cycles needed to redirect the load) or the plans by Cisco to further reduce the cost of LD by building it into their network switches.
When considered in the context of a large web farm, WLBS has a serious economic disadvantage that can only be justified by the value of its administrative and monitoring tools. There is considerable competition in the IP load balancing market, which drives costs down; the numbers quoted above are based on the price we paid in mid-1999, around $17,000 per unit. An existing system that has load balancing in place will presumably have adequate tools, so the added value of WLBS, in terms of operational flexibility and superior monitoring, must be considerable if it is to be economically justified.
The constraints called out earlier (the 8-week upgrade cycle, the need to keep the service running, and the small number of staff) produced enough pressure on the development and administrative staff that the team agreed to devote one cycle to the platform conversion and not change the application during that time. This allowed the developers and testers to focus on the specific conversion issues. During the conversion, the application itself was the same on both platforms. This means that a user may have successive pages served by either platform, and not notice the difference.
The same constraints led to a desire not to change operational practices without good reason, because of the investment in training staff at all skill levels, and the feeling that the fewer things were changed, the fewer were the potential blocking problems.
Finally, the economic necessity of not adding technical staff to the conversion means that there was no consideration given to major re-architecture of the application.
Installation Methodology Conserved
There is in place a method of remotely bootstrapping a server to a new OS and application suite, and converting one rack (21 machines) in about 20 minutes. Replicating the installation capability was a goal of the project, and conserving as much as possible of the infrastructure to do it was strongly desired.
Conversion to ISAPI
The web server application suite consists of about 90 different transactions, each corresponding to a click on a web page. Using Apache, each one is implemented as an executable program using the CGI interface, and run in a separate process managed and owned by the web server. Processes are the natural way of encapsulating a single stateless transaction using UNIX.
Converting to Windows, the development team decided not to use the CGI interface to IIS. Creating a new Windows process is more expensive than creating a UNIX process. Instead, the team converted the CGI code to run as an ISAPI application, in which the transactions are processed by code that (in the most basic implementation) runs within the IIS process.
Running in process will be more efficient than running as a CGI, because the process creation overhead is avoided. We could have brought that advantage to UNIX. Apache supports the same concept; the equivalent to an ISAPI filter is called a module. Naturally, we did not waste time building the module implementation just to throw it away.
Conversion from CGI to ISAPI was essentially automated by using a filter that effectively presents the standard CG interface (using data streams and environment variables) to the user code. Because the application code was well written and did not make assumptions about its environment, the major part of the conversion went very smoothly and did not require significant unexpected engineering [4] . There were some intentional pieces of re-engineering:
The spell, dictionary, and thesaurus functions were rewritten to use Microsoft technology from Office and Encarta. The UNIX versions use binaries from Merriam Webster. The spellcheck feature is much improved; there are coverage problems with the dictionary data that need to be addressed.
The SMTP service of IIS was used to handle outgoing mail, replacing a UNIX standard mail service.
Virus scanning of attachments used an external UNIX utility from McAfee; this was replaced by its NT equivalent.
The most challenging, and anticipated, problem with converting from CGI to ISAPI derives from the forgiving nature of the CGI architecture. Memory leaks, unclosed files and similar problems can be tolerated, because they are automatically cleaned up when the CGI process terminates. Even an occasional abort is tolerated; it results in an invalid page to one customer, but does not usually affect any other part of the system.
By contrast, ISAPI modules share a process with the web server, as do Apache modules. Resource leaks will accumulate, and crashes have the potential to bring down the server (although not the entire service, thanks to load balancing). There are process isolation techniques available in IIS to minimize these problems, but the team decided to use the in-process model for full efficiency. Among the actions taken:
Use a private heap that is cleared at the end of each web transaction.
In testing, monitor for resource leaks and fix them.
Implement an IIS heartbeat monitor that will quickly notice and restart any failed IIS service.
Converting to ASP was not considered. That would have been a complete rewrite of the application, with no great advantage (Hotmail does not use a WinDNA infrastructure, for example). In fact, the implementation uses some ASP ideas and terms, as much of the user content is determined by template files that look like ASP files, but the interpretation engine is completely homegrown. One motivation for borrowing ASP syntax was to use Microsoft development tools (for example, to aid internationalization).
1) Windows has more resources behind its development. It does have greater complexity than the free UNIX distributions, and used wisely (and with knowledge) that can lead to a more effective solution. For example, IIS is more self-tuning than Apache.
IIS and Windows have many more tuning parameters than Apache and FreeBSD. The problem here is to make them comprehensible to new administrators.
2) The development platform, specifically Visual Studio, is a major advantage. Even before the conversion to Windows was contemplated, Hotmail developers used Visual Studio on NT4 to develop and debug their code. The code was eventually recompiled for UNIX when the first level of testing was complete. There is nothing approaching the power of Visual Studio on any UNIX, let alone the free ones, with the possible exception of the Java development tools.
The superior development platform has also had a positive operational impact in the live site. In the first days of deployment, some server threads went into a CPU-consuming loop. Using Visual Studio, Hotmail developers were able to find the application-level problem in a few minutes. That would have been impossible using UNIX tools.
3) Vastly better monitoring infrastructure. UNIX has some rudimentary event reporting and performance monitoring tools, but nothing to approach the integrated power of the event logging and performance monitoring features. Again, it is necessary to use them wisely; event logging in particular has a human and system overhead that we?ll talk about later.
4) Better hardware detection. Setting up UNIX on a new PC is difficult, requiring a more intimate knowledge of how the hardware is built. That?s an up-front cost; given the existence of multiple identically configured systems, cloning an established system doesn?t present the same problems.
5) Internationalization. The software tools available in Windows to provide multiple localized solutions are far ahead of most UNIX systems.
Slashdot Mirror
Don't understand why he didn't take and leak the relevant emails?
haven't they heard of soundex?
wake up masher sounds like you've drinking a bit too much of that constitutional kool-aid all "democracies" have been screwing over their citizens from day 1. this just didn't happen under Obama's watch and blaming him is weak cop out.
here is a good example that Chomsky uses at http://www.salon.com/2013/12/2...
"So, for example, if you go back a century ago, right after the U.S. invasion of the Philippines — a brutal invasion that killed a couple hundred thousand people — there was a problem for the U.S. of pacification afterwards. What do you do to control the population to prevent another nationalist uprising? There’s a very good study of this by Alfred McCoy, a Philippines scholar at University of Wisconsin, and what he shows is that the U.S. used the most sophisticated technology of the day to develop a massive system of survelliance, control, disruption to undermine any potential opposition and to impose very tight controls on the population which lasted for a long time and in many ways the Philippines is still suffering from this. But he also points out the technology was immediately transferred home. Woodrow Wilson’s administration used it in their “Red Scare” a couple years later. The British used it, too."
pick any period in history and these power hungry cunts have always been trying to stay in power no matter what. I haven't got any answers just know that what we call democracy is not working because it always allows the people in power to get away with shit.
I dunno if they were lying but your article says
"therefore useless to anyone not running Windows (and, as it turned out, Explorer as well)."
I'm using firefox 2.0 on xp and I can use virtual earth the only thing that doesn't seem work is the 3d view but the rest of it seems ok.
Poor article I don't think roblimo really went along with an open mind so he seems to have spent his tme nitpicking e.g. he states in his article that you need Internet Explorer to use virtual earth but it works fine in FireFox 2.0 .
To modify install.rdf do the following
1. Close Firefox
2. Open %APPDATA%\Mozilla\Firefox\Profiles\
3. Delete extensions.rdf
4. Go to the extensions folder.
5. Now you'll have to go to every folder there and edit its install.rdf file with a texteditor such as notepad.
6. You will see something like this:
CODEChange maxVersion to 1.4, save the install.rdf.
shouldn't that be a onepod or maybe a wepod
I'm fairly certain that if Open Office significantly eats into Office's market share that Microsoft will release a cut down version that is significantly cheaper or rmaybe even free. They've already done this with their cut down version of XP
A Short History of Nearly Everything by Bill Bryson was the book I enjoyed the most this year. He did make a few mistakes and he does gloss things over but it's an excellent read for anyone that wants to know about most of the major scientific advances in the last 300 years and the people that have made them. For me the real strength of the book is the way he brings these people to life with his anecdotes and the fact that he makes the very important point of how incredibly little we know.
so why did they register AppleUniversal.com a few days ago?
Domain Name: APPLEUNIVERSAL.COM
Registrar: BULKREGISTER.COM, INC.
Whois Server: whois.bulkregister.com
Referral URL: http://www.bulkregister.com
Name Server: NSERVER2.APPLE.COM
Name Server: NSERVER.APPLE.COM
Status: ACTIVE
Updated Date: 11-apr-2003
Creation Date: 11-apr-2003
Expiration Date: 11-apr-2004
NOTICE: The expiration date displayed in this record is the date the
registrar's sponsorship of the domain name registration in the registry is
currently set to expire. This date does not necessarily reflect the expiration
date of the domain name registrant's agreement with the sponsoring
registrar. Users may consult the sponsoring registrar's Whois database to
view the registrar's reported date of expiration for this registration.
that should be #cnnnews also you can get fox captions in #livenews
I would have thought it would be a bold and intelligent move to ask the Chinese to
make a contribution. Wouldn't you rather have the Chinese to be working with the ISS partners rather than competing against them.
I agree that Chinese spies on ISS are simply unacceptable but hasn't China signed all the various non-proliferation agreements and surely the international prestige such a participation would bring them would stop the Chinese doing anything stupid.
Their human rights record does bother me and it needs to improve but that has never stopped the US getting into bed with countries with similar or worse records.
Although China has announced that it's planning a permanently manned space station this seems like a waste of time, effort and money. I think it would make more sense to let China either take Russia's place or just let them join the ISS program. But I guess relations between the US and China need to improve before this could happen.
Hmm don't they insure these satellites against any resulting financial loss? Surely this would negate the need need for any reduncies. But on the other hand their premiums are probably going to shoot through the roof so maybe....
Unfortunately a lot of people don't actually read the EULA. They just click through until the software is installed. Even if you do read it it's full of dense obscure legal language that mostly doesn't apply to you. Advertising software if implemented correctly can allow developers to make money from their software without requiring the end user to pay.
The problem is it's often not done properly. There are spyware apps like aureate that operate in stealth mode by passing themselves off as Windows system processes and making sure that they don't even show up the task list or binding themselves to winsock so that you delete or uninstall them your Internet connection stops working. Microsoft should be made to fix these holes in IE but I think some pressure should also be applied to the people that write these programs.
that there is life on Earth, but we don't know if there is any elsewhere in the Universe.
The four steps to necessary for Abiogenesis are:
Inorganic Molecules to Organic Monomers
Organic Monomers to Organic Polymers
Formation of membranes from the polymers
Acquisition of a means of reproduction
Maybe the asteroids instead of seeding the earth provided the energy required for the first step.
Finally the truth is out!. NASA Fakes Moon Landing
This approach would only work in the short term because there are already file hashing applications like sig2dat that help p2p users share exact copies of files that they have verified as good.
Conclusions
/rsh /command, or pushing configuration files
These are the main lessons that we can extract from the Hotmail conversion.
1) We need a consistent, comprehensive, thoughtful approach to
integrated management of a set of servers. This does not necessarily
mean that we should slavishly follow the UNIX model of iterating through
a list of machines with an
to a list of machines. The fundamental goal is to be able to manage
machines as an aggregate; doing this through a GUI is not necessarily
evil, so long as it can be done remotely, and once. The point applies to
application distribution as well as to system tuning.
2) NLBS is at an economic disadvantage, due to its association with
Advanced Server, and Hotmail operations staff were sufficiently
satisfied with the existing solution that they did not feel the need to
investigate NLBS?s operational advantages.
3) The metabase needs to be ripped out and replaced with something
that is much easier for an administrator to see and understand, and be
confident that there are no hidden surprises. The IIS6 planners have
heard this opinion.
4) It should be easier to tune and lock down a single system, and
have the changes propagated to all systems in a given class.
5) Windows is too complex to understand at first, particularly
during a conversion from UNIX. There are just too many things about it
for a planner in a startup to understand. Typically there is little time
to attend training. The problem is most Computer Science graduates come
to their startups already understanding enough about UNIX to be
confident that they can use it effectively. We do need to be careful to
balance the complexity and transparency carefully.
6) The basic need for an Internet site, converting from UNIX to
Windows, is to be able to quickly replace their application and
operational methodology with something at least equally good.
Improvements that come for free are good, but implementing new
technologies and programming methods will need to take a back seat so
long as they delay the main purpose, which is to keep a site online and
competitive. Anything else is a cost that needs to bring a clear benefit.
Converting the UNIX Administrator
Helping UNIX system administrators with the transition to Windows is an
experience in itself, and much was learned. Again, this is data from a
single corporate experience, but we suspect it is fairly typical. Here,
then, is the human engineering overview.
Initially, the plan to convert from FreeBSD to Windows was met with
responses ranging from skepticism to hostility, in a way that should be
familiar to those who share the attitudes of the various UNIX
communities to Microsoft software.
We engaged with the operations staff by asking them to define what their
everyday tasks are, in all areas of operating system and application
maintenance. Instead of a set of tasks, we were handed a set of the UNIX
commands and features that were used to carry out those tasks. While
this did not directly meet the need, it gave us an opportunity to
address all of the features directly, and show that Windows has an exact
equivalent in the core system, or in the Resource Kit, or easily
provided with a script. There were very few cases where no satisfactory
alternative could be found. Essentially, this was throwaway work, as the
eventual solution solved the problems in a more Windows-like way, but it
was an excellent opportunity to gain the confidence of the operations staff.
It was clear from the responses that some people from the UNIX side of
the house cannot distinguish our different systems that are marketed
under the Windows brand; there was an inbuilt assumption that Windows
2000 shares the features and faults of Windows 95. Those who were
somewhat familiar with Windows NT were not aware of the range of the
non-GUI offerings (to be fair, neither were we); the set of commands in
the product and the Resource Kit is fairly broad although, as we have
seen, there are gaps and they lack stylistic consistency.
Other staff members, not members of the regular operations team, carried
out the conversion. When deployment came near, the Operations staff had
to learn the new tools and paradigms. Their existence proved enough; the
main interest of Operations staff is, after all, to run and administer
the system, and once they found that there were tools, whether
custom-built or standard, that did the job well enough, they were able
to take control and gain a sense of ownership. Some standard one-day
courses were also given to the staff, to prepare them for handling
system debugging, hotfix application and so on. By this time, the staff
had become convinced that Windows is, after all, a real operating system
with surprising richness.
Application update styles
It is naturally a requirement that a web-based service operate
continually, without customer-visible degradations of service. This is
not just a matter of pride; even a loss of availability for a few
minutes every month can produce too much degradation in the perceptions
and (assuming we publish uptime numbers) the availability measurement.
It?s a solution, but a weak one, to put servers behind load-balancing
equipment and take them out of service when required for upgrade or
other maintenance. The challenge is to keep each server running
continuously as much as possible. Except for operating system upgrades,
a system based on FreeBSD and Apache can keep operating while the
application is upgraded, and Windows should be able to do the same.
Application updates at Hotmail are of two kinds: content and code.
Content updates change only data files, generally those that directly
determine what the customer sees on the screen, and they are carried out
on their own schedule. Apache can handle both content and code updates
without stopping the service. Updates can be rolled out directly, when
the data is updated in place. They can also be timed, when the updates
are put on the servers in a staging location together with an update
batch job that will be triggered at the desired moment. The timed update
is used when it is important for the application?s integrity that the
entire site be updated simultaneously, something that is impossible to
achieve when updating several thousand servers across a single network.
Application update techniques
Apache running under UNIX supports both kinds of updates very simply. A
CGI application can be replaced, even while the old file is being
executed, and the next execution will use the new file. The same is true
of content. If Apache?s own configuration files must be updated, there
is a procedure to signal the server to reset itself and reread its
configuration, and that takes around a second.
Unfortunately, IIS 5.0 does not support either kind of update well. When
IIS accesses content directly, it locks the folders. Fortunately, this
doesn?t apply to most of the Hotmail upgrades. The bigger issue is
updating the ISAPI filters, which must be done while the IIS server is
stopped. The entire process can take a minute or so.
The Hotmail staff has invented a technique that uses a thin ISAPI filter
(the ?shim?). It loads the application as a separate DLL and passes on
all the ISAPI requests. It also watches for updates to the application
DLL in a predetermined place, and when it is notified of an update it
maps the new DLL, sends it all new requests, and allows the old requests
to terminate before removing the old DLL. This technique has been made
available to the IIS team.
Intellimirror
The team investigated, but decided not to use, Intellimirror-based
update. First, Intellimorror requires AD to be implemented. Second,
Intellimirror (working with the Installer) only makes updates to
applications when a user logs in or when the system is rebooted. Since
user login is an irrelevant activity in this context, and the whole idea
is to prevent a reboot, Intellimirror-based update does not meet the need.
Distribution mechanism and format
The UNIX implementation packaged new code as a compressed file using the
UNIX
code) using the UNIX
The team investigated use of MS Installer technology for a packaging
format. Although it would probably have met the requirements (including
the ability to unpack versioned files into specific locations, make
registry changes, and run arbitrary code during installation) it proved
too difficult to learn, despite the availability of a few decent
authoring packages. The team stayed with the zipfile method of packaging.
The UNIX
updates on a large number of identical machines. From a central
location, the administrator can iterate over a list of servers and push
packages to them. The
systems will extract files from the packages into their specified
locations and run arbitrary commands before and after installation. This
is approximately equivalent to MS Installer features, with the
additional ability to push distributions over a list of machines. The
Hotmail team implemented a version of the
Monitoring and Logging
Network Operations Center
The Hotmail infrastructure is monitored remotely, in an operations
center located with the development staff in the Sunnyvale campus. There
are many tools in place to monitor the performance of the server farm.
Some of these measure the systems by their external behavior, and they
did not need modification. Others use information gathered by the
servers themselves (performance counters, disk statistics and so on). It
proved to be relatively simple to write scripts that would extract the
desired information from the Windows performance counters and send them
to the Operations consoles.
Autonomous monitoring
Some of the self-test and monitoring features of the servers are
performed by customized operations (usually scripts) executed at
predetermined intervals. These intervals are anything between a minute
and a week.
Using FreeBSD, such tasks are scheduled by the
scheduled by being listed in a file, one line per job. Changing the file
is easy to accomplish using the command line (or
the entire file is a good way to ensure that each server has exactly the
schedule of jobs that the administrator intended. Jobs can be scheduled
to execute once, or at intervals down to one minute.
Although the Windows Task Scheduler service is fundamentally able to
look after such jobs, the interfaces provided in Windows does not
measure up to the task.
The usual interface is the GUI, which is appropriate for
setting up jobs on a machine at a time, is labor-intensive and error-prone.
The command /at /is deprecated, is not able to schedule
repeated jobs at a frequency of less than one day.
The command /jt /was offered by the Task Scheduler team, but it
is unsupported and awkward to use (it was intended for testing).
None of the three interfaces offers an easy way to replace the
/cron /service provided in Services
/syslog/, and a single configuration file is used
/syslog /to write statistical data of business interest (such as creation of a
/syslog /feature from the Interix subsystem, introducing the
/syslog/ in order to keep down the amount of source code
/rdist /mechanism can be used for configuration changes; if the change /rsh/ can be used. The key fact about UNIX that makes
current task schedule entirely.
The team met the need by running the
for UNIX. As described earlier, relying on Services for UNIX (or any
other package subject to extra license costs) provides a bad model for
other customer deployments. We have provided input to the Whistler
command line team for an improved interface to Task Scheduler.
Logging
There was a minor issue concerning the UNIX integrated logging feature
(/syslog/). The kernel, standard services, and application code can
write lines of text to
to determine the destination of the text lines. Thus an important alert
can result in a console message and email, while an informational
message can be written to a log file. The administrator can change the
destinations without code having to be recompiled.
An application like Hotmail often uses the application access to
new account or sending of an email message). Administrators can use
other tools to analyze the logs, archive them, or simply count
occurrences and throw the logs away. Typical usage is at the order of
one event per second; the high performance associated with the kernel
log is not required.
There are no features in Windows 2000 that provide the same combination
of convenience and configurability, although the kernel event log comes
close. For convenience, and also to avoid recoding, the team elected to
use the
issues about notional cost that have already been discussed.
Whistler introduces the Enterprise Event Log, a lightweight WMI feature,
which seems to provide the desired functionality. A closer examination
of the kernel logging may show that it too can meet the need, Any
replacement should involve trivial change to the existing application
code (perhaps even using a macro); it would be desirable not to have to
recode calls to
conversion.
Ad-hoc Maintenance
There are occasions when, after deployment, the administrators need to
make a configuration change consistently across the entire farm. The
is simple then
this work is, again, that all system administration tasks can be done
from the command line.
Windows should provide the same functionality, given some means of
aggregating a group of servers and some way of performing an operation
consistently across all the servers. Single commands, pipelines, or
scripts (command scripts or COM-based scripts) would be appropriate
actors; however, scripts need to be downloaded, executed (and, if
necessary, cleaned up). There should be the ability to defer the
activity until a specific time, presumably using the improved Task
Scheduler. In other words, Windows must support old-fashioned batch
processing.
One specific example of a feature that is not accessible to the batch
model is Network Interface Card customization; for example, there have
been requirements to change the card?s speed from 10 Mbps to 100Mbps (at
a specific time) or to change the MTU setting. The configuration model
of an Ethernet NIC varies between manufacturers, and the standard GUI is
driven by a schema that is found in the registry. Such a GUI is not at
all adaptable to the batch model. It is possible to make the required
changes to the registry, but that would require a subsequent reboot,
which is not acceptable. A brief period off the network, while the card
resets itself, is the most downtime that can be accepted.
The Hotmail team, with help from a network engineer, developed a
rudimentary application that would put a specific value in the registry
and (using an undocumented interface) reset the card in a way that will
make it pick up the new value. We strongly urge that the feature be put
into the shipping system.
OS installation and configuration
Each of several thousand systems must be converted to the new operating system and application suite, and this process must be carried out while
the service is operating, and within a short timespan. Required are a mechanism for packaging the image and a method for delivering it. Among
the special requirements:
Each server already has a name and static IP address; to fit in with existing operating practices and configurations, they should retain
the same name and IP address. Using a static address, compared with DHCP, makes system administration simpler and more transparent. A
machine?s name relates to its physical position within a cluster.
It should be possible to convert a machine without physical access.
It should be possible to revert systems quickly to FreeBSD in case of serious problems with the Windows conversion.
Downtime for reboots and service restarts should be minimized.
Several technologies were investigated and rejected. In most cases, there were blocking issues that were seemingly small, but without
guarantee of resolution the team had to adopt a method that they could control. Some of the issues were:
RIS can be used for automatically installing an image from a server when a machine is initially booted. Drawbacks include: physical
access is required to the machine (to force a network boot), and the system requires that an IP address be supplied with DHCP (DHCP is not
used at Hotmail, because of the requirement for static IP addresses). It was impossible to control the name of the new server as required. In
addition RIS was not supported for installing Server, although it was known to work.
AppCenter is intended for this kind of application. However, the initial release of AppCenter is targeted for small installations. It
also lacks some features needed by application installation and update.
Unattended setup performs a standard installation across the network; because of all the file copying and calculation involved, it is
/mdutil /utility, and some special-purpose pieces of scripting
too slow.
The team opted to extend an existing technology, ?kickstart?. This uses the OS existing on the machine to bootstrap an image, prepared using
sysprep, and then run scripts to perform the remaining configuration tasks that need to be carried out after the install. The image copy is
sufficiently fast, and the post-install steps are minimal.
IIS configuration
It proves to be difficult to configure IIS in a precisely controlled way. The metabase is obscure and poorly documented, and produced too
many surprises. Furthermore, a system created using sysprep does not
produce a ready-to-run metabase.
Consequently, it was necessary to construct the metabase by using
scripts. The scripts were a mixture of command files that repeatedly
call the
code (VBScript in this case, although any language that supports COM
would work). The scripts are run as part of the mini-setup step that
follows construction of the operating system on the target computer.
Figuring out the metabase structure, which elements needed to be set,
and how to suppress the unwanted elements (for example, the trees
defining the default and administration site) was the most complex and
error-prone part of the entire setup design. Considerable reverse
engineering was necessary. Major improvement is needed in the way the
metabase is described to users, and the way that administrators can
script the commonest tasks.
Tuning and hardening the system
The task was to tune the system for the best combination of throughput
and performance, and also to harden it against attack from outside. This
required attention in several areas:
System configuration, in removing all unnecessary system
services and making sure the remaining services are configured as
effectively as possible.
Registry settings for performance and security.
Metabase settings for performance and security.
/root /account. It is useful to allow single sign-in, to allow an
The team was unable to find a comprehensive set of published settings
that covered the above areas, perhaps because there are so many sets of
demands on system configuration in general. However, we feel that
configuring a system to be a locked down web server will be a common
enough task that it would be useful to establish and publish a set of
recommended actions and settings.
Use of Active Directory
Active Directory (AD) is a key addition in Windows 2000, yet it has been
difficult to justify its use in the web server farm context.
Users in AD
AD is generally used to manage populations of users and machines. At
Hotmail, it is not interesting to use AD to manage customers. User
privileges and restrictions are already handled by the Hotmail
application code, and there is no concept of granting or restricting
access to customers within the Hotmail infrastructure. Furthermore,
there is a constantly changing population of many usernames (over 100
million in July, 2000), a size that may be beyond the capabilities of
Windows 2000.
The site has users in another sense: administrator accounts that are
used to manage the machines by hand or by script. However, all
administrators are fully trusted in the system (once they are inside the
firewall), and it is normal to allow them to log in with full
administrative privileges. This is the equivalent to the UNIX
administrator to move from one machine to the next, and also to add new
users at a central point; however, these needs are easily met by NT4?s NTLM.
Computer systems in AD
There is a stronger argument for entering the servers in AD. This will
provide integration with DNS, and holds out the potential for
administrators to classify machines in whatever ways they find useful
operationally.
The Hotmail server farm is organized as a series of clusters, each
containing several hundred servers. These machines must be named
systematically. In practice, server names are duplicated between
clusters, as they are identified uniquely by the fully qualified domain
name (each cluster is a subdomain). This presents a problem for AD,
which (apparently because of NetBIOS compatibility) does not permit
duplicate short names anywhere within a set of subdomains. Getting rid
of the NetBIOS legacy will be a great boon for Microsoft.
This apparently trivial restriction was enough to postpone the idea of
constructing an AD, which in any case is additional work without obvious
benefit. It was necessary to maintain the names of systems through the
upgrade, because of legacy monitoring and administrative tools. Existing
administrative mechanisms were adapted and did not need the benefits of
AD. It is expected that, later, administrative staff will be able to
develop tools that can make use of AD (for example, the ability to query
on servers with a particular characteristic may be useful) but for now
there is no need to break into the circle.
The Windows DNS service, operating without AD, proved perfectly capable
of handling the load, and was able to take up the data from a UNIX BIND
server easily. Windows DNS is used at the site for both internal and
external name resolution.
Hotmail has a large investment in Cisco Local Director every web access
goes to an LD, which redistributes the load among real servers. Hotmail
chose to continue with LD, rather than use the Windows load balancing
technology, because the infrastructure was in place and did not need to
be reconfigured (reducing the learning curve). Also, LD fits the Hotmail
model well; it is possible to place up to 400 servers behind the virtual
address, and each Hotmail cluster can have over 300 identically
configured servers.
Another major issue is the potential cost. Although Hotmail uses
Microsoft software without license fees, we must consider this project
as a model for real customers. Use of WLBS requires Advanced Server, but
Server provides all the other features used by Hotmail. Using list
prices, the cost comparison for a farm of 3500 servers is:
Using WLBS (hence Advanced Server): $15M+
Using LD and Server: $6M+
This does not take into account any extra PCs necessary to handle WLBS
overhead (administrative, as well as the cycles needed to redirect the
load) or the plans by Cisco to further reduce the cost of LD by building
it into their network switches.
When considered in the context of a large web farm, WLBS has a serious
economic disadvantage that can only be justified by the value of its
administrative and monitoring tools. There is considerable competition
in the IP load balancing market, which drives costs down; the numbers
quoted above are based on the price we paid in mid-1999, around $17,000
per unit. An existing system that has load balancing in place will
presumably have adequate tools, so the added value of WLBS, in terms of
operational flexibility and superior monitoring, must be considerable if
it is to be economically justified.
Project constraints
The constraints called out earlier (the 8-week upgrade cycle, the need
to keep the service running, and the small number of staff) produced
enough pressure on the development and administrative staff that the
team agreed to devote one cycle to the platform conversion and not
change the application during that time. This allowed the developers and
testers to focus on the specific conversion issues. During the
conversion, the application itself was the same on both platforms. This
means that a user may have successive pages served by either platform,
and not notice the difference.
The same constraints led to a desire not to change operational practices
without good reason, because of the investment in training staff at all
skill levels, and the feeling that the fewer things were changed, the
fewer were the potential blocking problems.
Finally, the economic necessity of not adding technical staff to the
conversion means that there was no consideration given to major
re-architecture of the application.
Installation Methodology Conserved
There is in place a method of remotely bootstrapping a server to a new
OS and application suite, and converting one rack (21 machines) in about
20 minutes. Replicating the installation capability was a goal of the
project, and conserving as much as possible of the infrastructure to do
it was strongly desired.
Conversion to ISAPI
The web server application suite consists of about 90 different
transactions, each corresponding to a click on a web page. Using Apache,
each one is implemented as an executable program using the CGI
interface, and run in a separate process managed and owned by the web
server. Processes are the natural way of encapsulating a single
stateless transaction using UNIX.
Converting to Windows, the development team decided not to use the CGI
interface to IIS. Creating a new Windows process is more expensive than
creating a UNIX process. Instead, the team converted the CGI code to run
as an ISAPI application, in which the transactions are processed by code
that (in the most basic implementation) runs within the IIS process.
Running in process will be more efficient than running as a CGI, because
the process creation overhead is avoided. We could have brought that
advantage to UNIX. Apache supports the same concept; the equivalent to
an ISAPI filter is called a module. Naturally, we did not waste time
building the module implementation just to throw it away.
Conversion from CGI to ISAPI was essentially automated by using a filter
that effectively presents the standard CG interface (using data streams
and environment variables) to the user code. Because the application
code was well written and did not make assumptions about its
environment, the major part of the conversion went very smoothly and did
not require significant unexpected engineering [4] . There were
some intentional pieces of re-engineering:
The spell, dictionary, and thesaurus functions were rewritten
to use Microsoft technology from Office and Encarta. The UNIX versions
use binaries from Merriam Webster. The spellcheck feature is much
improved; there are coverage problems with the dictionary data that need
to be addressed.
The SMTP service of IIS was used to handle outgoing mail,
replacing a UNIX standard mail service.
Virus scanning of attachments used an external UNIX utility
from McAfee; this was replaced by its NT equivalent.
The most challenging, and anticipated, problem with converting from CGI
to ISAPI derives from the forgiving nature of the CGI architecture.
Memory leaks, unclosed files and similar problems can be tolerated,
because they are automatically cleaned up when the CGI process
terminates. Even an occasional abort is tolerated; it results in an
invalid page to one customer, but does not usually affect any other part
of the system.
By contrast, ISAPI modules share a process with the web server, as do
Apache modules. Resource leaks will accumulate, and crashes have the
potential to bring down the server (although not the entire service,
thanks to load balancing). There are process isolation techniques
available in IIS to minimize these problems, but the team decided to use
the in-process model for full efficiency. Among the actions taken:
Use a private heap that is cleared at the end of each web
transaction.
In testing, monitor for resource leaks and fix them.
Implement an IIS heartbeat monitor that will quickly notice and
restart any failed IIS service.
Converting to ASP was not considered. That would have been a complete
rewrite of the application, with no great advantage (Hotmail does not
use a WinDNA infrastructure, for example). In fact, the implementation
uses some ASP ideas and terms, as much of the user content is determined
by template files that look like ASP files, but the interpretation
engine is completely homegrown. One motivation for borrowing ASP syntax
was to use Microsoft development tools (for example, to aid
internationalization).
Strengths of Windows
1) Windows has more resources behind its development. It does have greater complexity than the free UNIX distributions, and used wisely
(and with knowledge) that can lead to a more effective solution. For example, IIS is more self-tuning than Apache.
IIS and Windows have many more tuning parameters than Apache and FreeBSD. The problem here is to make them comprehensible to new administrators.
2) The development platform, specifically Visual Studio, is a major advantage. Even before the conversion to Windows was contemplated, Hotmail developers used Visual Studio on NT4 to develop and debug their code. The code was eventually recompiled for UNIX when the first level of testing was complete. There is nothing approaching the power of Visual Studio on any UNIX, let alone the free ones, with the possible
exception of the Java development tools.
The superior development platform has also had a positive operational impact in the live site. In the first days of deployment, some server
threads went into a CPU-consuming loop. Using Visual Studio, Hotmail developers were able to find the application-level problem in a few
minutes. That would have been impossible using UNIX tools.
3) Vastly better monitoring infrastructure. UNIX has some rudimentary event reporting and performance monitoring tools, but nothing to approach the integrated power of the event logging and performance monitoring features. Again, it is necessary to use them wisely; event logging in particular has a human and system overhead that we?ll talk about later.
4) Better hardware detection. Setting up UNIX on a new PC is difficult, requiring a more intimate knowledge of how the hardware is
built. That?s an up-front cost; given the existence of multiple identically configured systems, cloning an established system doesn?t
present the same problems.
5) Internationalization. The software tools available in Windows to provide multiple localized solutions are far ahead of most UNIX systems.