I haven't seen Parasoft's stuff in years, but it was good when I last looked. Nice that it runs on Linux.
So.. is this Saber-C stuff Open Source/Free yet?
Nope, but it doesn't matter because it needed a rewrite anyway. Were I to do it today I'd move the instrumentation code into gcc, augment glib, and add some hooks to gdb to allow programmatic control (ie allow the inferior to ask the debugger to do things like stop). That would all be easier than most of our ports of Saber-C to new platforms, and would end up being largely platform independent.
Getting module swapping/incremental loading to work is a royal pain. Valuable, for sure, but maybe not worth the development agony. It's a lot less of an issue with demand loaded languages like C# and Java anyway.
Assuming you know the pre- and post-conditions (you don't always) then assertions can be a very valuable debugging technique, I agree.
But the real fallacy with "write-time debugging", and in fact all of the code-bug-free methodologies, is that there is very significant amounts of code that you must use to write your program that was written by someone else who is almost certainly not using your methodology (or, often, any methodology). So, even if your methodology worked on your own code it wouldn't work on the application as a whole.
I note that some of the most insidious bugs I've ever had to track down were caused by operating system errors. My favorite bug ever was under AIX where a call to select() with a 30-second timeout was returning in about 900 nanoseconds saying it had timed out. This caused a chain of failures. And, of course, it would only happen in some pretty unpredictable situations. In this case you needed two programs running under ptrace at the same time on the system for the bug to show itself - and it would only occur about one in five thousand calls to select() even under the failure conditions.
No amount of programmer effort on our part would have prevented this bug from biting us. We could certainly have put in a postcondition tester to check to see if the system call operated correctly (in fact that's how we worked around this problem) but can you realistically do that for every system call you ever make? Every library call you ever make? It's not practical to do so.
I'm all for doing things to minimize code errors, although to be completely honest the best method (as opposed to methodology) I've found to do that is to use languages that include their own run-time error checking, which are strongly typed, have mandatory exception handling, and which use garbage collection for heap management (Java being the only one I've used). The combination of such features leads to about 90% fewer bugs out-the-door and boosts programmer productivity by 200-400% over the long term versus something like C++ (that's data from several large programming projects I've been on).
It's that good because the environment tends to report problems as early as possible in the development process, when they are easiest to fix, and because of mandatory exception handling code more often gets written with failure handling in mind.
Even so, and even with methodologies that reduce bug rates even more, tools are often invaluable to producing good code. I find that heap inspection and profiling tools are very valuable for Java, for instance, as are techniques such as log instrumentation and assertion. (Unfortunately Java provides no facilities for any of this in-the-box, sigh.)
Anyway, to maximize programmer productivity and minimize bugs it pays to use an array of techniques and tools. Those who believe debuggers don't have their place, that they can be rendered pointless by technique, are badly mistaken.
Then you haven't written enough tests. Or you're trying to integrate too much at once.
Must be nice to work in a world where you have time to write comprehensive supra-unit tests and can rely on all your co-workers to have fully tested their code.
I'd like to visit that world sometime, because in my world schedules are allocated such that it's rare you have time to do more than just unit testing and they call integrated testing "beta".
I'm only half joking. Then again, our problem domain scale is probably offset by a couple of orders of magnitude and that will certainly make a difference.
Since changing from C to Java several years ago I find that I never use debuggers anymore
How much of that is because you wouldn't find it useful, versus the fact that there really haven't been any good Java debuggers?
With program restarts in our Java applications into the ten-plus minute range, a debugger would be a serious time saver versus adding log or println entries.
I note that with JDK 1.4 the JVM debug interfaces are finally good enough to actually make a working debugger, which will help things a lot.
Back in the early 90s there was a group at Boeing whose entire job was trying to find this one bug in their avionics code. It was nasty and subtle enough that they hadn't been able to find it for years.
They got a demo copy of ObjectCenter (formerly Saber-C), a superdebugger I used to work on which performed a lot of runtime error checking. They loaded the code, ran it, and it pointed out the error for them. Their job finished, the group was disbanded.
I don't know if we got the product sale:-) but certainly they shared your opinion. I do, too, although the near total lack of functional debuggers in Java (until 1.4, which unfortunately I still can't use) has kept me in printlns. Thank God that Java incorporates runtime checking and dumps stack traces on error or we'd never get anything written.
More cooperation with the code itself. I'd like the debugger to provide a library against which I can link my code so that the code being debugged can control things like which variables are being watched and which breakpoints are active.
In a different posting I talked about Saber-C and some of what we did that was useful, but I totally forgot about this particular feature. We did have the ability to link in debugger control functions like "break now". It was, indeed, quite useful.
Alot of what your product does really reminds me of Lisp
When I interviewed one of my comments was, "Hey, that's like a Lisp environment for C!" I had been building a lisp-based shell at the time so that was really cool:-).
Well, it just so happens that there have been a few debuggers much more functional than the likes of Visual Studio.
I used to work for a company called Saber, who later changed their name to Centerline. We produced a product called Saber-C (and later Saber-C++, and the products were renamed CodeCenter and ObjectCenter).
This product was a C/C++ superdebugger. It started out as a C interpreter developed at Harvard. Now, an interpretive environment allowed a lot of neat features: You could tell whether or not variables had been initialized at reference time, you had complete stack and context information, it was pretty easy to unload and reload code in source modules (dynamic reloading). You also got dynamic type checking, which caught errors in downcasting (for instance). And everyone's favorite, array bounds checking.
With this tool debugging C code often meant: Loading the code, running your test case, and seeing where the debugger complained. The first time you ran your code through it was a humbling experience... almost all production code has serious (but normally benign) errors.
Unfortunately the tool didn't scale well because it took time to load the interpreted code into the debugger and the heap and runtime requirements of interpreters stressed machines of the time (this was when Sun3s were common, and 3/50s couldn't even have more than 4MB memory).
I think it was 1990 when we released version 3, whose big new feature was object code debugging (it's not really that easy to have object code debugging as well as interpretive code, although we could certainly have implemented it more easily than we did). We of course maintained the ability to dynamically reload code fragments, so you could do a "make", reload the affected modules, and be on your way in a few seconds. This sure beat the several minute wait you normally had with the linker and debugger restart. But what you lost was some of the runtime checks -- including most of what made the product interesting. Still, the ability to have superdebugger capabilities on some code and still scale to large programs was very valuable.
About a year, maybe two years later we got visited by the people who eventually started Purify. They had a neat approach where they'd get most of the benefit of our tool, but in object code, by disassembling the object code, adding extra code to perform checks, and putting it back together. This would give about 80% of the functionality of our debugger but with much lower runtime costs and work even if you didn't have source code.
We ended up re-implementing this functionality such that we had a Purify competitor (and getting sued, and losing, even though Sun later patented the same technique we used).[1] Integrating it into the debugger allowed you to fine-tune the debugging capability you wanted on a module-by-module basis. This was a superb tool.
Now, what are the raw capabilities that made this stuff really neat?
Incremental reload of modules. That removes the debugger restart cost (symbol loading is a huge time sink).
Supply worst-case scenarios for code. In our tool, we'd initialize heap and stack memory to 0xbfbfbfbf patterns, which tended to trigger bugs in code that had intitialization errors.
Track heap allocations. Knowing who allocated something is remarkably valuable, particularly for tracking down memory leaks.
Augment the code with error checks that are sensible for the source language. For instance, we knew the type that was assigned to memory so we used that to detect all manner of type mismatches.
Augment the code with bounds checking for arrays. The bounds checkers worked with the heap system, which of course tracked the size of all allocations. This caught a lot of off-by-one errors.
The tool also provided two additional interesting features. One was called "action points". Since we had a C interpreter in the product it was no big deal to allow you to type C code into it and run it at arbitrary times. We'd allow you to attach any code you wanted to any line in the program or to just type code at the debug prompt and it would run in the current scope. This was useful for things like adding print statements without recompiling, for instance, and for adding assertions, and many other things.
The other feature was a graphical heap inspector. All this did was display structures as tables of their internal values and allow traversal of references to other structures, which would be displayed (including links between the structures). This made the shape of data a LOT easier to determine.
Several people talked about adding reverse execution. That was actually done for Java by a tools company back in 1996 (maybe 1997). It didn't always work, but it was pretty neat. The problem is that the recording code was pretty expensive and there's a limit to people's patience and what they'll spend on hardware (although I grant that hardware is a lot less of an issue today than it was even in 1997, to say nothing of 1989). Anyway this feature is a lot less interesting if your debugger can find errors at the point they occur.
Anyway, back in 1995 when I started to work with Visual-C++ on a regular basis I was appalled by what Microsoft was calling a "state of the art" debugger. Feh! It was 1985 technology with a pretty face. It really has not improved much since then either, although they do at least have a pessimistic heap allocator now. With their ability to hook in with the compiler they could do a phenomenal job with heap allocation tracking and instrumentation of code with error checks but they don't bother (because, I suppose, they make their money even without it).
Now, using a language with runtime error checking built in - such as Java, C#, or Visual Basic, seriously undermines the amount of effort you have to put into the debugger. Still, I'd like to see a lot better heap debugging features built into compilers and runtime environments.
Hope this gives people some ideas, and that they lead to tools I can use.
[1] One thing that could be done to avoid the Purify patents would be to have the compiler emit the instrumentation. For the OSS world, where you always have the source code, that is eminently practical. Furthermore you can get all the benefits of an interpreter but at higher performance. A big problem with Purify's technology, and the technology we came up with, was that you're doing a lot of heuristics during code inspection. It's easy to miss code or misinterpret code, and of course the code inspector is VERY architecture and compiler dependent. What I would like to see is a gcc/glibc/gdb combination where the compiler, libraries, and debugger all worked together. That wasn't possible back when the vendor controlled the compiler and you didn't have access to library source. Today it is, at least for the OSS world.
If it's that few then you could easily get by with only one admin, keeping in mind that he'll have to sleep and go on vacation on occasion. That's a pretty small site though.
I think the hundred-million dynamic pageviews site had three admins, but they switched hats with other jobs. One did double duty as the group manager, and the other two were part-time programmers. Multiple admins also meant that there was the possibility of time off:-).
They had a lot of outside help, though, since Exodus was hosting their machines for them and there was another IT department that did desktop management for the rest of the organization.
One person can do the maintenance of the main servers with only part-time effort, although generally such operations are well staffed for other reasons. The online servers are only the tip of the iceberg in such an operation -- you also have the database(s) with its associated guru, staging system(s), some number of developers, artists, etc. each with one or more systems, and of course the network infrastructure for such a system is very substantial.
Keep in mind that with that kind of horsepower you're talking about a pretty darn large site -- like way into the tens of millions of dynamic page views per day. One customer I worked with was handling more than ten million dynamic page views per day on just three systems running at less than half utilization each. (There were three or four smaller boxes doing static content up-front, and a larger database box behind however.)
The ancillary systems tend to far outnumber the main systems. Generally, at least in the places I've seen, IT handles the lot of them.
But when people pay for books, they expect certain rights; the right to read as often as they like, to loan, to mark-up, to give away, to take quotes from, to put in a library, etc. Until e-books give people all the same rights they have with regular paper-back books, they will not catch on.
Asking people to buy e-books as they currently exist is like saying "why don't you pay me 30,000 dollars for the same Ford except that you can't loan it to anyone, modify it, etc etc". People aren't going to buy into this bullshit.
It's not so much that I can't lend them; the Palm Reader format that I most often get books in uses a credit card to unlock the book. I can give it away, but I have to either type in or give away the credit card with it. That immediately restricts the number of copies that will get made, and of course if the thing gets massively given away they can track it back to me. This is a pretty effective and simple means of DRM, and yet it does give me much of the flexibility of paper in terms of lending.
I don't mind that at all.
What I mind is when they charge me $20 for a contemporary novel, same price as the hardcover book, and all I'm getting is bits. That drives me nuts. I think if they're not going to have to pay to print and ship it, I ought to benefit from the distribution cost savings.
There are some good arguments as to why that won't make them really cheap (much of the cost of producing most books is in the preprint production) but if you're offering in both formats there should be a discount for ebooks. On the other hand, I'm obviously an early adopter and there is infrastructure to pay for.
I find that these days it's likely that a title bought in ebook format will indeed be a couple of bucks less money than a paper one, and shipping is free. Unfortunately availability of new titles is still very limited, although vastly better now than even one year ago.
Ebooks are happening, even commercial ebooks, even though they are not yet mainstream. They still have their limitations relative to paper, but the convenience of the format (I regularly carry three or more in my PDA, whereas you'd be lucky to find me with even one paper book) is worth quite a bit. No more reading the National Enquirer in checkout lines.
I wonder if a lot of the problem with ebooks as a profit zone wasn't largely the result of the ebook initiative people of giving you basically two ebook alternatives:
Go buy this $300+ ebook reader, plus pay them premium prices, in order to read them.
Read them on your personal computer.
I dunno about the rest of you, but I wasn't going to buy a rocketbook or any of the others so that I could pay a bunch of money to download books over a slow-as-molasses modem. Why the heck can't I download them over my broadband connection to my PC and maintain my own library? And I wasn't going to buy books that I could only read on my PC, which I don't happen to be sitting in front of at any time when I want to be reading books.
I want to read books wherever I am, like you'd be able to do with a dedicated ebook reader, but I don't want to pay for or carry around a dedicated ebook reader.
As it turns out, I've been carrying a portable computer since early 1997 - a palm. So why not use that? The screen is small, but I always have it with me, and its print is really not all that much smaller than a lot of paperbacks anyway.
I thought that was a natural fit. I started reading ebooks on it in I think 1998, but certainly by the end of 1999. Back then there were only a few places you could get them, and peanutpress was the only place I could get contemporary stuff from well-known authors (plus the peanut reader did a very nice display job given the limitations of the device).
Since that time the number of ebook vendors has exploded. I still can't get them from Barnes and Noble or Amazon in a palm reader format (isn't it interesting that both support Microsoft's format but neither supports the much more popular palm reader format) but there has been an explosion of free and commercial ebook services serving the palmtop market. My current favorite is fictionwise.
Anyway, my point in all of this is that ebooks are selling commercially and have been selling for years. Not on high volumes, but I wonder if that's not because of the failure of the large booksellers to target the largest of the palmtop markets. The smaller vendors have existed for years and are obviously doing something right given that they're still around and their inventories are exploding, but they don't have the marketing push to really get ebooks out there.
Whatever, ebooks really are here if you want them and most likely you don't have to buy anything extra to read them.
This is true if you're using Apache as the application server, but most large web applications use the HTTP server as a front end, serving only static content. They refer requests to a back-end application for dynamic page generation, and often that application is running on a cluster of machines.
If you're using session affinity to bind a session to a particular application server, which is pretty much a necessity for high-volume applications, then it's to your benefit if each HTTP server can hold a connection open to every application server.
You can't do that on a 1.x Apache server because you'd end up having one connection for every app server and every Apache instance, and that can easily run into the tens of thousands of connections.
With Apache, therefore, you usually build a new TCP connection with each request, which is not very efficient.
Heh. My favorite from a customer call was from someone who got the message:
"Internal error - call ech."
I personally got yelled at for throwing some expletives in our source code after spending a really long period of time tracking down an OS bug in HP/UX such that if you closed a pty the wrong way your program became immortal (ie kill -9 wouldn't kill it, nor anything else). Who knew we had source licensees?
Heh. I remember helping out tech support with one particular customer who couldn't get our software to run. It fell to me because I wrote the error message that he was seeing.
Our software could run on either X11 or SunView (remember SunView?). I had detection code in there to tell you to try the other version of the program if it encountered certain graphics environment initialization problems.
Well, this guy couldn't get either one to run for some reason. I asked him which window system he was running.
Him: I'm not running either of them.
Me: Well, you have to run one of them.
Him: I can't. Every time I try to start the window system, it says "/dev/fb: File not found".
Me: Oh, ok. Someone must have deleted the framebuffer device. You'll have to talk to your system administrator to restore it.
Him: I/am/ the system administrator.
Me (moaning "oh god" to myself): Ok, then I suggest you call Sun technical support.
If someone was going to walk him through the task of figuring out which video hardware he had installed, I wanted it to be somebody else:-)
My favorite error message came out of an AT&T 3b2 running SysV.2 or maybe V.3. We were trying to break into it because the root password had been forgotten and unfortunately you needed a hardware password to boot it single user... and of course someone had changed that from the default and nobody knew it, either. We got the bright idea of disconnecting the battery to reset the machine to hardware defaults.
The next reboot gave us "NVRAM insanity error." Quite descriptive.:-)
Yes, Google is one such site, although their runtime is simplistic enough that it's not a really good example of a typical large-volume site. Amazon would be better, or eBay; I know eBay uses larger machines, don't know about Amazon. The only really high volume site I know off the top of my head that uses Intel-based hardware and individual personalization is hotmail and again they're dealing with thousands of servers.
If you're building a site like that then you've got to make your decision as to whether you'd rather use thousands of Intel servers or a few tens of larger servers. If it were my decision I'd go for the smaller number of larger servers simply because they require a lot fewer IT people to keep running, and every IT person you don't have to hire is another new machine or two you could buy every year. It adds up.
It's smarter to manage affinity by session, not by IP, since a variety of sources have rotating IPs (most notably AOL, but some business firewalls do it too).
Anyway, thanks for the tip. I haven't seen the LVS stuff at all yet.
I have more than a few problems with that idea, but amongst them is:
Diskless systems start to collapse the central servers even by forty or fifty clients. By the time you're talking the thousand or more Intel systems necessary for a big site you're looking at having to have a tiered system just to do software deployments, forget about data serving.
Diskless systems don't work well if you have more data than you can realistically afford to store in memory. You start to see practical limits (like hardware limitations) in the low gigabyte range, when most larger websites have static content to deliver in the hundreds of gigabyte range.
Applications are notoriously hungry because they have to do a lot of caching to offload the database since databases generally don't scale well. It's pretty common to see our application servers running with 2+ gig heaps, and we'll run one application server per CPU on a system, and you're probably running three or more 6 or 8 CPU systems just for the application server part. Try to make that diskless and you're now talking about machine configurations with something like 30G of RAM... very expensive and impractical.
We're talking about a totally different scale, really.
I think we're going to see more and more of this kind of server. The Zseries mainframes running Linux are really interesting because you're not so dependent on scalable SMP capabilities and yet you get the same kind of manageability as if you were working with a big SMP box. Nice.
I haven't personally done any deployments on such a system, but I like the idea.
If you're just serving static pages you're right. If you're doing dynamic content then you're wrong.
But 2.5 million hits a day is still just a moderate volume site to me. One of the sites I worked on sees in excess of a hundred million hits per day these days; it was up over ten million hits per day back in 1998.
I don't happen to know what Slashdot does for volume, but Slashdot is a very simplistic site when it comes to content production. Each page render doesn't take much horsepower and sheer replication can be used effectively. Things get more complicated when you're doing something like trying to figure out what stuff a user is likely to buy given their past buying history and/or what they're looking at right now.
If you really think a 4-way Intel box is equivalent to a 12-way Sun, well, it's clear you don't know what you're talking about. You're wrong even if all you're talking about is CPU, and of course I/O bandwidth is what makes or breaks you -- and there's no comparison in that respect.
Yea, big Suns are too expensive and you do need to keep the server count high enough that a failure or system taken down for maintenance isn't a really big impact on the site. I mentioned in a different posting that my cut on this is that the midrange Suns, 4xxx and 5xxx class, provide good bang-for-the-buck for high-volume sites.
Beware of false economy when looking at hardware. While it's true that smaller boxes are cheaper, they still require about the same manpower per box to keep them running. You rapidly get to the point where manpower costs dwarf equipment cost. People are expensive!
Capacity is an issue. We try to plan for enough excess at peak that the loss of a single server won't kill you, and hope you never suffer a multiple loss. Unfortunately most often customers underequip even for ordinary peak loads, to say nothing of what you see when your URL sees a real high load.[1] They just don't like to spend the money. I can see their point, the machines we're talking about are not cheap; it's a matter of deciding what's more important to you, uptime and performance or cost savings. Frankly most customers go with cost savings initially and over time (especially as they learn what their peak loads are and gain experience with the reliability characteristics of their servers) build up their clusters.
[1] People here talk about the slashdot effect, but trust me when I tell you that that's nothing like the effect you get when your URL appears on TV during "Friends".
You're talking about one particular application I imagine. MQ Series is actually pretty rare in large scale deployments, DB2 is like my third choice in databases, and I'd prefer not to use HTTP servers as the actual application server.
I've seen both kinds and take it from me, many small servers is more of a headache than the hardware cost savings is worth. Your network architecture gets complicated, you end up having to hire lots of people just to keep the machines running and with up-to-date software, and database connection pooling becomes a lot less efficient.
You save money in the long run by buying fewer, more powerful machines.
Apache is more flexible, but in traditional versions (1.x) you have a problem in that a new program instance is used for each request. That makes things like maintaining persistent connections to the application servers really hard.
Using something like iPlanet each server instance opens a number of connections to each application server in your cluster; you get a nice connection pool that way. With the Apache design (again this is 1.x) you can't use a pool so TCP setup/teardown costs between the web server and the application servers start to be an issue.
Not that people don't do it, but it's a lot less efficient.
I can't speak for Zeus, and as I understand it the most recent version of Apache allows threaded deployments that can take advantage of connection pooling, but most high volume sites use IIS or iPlanet as their front end web server.
Slashdot Mirror
So.. is this Saber-C stuff Open Source/Free yet?
Nope, but it doesn't matter because it needed a rewrite anyway. Were I to do it today I'd move the instrumentation code into gcc, augment glib, and add some hooks to gdb to allow programmatic control (ie allow the inferior to ask the debugger to do things like stop). That would all be easier than most of our ports of Saber-C to new platforms, and would end up being largely platform independent.
Getting module swapping/incremental loading to work is a royal pain. Valuable, for sure, but maybe not worth the development agony. It's a lot less of an issue with demand loaded languages like C# and Java anyway.
But the real fallacy with "write-time debugging", and in fact all of the code-bug-free methodologies, is that there is very significant amounts of code that you must use to write your program that was written by someone else who is almost certainly not using your methodology (or, often, any methodology). So, even if your methodology worked on your own code it wouldn't work on the application as a whole.
I note that some of the most insidious bugs I've ever had to track down were caused by operating system errors. My favorite bug ever was under AIX where a call to select() with a 30-second timeout was returning in about 900 nanoseconds saying it had timed out. This caused a chain of failures. And, of course, it would only happen in some pretty unpredictable situations. In this case you needed two programs running under ptrace at the same time on the system for the bug to show itself - and it would only occur about one in five thousand calls to select() even under the failure conditions.
No amount of programmer effort on our part would have prevented this bug from biting us. We could certainly have put in a postcondition tester to check to see if the system call operated correctly (in fact that's how we worked around this problem) but can you realistically do that for every system call you ever make? Every library call you ever make? It's not practical to do so.
I'm all for doing things to minimize code errors, although to be completely honest the best method (as opposed to methodology) I've found to do that is to use languages that include their own run-time error checking, which are strongly typed, have mandatory exception handling, and which use garbage collection for heap management (Java being the only one I've used). The combination of such features leads to about 90% fewer bugs out-the-door and boosts programmer productivity by 200-400% over the long term versus something like C++ (that's data from several large programming projects I've been on).
It's that good because the environment tends to report problems as early as possible in the development process, when they are easiest to fix, and because of mandatory exception handling code more often gets written with failure handling in mind.
Even so, and even with methodologies that reduce bug rates even more, tools are often invaluable to producing good code. I find that heap inspection and profiling tools are very valuable for Java, for instance, as are techniques such as log instrumentation and assertion. (Unfortunately Java provides no facilities for any of this in-the-box, sigh.)
Anyway, to maximize programmer productivity and minimize bugs it pays to use an array of techniques and tools. Those who believe debuggers don't have their place, that they can be rendered pointless by technique, are badly mistaken.
Must be nice to work in a world where you have time to write comprehensive supra-unit tests and can rely on all your co-workers to have fully tested their code.
I'd like to visit that world sometime, because in my world schedules are allocated such that it's rare you have time to do more than just unit testing and they call integrated testing "beta".
I'm only half joking. Then again, our problem domain scale is probably offset by a couple of orders of magnitude and that will certainly make a difference.
How much of that is because you wouldn't find it useful, versus the fact that there really haven't been any good Java debuggers?
With program restarts in our Java applications into the ten-plus minute range, a debugger would be a serious time saver versus adding log or println entries.
I note that with JDK 1.4 the JVM debug interfaces are finally good enough to actually make a working debugger, which will help things a lot.
They got a demo copy of ObjectCenter (formerly Saber-C), a superdebugger I used to work on which performed a lot of runtime error checking. They loaded the code, ran it, and it pointed out the error for them. Their job finished, the group was disbanded.
I don't know if we got the product sale :-) but certainly they shared your opinion. I do, too, although the near total lack of functional debuggers in Java (until 1.4, which unfortunately I still can't use) has kept me in printlns. Thank God that Java incorporates runtime checking and dumps stack traces on error or we'd never get anything written.
In a different posting I talked about Saber-C and some of what we did that was useful, but I totally forgot about this particular feature. We did have the ability to link in debugger control functions like "break now". It was, indeed, quite useful.
When I interviewed one of my comments was, "Hey, that's like a Lisp environment for C!" I had been building a lisp-based shell at the time so that was really cool :-).
I used to work for a company called Saber, who later changed their name to Centerline. We produced a product called Saber-C (and later Saber-C++, and the products were renamed CodeCenter and ObjectCenter).
This product was a C/C++ superdebugger. It started out as a C interpreter developed at Harvard. Now, an interpretive environment allowed a lot of neat features: You could tell whether or not variables had been initialized at reference time, you had complete stack and context information, it was pretty easy to unload and reload code in source modules (dynamic reloading). You also got dynamic type checking, which caught errors in downcasting (for instance). And everyone's favorite, array bounds checking.
With this tool debugging C code often meant: Loading the code, running your test case, and seeing where the debugger complained. The first time you ran your code through it was a humbling experience ... almost all production code has serious (but normally benign) errors.
Unfortunately the tool didn't scale well because it took time to load the interpreted code into the debugger and the heap and runtime requirements of interpreters stressed machines of the time (this was when Sun3s were common, and 3/50s couldn't even have more than 4MB memory).
I think it was 1990 when we released version 3, whose big new feature was object code debugging (it's not really that easy to have object code debugging as well as interpretive code, although we could certainly have implemented it more easily than we did). We of course maintained the ability to dynamically reload code fragments, so you could do a "make", reload the affected modules, and be on your way in a few seconds. This sure beat the several minute wait you normally had with the linker and debugger restart. But what you lost was some of the runtime checks -- including most of what made the product interesting. Still, the ability to have superdebugger capabilities on some code and still scale to large programs was very valuable.
About a year, maybe two years later we got visited by the people who eventually started Purify. They had a neat approach where they'd get most of the benefit of our tool, but in object code, by disassembling the object code, adding extra code to perform checks, and putting it back together. This would give about 80% of the functionality of our debugger but with much lower runtime costs and work even if you didn't have source code.
We ended up re-implementing this functionality such that we had a Purify competitor (and getting sued, and losing, even though Sun later patented the same technique we used).[1] Integrating it into the debugger allowed you to fine-tune the debugging capability you wanted on a module-by-module basis. This was a superb tool.
Now, what are the raw capabilities that made this stuff really neat?
The tool also provided two additional interesting features. One was called "action points". Since we had a C interpreter in the product it was no big deal to allow you to type C code into it and run it at arbitrary times. We'd allow you to attach any code you wanted to any line in the program or to just type code at the debug prompt and it would run in the current scope. This was useful for things like adding print statements without recompiling, for instance, and for adding assertions, and many other things.
The other feature was a graphical heap inspector. All this did was display structures as tables of their internal values and allow traversal of references to other structures, which would be displayed (including links between the structures). This made the shape of data a LOT easier to determine.
Several people talked about adding reverse execution. That was actually done for Java by a tools company back in 1996 (maybe 1997). It didn't always work, but it was pretty neat. The problem is that the recording code was pretty expensive and there's a limit to people's patience and what they'll spend on hardware (although I grant that hardware is a lot less of an issue today than it was even in 1997, to say nothing of 1989). Anyway this feature is a lot less interesting if your debugger can find errors at the point they occur.
Anyway, back in 1995 when I started to work with Visual-C++ on a regular basis I was appalled by what Microsoft was calling a "state of the art" debugger. Feh! It was 1985 technology with a pretty face. It really has not improved much since then either, although they do at least have a pessimistic heap allocator now. With their ability to hook in with the compiler they could do a phenomenal job with heap allocation tracking and instrumentation of code with error checks but they don't bother (because, I suppose, they make their money even without it).
Now, using a language with runtime error checking built in - such as Java, C#, or Visual Basic, seriously undermines the amount of effort you have to put into the debugger. Still, I'd like to see a lot better heap debugging features built into compilers and runtime environments.
Hope this gives people some ideas, and that they lead to tools I can use.
[1] One thing that could be done to avoid the Purify patents would be to have the compiler emit the instrumentation. For the OSS world, where you always have the source code, that is eminently practical. Furthermore you can get all the benefits of an interpreter but at higher performance. A big problem with Purify's technology, and the technology we came up with, was that you're doing a lot of heuristics during code inspection. It's easy to miss code or misinterpret code, and of course the code inspector is VERY architecture and compiler dependent. What I would like to see is a gcc/glibc/gdb combination where the compiler, libraries, and debugger all worked together. That wasn't possible back when the vendor controlled the compiler and you didn't have access to library source. Today it is, at least for the OSS world.
I think the hundred-million dynamic pageviews site had three admins, but they switched hats with other jobs. One did double duty as the group manager, and the other two were part-time programmers. Multiple admins also meant that there was the possibility of time off :-).
They had a lot of outside help, though, since Exodus was hosting their machines for them and there was another IT department that did desktop management for the rest of the organization.
Keep in mind that with that kind of horsepower you're talking about a pretty darn large site -- like way into the tens of millions of dynamic page views per day. One customer I worked with was handling more than ten million dynamic page views per day on just three systems running at less than half utilization each. (There were three or four smaller boxes doing static content up-front, and a larger database box behind however.)
The ancillary systems tend to far outnumber the main systems. Generally, at least in the places I've seen, IT handles the lot of them.
Asking people to buy e-books as they currently exist is like saying "why don't you pay me 30,000 dollars for the same Ford except that you can't loan it to anyone, modify it, etc etc". People aren't going to buy into this bullshit.
It's not so much that I can't lend them; the Palm Reader format that I most often get books in uses a credit card to unlock the book. I can give it away, but I have to either type in or give away the credit card with it. That immediately restricts the number of copies that will get made, and of course if the thing gets massively given away they can track it back to me. This is a pretty effective and simple means of DRM, and yet it does give me much of the flexibility of paper in terms of lending.
I don't mind that at all.
What I mind is when they charge me $20 for a contemporary novel, same price as the hardcover book, and all I'm getting is bits. That drives me nuts. I think if they're not going to have to pay to print and ship it, I ought to benefit from the distribution cost savings.
There are some good arguments as to why that won't make them really cheap (much of the cost of producing most books is in the preprint production) but if you're offering in both formats there should be a discount for ebooks. On the other hand, I'm obviously an early adopter and there is infrastructure to pay for.
I find that these days it's likely that a title bought in ebook format will indeed be a couple of bucks less money than a paper one, and shipping is free. Unfortunately availability of new titles is still very limited, although vastly better now than even one year ago.
Ebooks are happening, even commercial ebooks, even though they are not yet mainstream. They still have their limitations relative to paper, but the convenience of the format (I regularly carry three or more in my PDA, whereas you'd be lucky to find me with even one paper book) is worth quite a bit. No more reading the National Enquirer in checkout lines.
I dunno about the rest of you, but I wasn't going to buy a rocketbook or any of the others so that I could pay a bunch of money to download books over a slow-as-molasses modem. Why the heck can't I download them over my broadband connection to my PC and maintain my own library? And I wasn't going to buy books that I could only read on my PC, which I don't happen to be sitting in front of at any time when I want to be reading books.
I want to read books wherever I am, like you'd be able to do with a dedicated ebook reader, but I don't want to pay for or carry around a dedicated ebook reader.
As it turns out, I've been carrying a portable computer since early 1997 - a palm. So why not use that? The screen is small, but I always have it with me, and its print is really not all that much smaller than a lot of paperbacks anyway.
I thought that was a natural fit. I started reading ebooks on it in I think 1998, but certainly by the end of 1999. Back then there were only a few places you could get them, and peanutpress was the only place I could get contemporary stuff from well-known authors (plus the peanut reader did a very nice display job given the limitations of the device).
Since that time the number of ebook vendors has exploded. I still can't get them from Barnes and Noble or Amazon in a palm reader format (isn't it interesting that both support Microsoft's format but neither supports the much more popular palm reader format) but there has been an explosion of free and commercial ebook services serving the palmtop market. My current favorite is fictionwise.
Anyway, my point in all of this is that ebooks are selling commercially and have been selling for years. Not on high volumes, but I wonder if that's not because of the failure of the large booksellers to target the largest of the palmtop markets. The smaller vendors have existed for years and are obviously doing something right given that they're still around and their inventories are exploding, but they don't have the marketing push to really get ebooks out there.
Whatever, ebooks really are here if you want them and most likely you don't have to buy anything extra to read them.
If you're using session affinity to bind a session to a particular application server, which is pretty much a necessity for high-volume applications, then it's to your benefit if each HTTP server can hold a connection open to every application server.
You can't do that on a 1.x Apache server because you'd end up having one connection for every app server and every Apache instance, and that can easily run into the tens of thousands of connections.
With Apache, therefore, you usually build a new TCP connection with each request, which is not very efficient.
"Internal error - call ech."
I personally got yelled at for throwing some expletives in our source code after spending a really long period of time tracking down an OS bug in HP/UX such that if you closed a pty the wrong way your program became immortal (ie kill -9 wouldn't kill it, nor anything else). Who knew we had source licensees?
Our software could run on either X11 or SunView (remember SunView?). I had detection code in there to tell you to try the other version of the program if it encountered certain graphics environment initialization problems.
Well, this guy couldn't get either one to run for some reason. I asked him which window system he was running.
Him: I'm not running either of them. /am/ the system administrator.
Me: Well, you have to run one of them.
Him: I can't. Every time I try to start the window system, it says "/dev/fb: File not found".
Me: Oh, ok. Someone must have deleted the framebuffer device. You'll have to talk to your system administrator to restore it.
Him: I
Me (moaning "oh god" to myself): Ok, then I suggest you call Sun technical support.
If someone was going to walk him through the task of figuring out which video hardware he had installed, I wanted it to be somebody else :-)
The next reboot gave us "NVRAM insanity error." Quite descriptive. :-)
If you're building a site like that then you've got to make your decision as to whether you'd rather use thousands of Intel servers or a few tens of larger servers. If it were my decision I'd go for the smaller number of larger servers simply because they require a lot fewer IT people to keep running, and every IT person you don't have to hire is another new machine or two you could buy every year. It adds up.
Anyway, thanks for the tip. I haven't seen the LVS stuff at all yet.
We're talking about a totally different scale, really.
I haven't personally done any deployments on such a system, but I like the idea.
But 2.5 million hits a day is still just a moderate volume site to me. One of the sites I worked on sees in excess of a hundred million hits per day these days; it was up over ten million hits per day back in 1998.
I don't happen to know what Slashdot does for volume, but Slashdot is a very simplistic site when it comes to content production. Each page render doesn't take much horsepower and sheer replication can be used effectively. Things get more complicated when you're doing something like trying to figure out what stuff a user is likely to buy given their past buying history and/or what they're looking at right now.
If you really think a 4-way Intel box is equivalent to a 12-way Sun, well, it's clear you don't know what you're talking about. You're wrong even if all you're talking about is CPU, and of course I/O bandwidth is what makes or breaks you -- and there's no comparison in that respect.
Beware of false economy when looking at hardware. While it's true that smaller boxes are cheaper, they still require about the same manpower per box to keep them running. You rapidly get to the point where manpower costs dwarf equipment cost. People are expensive!
Capacity is an issue. We try to plan for enough excess at peak that the loss of a single server won't kill you, and hope you never suffer a multiple loss. Unfortunately most often customers underequip even for ordinary peak loads, to say nothing of what you see when your URL sees a real high load.[1] They just don't like to spend the money. I can see their point, the machines we're talking about are not cheap; it's a matter of deciding what's more important to you, uptime and performance or cost savings. Frankly most customers go with cost savings initially and over time (especially as they learn what their peak loads are and gain experience with the reliability characteristics of their servers) build up their clusters.
[1] People here talk about the slashdot effect, but trust me when I tell you that that's nothing like the effect you get when your URL appears on TV during "Friends".
YMMV.
You save money in the long run by buying fewer, more powerful machines.
Using something like iPlanet each server instance opens a number of connections to each application server in your cluster; you get a nice connection pool that way. With the Apache design (again this is 1.x) you can't use a pool so TCP setup/teardown costs between the web server and the application servers start to be an issue.
Not that people don't do it, but it's a lot less efficient.
I can't speak for Zeus, and as I understand it the most recent version of Apache allows threaded deployments that can take advantage of connection pooling, but most high volume sites use IIS or iPlanet as their front end web server.