Slashdot Mirror
Interview: Queen Elizabeth II's Webmaster Answers
fprintf asks:
Seems like a simple question, but why Linux? It seems like all the other high powered sites are using BSD of one variant or another.
...and...
Raul Acevedo asks:
In the original Sunday Times article, you are quoted as saying:
"... you can't beat them [Linux on Intel] in the bangs for your buck department. It blows Sun out of the water..."
Could you elaborate on how Linux compares to Solaris? Did you mean that Linux blows Sun out of the water in terms of price/performance (which is obvious since Linux is free), or just in general for your particular needs?
I'd be curious to hear your thoughts on Linux vs. Solaris, not just in terms of price, but overall performance, reliability, maintainability, and ease of use. As a developer, I'm seeing Linux considered as an alternative to Solaris in many places, but there's little factual (or even anecdotal) information comparing the two.
ANSWER:
I'll take these two together since the answers overlap.
In retrospect, I wish I /had/ chosen OpenBSD ;-)
And I would certainly choose OpenBSD over GNU/Linux if I were building a firewall, or an intrusion detection system (based on say, Marcus Ranum's NFR) where packet capture at wire speed was important. (No - that tells you nothing about CCTA's network architecture....)
The choice of GNU/Linux seems to have caused all sorts of interest (witness this interview itself) when a *BSD may not have been so "controversial". Frankly I'm a little surprised at the reaction the choice seems to have generated. After all, we are just talking about web servers here. Many ISPs choose GNU/Linux on Intel for exactly the same reasons I have done - best value for money for the task in hand.
Let's put this into perspective first though - and dispel a few myths which seem to have cropped up in the press. I have emphatically /not/ ditched Solaris in favour of GNU/Linux. I still have 14 operational Solaris boxes running on the network. I have GNU/Linux running on 5 Dell Poweredge 2300s (with half a gig of RAM each - the Times article suffered from poor editing). I also run GNU/Linux on my desktop in the office, on my laptop and desktop machine at home and on a couple of internal servers handling DNS and proxy services for CCTA.
The GNU/Linux choice came about for two reasons:
- - I had operational experience of GNU/Linux on a day to day basis.
- - I was faced with replacing life expired Sun hardware (including a SPARC 1000E and a couple of Sparc 20s) as part of the normal process of hardware/maintenance/upgrade.
On the second point. When the usual business planning round came up and I had to make decisions about hardware replacement for some of the older servers, it was obvious GNU/Linux on Intel could be a much cheaper option than simple replacement of the Sun hardware. Consider: a Dual 450MHz Pentium II, with 27 gig of disk, internal DDS3 and CDROM and half a gig of RAM costs less than £5000; a dual 300MHz UltraSPARC 2 with similar configuration costs around three times that. Question. Do I need to spend that kind of money simply to run a Web server? So I ran some tests and concluded that - no I didn't need to spend that kind of money (taxpayers money I should add) and plumped for the GNU/Linux on Intel combination on the purely pragmatic grounds of best value for money for the job in hand.
For the purpose of testing I took as a benchmark the maximum real life hit rate I had ever seen on one of the Solaris servers - around 1.5-2 million hits in a day. (By hit, I mean http GET or POST request). Then I doubled that as working assumption of a realistic maximum load in my environment.
For testing I took a fairly standard, but reasonably specced PC (a single Pentium 450MHz processor, 256Mb ECC SDRAM, single 18Gb LVD 10,000 RPM SCSI disk) and loaded Redhat Linux 5.2 running Apache 1.3.3. (Because that was what I had to hand). Apart from the Web server, I turned off all other daemons. I then loaded that server with a complete copy of my main www.open.gov.uk web.
In order to simulate a real life load, I had to find some way of grabbing a randomised list of URLS from the server which reflected the real world as closely as possible. After some testing with a variety of home spun scripts and commmand line web testers (such as webgrab) it quickly became clear that I would bog down the clients long before I made any real demands on the server. Some searching around and questions of colleagues lead me to http://alumni.caltech.edu/~dank/fixing-overloaded-web-server.html which is a useful site pointing to benchmarks and tools. This pointed me to http_load at http://www.acme.com/software/http_load/ which turned out to be pretty nifty since it runs in a single process. And of course, being OSS, I could tweak the code slightly to match my requirements. Thus armed I built some lists of URLs which were deliberately chosen to represent small text/HTML files, medium sized gif/jpeg files and large PDFs since this is the real life world on the public web servers. In load testing the server I then fired up just three client machines (one SPARC 5 running Solaris and two low end Pentiums running GNU/Linux since that was all I had to hand).
In peak load testing over a sustained 4 hour period I managed to get the server to deliver over 13,000 Mbytes in just under 500,000 HTTP transfers. During that period, CPU utilisation never went above 10%, and was usually around the 5% mark. Disk utilisation was minimal. The network connection rate was much higher than anything I'd seen in real life on the existing external servers (some 500 established connections during snapshots on the load testing period). Also during the test, Apache complained that it had reached the MaxClients setting (then 150) with no adverse effects.
Given that such a reasonably low end server handled most of what I could throw at it in my test environment, I concluded that GNU/Linux on only slightly beefier hardware made eminent sense.
----------
anthonyclark asks:
Do you get many cracker/script kiddie attacks on the various web sites you run?
ANSWER:
Yes ;-)
Any high profile site is going to attract unwelcome visitors. My job is made harder, and more stressful, by such attention - but that is what I am paid for. My friends know that I have nightmares about waking up to find graffiti (which is all it is) on one of my customers sites.
Like any other conscientious sysadmin I take a personal interest in the security of my servers. Naturally I will use all the tools at my disposal to minimise the vulnerabilities. But of course I get unwelcome attention.
A plea to the community if I may. And here I can do no better than quote from Fyodor's article in Phrack Volume 8 issue 54 where he discusses remote OS fingerprinting:
"A worse possibility is someone scanning 500,000 hosts in advance to see what OS is running and what ports are open. Then when someone posts (say) a root hole in Sun's comsat daemon, our little cracker could grep his list for 'UDP/512' and 'Solaris 2.6' and he immediately has pages and pages of rootable boxes. It should be noted that this is SCRIPT KIDDIE behavior. You have demonstrated no skill and nobody is even remotely impressed that you were able to find some vulnerable .edu that had not patched the hole in time. Also, people will be even _less_ impressed if you use your newfound access to deface the department's web site with a self-aggrandizing rant about how damn good you are and how stupid the sysadmins must be."Sysadmins are not stupid. They are simply usually overworked and have to balance the need to provide services to their customer base with the need to minimise the risks to those services. Attacking public servers (whoever owns them) merely serves to irritate sysadmins, and usually nobody else.
I was not overjoyed to notice comments on /. of the form "whoo, so the Royal Web site has moved to Linux. I've got a rootkit with your name on it" (you know who you are). Consider. I have just moved some high profile web sites to the OS of choice to you readers. You want to see that OS taken seriously. Scribbling graffiti all over such a web site would have all sorts of negative impacts on the perceptions of people who matter.
Besides, you'd upset me.
----------
chromatic asks:
If you could add or change three things about Linux to make your job easier or more enjoyable, what would they be?
ANSWER:
1. The ability to read BUGTRAQ, evaluate the threat, consider vulnerability to that threat and auto patch or upgrade accordingly. It should then email me saying "I'm OK now, you can go back to reading /.".
2. An artificial intelligence based real time log watcher and network daemon which could learn network connect patterns and modify either the stack or the services running accordingly. The system should be capable of real-time blocking (a la portsentry) of "hostile" connects, co-operation with external IDS systems and firewalls, real-time reconfiguration of external security components, real-time alerts to other hosts on the lines of "hey guys, I'm being hit by X, watch it." It should then email me saying "I'm OK now, you can go back to reading /." :-)
3. An ASCII character based version of rogue. I miss it.
----------
Ryandav asks:
What kind of redundancy do you build into the server system for such a large and important site, ie. round-robin style servers or large, beefy superboxes, etc...
ANSWER:
You can see from answer above that I do not use "large, beefy superboxes". Frankly you don't need to to run a Web server. Nor do I use round robin DNS or other load balancing such as CISCO local director. In my experience of the sites I run, I don't need to do so. None of the sites gets hit hard enough to warrant the additional complexity of mirrored, load balanced servers. Our most popular site by far is the Royal Household site. That takes around 2-2.5 million hits per week (though I expect that to go up slightly now). The highest consistent hit rate I have seen is around 1.5-2 million hits per day. Any of the servers I have could cope with that. The redundancy we build in is in having backup hardware ready to run.
----------
wowbagger asks:
To what extent is the Royal Family involved with the site (e.g. content creation)?
ANSWER:
The Royal Family take an active interest in both of the royal web sites (one of which is hosted by the Press Association - www.royalinsight.gov.uk -). This interest includes both the current content of the sites, as well as future developments. The Queen herself launched royal.gov.uk in March 1997.
jd asks:
What's the official reaction to these sites running Linux? Assuming the British Government, and Her Majesty, are aware that their public image on the Internet is being presented via software that is non-traditional and non-commercial, what do they think of it all?
ANSWER:
The priority for the heavily visited royal web site is accessibility, balanced of course by reliability and security. These are the important issues, rather than the nature of the server operating system.
----------
Dicky asks:
What is your background? Are you a techie, an admin person, or an other? Do you use Linux personally? If so, did you come from a Unix, Windows or other background?
ANSWER:
I am a techie (though some of my friends and colleagues are a little less complimentary than that). My background is in Unix sysadmin and network management. I joined CCTA in 1993 from the UK Treasury where I was responsible for their Unix based OA system. Prior to that I was responsible for IT security in the Treasury. I have done some small systems development work in the past on MS/DOS machines (way before windows really took off) and CP/M micros. Most of my early career was in specialist support areas such as statistics, though I did a short stint in policy for a while in the mid to late 80's - didn't like it much.
Yes, I use GNU/Linux personally. It is my preferred platform for home use.
Dicky also asks:
And a related question: What is the primary system around your department?
ANSWER:
Depends what you mean by my department. In my area of responsibility the main systems are all *nix based. But the corporate desktop is NT4.
----------
Brian Knotts asks:
The obvious question: Does the Queen read Slashdot? :-/
ANSWER:
No. The Queen's interest in Internet matters is non-technical, although she sees on her visits to a wide variety of organisations the increasingly imaginative uses for the Internet.
----------
Simon Brooke asks: I've been very pleased lately to see Open.Gov's clear policy statement on the use of open standards. I'm personally involved in working with some large UK companies on their own Web standards policies, and having this to point to has been extremely useful to me. How difficult was it to get buy in to these standards by all the people who 'own' different Government sites, and how difficult is it to enforce?
I notice, for example, that the Scottish Parliament's web site, and my local Council's Web site, do not yet conform. Without wishing to point fingers at specific organisations, is it your intention to cajole all sites within .gov.uk to conform to these standards? Is it appropriate for members of the public to draw administrators of these sites attention to these standards?
ANSWER:
CCTA has long been a standards based organisation. My colleague Neil Pawley is CCTA's representative to W3C. Neil is also lead designer on the open.gov.uk site. Since CCTA is a member of W3C it is entirely appropriate that we should take a lead in using standards set by that organisation. Using HTML4, CSS2 and XHML1 for example on a real life server gives us valuable information on usability issues such as browser compatibility. Much of the feedback we have received has been very positive. On occasion we have had to deviate slightly from the standards where their use causes our public difficulty because of some incompatibility with a particular client setup. That experience itself is very helpful, since it allows us to feed back into the standards making process.
CCTA has an advisory role on best practice in the use of IS/IT in the UK Public sector. We have no authority to mandate particular standards, nor would we seek to do so. If the use of standards is to be effective in any way, it is because the standards themselves make sense in the real world (witness the growth in the use of the TCP/IP protocol set at the expense of the OSI standards).
Simon Brooke adds... Oh, and, by the way, keep up the good work!
We intend to.
Thanks for your interest. It has been educational for me.
-- Mick Morgan
-- end --
Next week: John Vranesevich of AntiOnline.
That's one hell of a throughput, for a single box that size!
These are static pages, so you can use an accelerator, such as Squid. This might easily bump the number of access up by an order of magnitude, taking you to 800 million hits per day. Not far from that billion you mention. :) And we're -still- talking about a meagre low-end Pentium!
It would seem kind-of silly to talk about anything higher-end than this setup, but I might as well, just for amusement.
Throw in PGCC, and you're going to increase performance by perhaps as much as 20%. Let's use that figure, as a plausable guesstimate. We're now up to 960 million.
Linux is scalable to at least 2 processors, so let's say we do that. It's approximately linear, to 2, bringing us to 1,920 million.
Now, let's upgrade that natty old Pentium, and install K6's or P3's instead. The impact of this is harder to guesstimate, but let's say that you can squeeze a doubling in performance from such an upgrade. The maximum capacity now stands at 3,840 million hits.
This, then, is a first guess at the theoretical maximum load you can get out of Linux and Apache - close to 4 US Billion hits per day! (Probably slightly over, for *BSD and Apache)
This doesn't knock what this guy's done, in the least! On the contrary, it shows that what he's achieved is a staggering feat, AND that he can keep making staggering achievements for a long time to come.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
First, I'd like to join the chorus of those commenting on what a fine read this was. Thanks, Mick!
Second, I'd like to call attention to this casually mentioned statement:
> The GNU/Linux choice came about for two reasons... [1] I had operational experience of GNU/Linux on a day to day basis. [2]...
Wow. Linuxers in decision-making positions. Brace yourself, O World.
--
It's October 6th. Where's W2K? Over the horizon again, eh?
Sheesh, evil *and* a jerk. -- Jade
...the Queen, Linus Torvalds, official purveyor of Operating Systems to Her Majesty's Household.
illegitimii non ingravare
Well he certainly had 25 crashes and 17 cracks since he installed linux ;-)).
No, read on and you'll see
The choice of GNU/Linux seems to have caused all sorts of interest (witness this interview itself) when a *BSD may not have been so "controversial". Frankly I'm a little surprised at the reaction the choice seems to have generated. After all, we are just talking about web servers here. Many ISPs choose GNU/Linux on Intel for exactly the same reasons I have done - best value for money for the task in hand.
He also stated he wanted to dispell some myths and made it very clear what he thinks about every cheesy computer newssite/paper riding on fact they choose linux. That's the argument.
Each hour I spend on security is an hour I don't spend improving the content of my site - and that content benefits a lot of people.
Why shouldn't I be complacent about security? Why should I waste my time downloading patches, installing new stuff, and so on?
I suppose if there was top-secret information on my systems, I'd think otherwise, but the only really valuable stuff on my sites is what's already exposed to the public.
To me, script kiddies are evil, and the whole idea of breaking into other people's systems for a thrill is childish.
I think there should be extremely stiff penalties for being a script kiddie, and they should be enforced. People should go to jail for invading systems -- that would stop it right proper. It's no different morally than breaking into someone's house and burning it down.
D
----
>He had some pretty good responses. And I hope, those of you that are script kiddies take heed of his request. Discovering a security hole
>and reporting it is respectable, but taking a reported security hole and exploiting it is despicable.
I wouldn't say "despicable"... maybe "pathetic".
Wonderful peek into the life of a fellow sysadmin. True, script kiddies are annoying, but they're just that - annoying. Not dangerous, or threatening. In my experience, it's the PHBs that scream bloody murder when they find out we get routinely probed every day.
This is why web page defacements are more irritating that root break-ins - because the former makes the PHBs take notice, even though the latter takes a LOT more work to clean up. I'd prefer to deal with a machine than a few irate VPs.
Funny, though. The few script kiddies I've actually taken the time to talk to all tell me they dream of getting hired by some company on the basis of their 'leet skills. Heh. Just goes to show you how much they know about our jobs.
Personally, I prefer nethack, but here are a few rogues (all character-based)
ftp://metalab.unc.edu/pub/Linux/games/dungeon/
If you're reading this, Mick, you can grab the Linux port of Rogue from http://www.win.tue.nl/games/roguelike/rogue/index. html
Top regards!
I have the source somewhere. I'll see if I can dig it out. From what I remember, though, it's not freely distributable. Other than that, you may want to check out zangband. It's an enhancement of angband, which in itself is an enhanced version of moria, a rogue-like game. Yes, it has graphics if you want them, but I always compile it without them. Nothing like the good-old text based interface. More details at http://thangorodrim.angband.org.
"The invisible and the non-existent look very much alike." -- Delos B. McKown
On the response of
/had/ chosen OpenBSD ;-)
In retrospect, I wish I
I would probably agree. Although I use GNU/Linux for my personal web page, OpenBSD has been know for its security. A homogeneous solution is usually a bad one. This is my main argument against Microsoft. My experience with Unix is that, although not completely compatible, they all work well together. At work I use Solaris, AIX and Linux. Each with a separate duty. I'll probably start using *BSD OS soon too. Linux I feel is probably the best for interface and General setup. I'm looking at BSD for firewalls and some servers. Unix works because all of them try to follow standards. Again, Microsoft tries to implement their own "better" standards. I've been to two microsoft presentations, and both times they touted their proprietry solutions as the best thing out their. Unfortunately, their presentations are good, and they easily convince the higher ups. I wouldn't mind MS so much if they try to get along with other OS's instead of dominating them.
He had some pretty good responses. And I hope, those of you that are script kiddies take heed of his request. Discovering a security hole and reporting it is respectable, but taking a reported security hole and exploiting it is despicable.
Steven Rostedt
Steven Rostedt
-- Nevermind
To paraphrase Frank Drebin (from the Naked Gun movies): No matter how silly we find the idea of having a queen, we'll make her feel at home
- He is under attack from script kiddies and thinks he might be more secure with OpenBSD, or
- He is getting tired of all the people asking him why he didn't choose OpenBSD
BTW, if you're already using Linux, and want to increase security at the kernel level, you might want to look at Secure-Linux, a kernel patch which adds some nice security options to the Linux kernel.--
Interested in XFMail? New XFMail home page
Read it again. He wishes he chose BSD because the Linux choice has attracted too much pointless attention. If he had chosen BSD, he wouldn't have been mentioned on ./ or interviewed here, for example. With Linux come the groupies :)
Secure-Linux won't necessarily keep you safe from every potential exploit, but if you keep up to date, it will at least lessen your chances of getting bit by a heretofore-unknown exploit.
--
Interested in XFMail? New XFMail home page
In recognition of Her loyal subject's being interviewed on Slashdot, Her Royal Highness the Queen has decided to surf the 'web today. If she should show up here in Slashdot, we would like to request that all our readers please stand out of respect for HRH. You will be notified when she is visiting.
{a few minutes later}
We understand that HRH is currently reading something on C|Net. More information when we have it.
{A couple minutes later, a MIDI of God Save The Queen begins to play, then is cut off quickly}
We thought for a moment that she was going to click on a link Slashdot, but she decided to read NTKNow first.
{A couple minutes later, the MIDI starts to play again and a little Javascript "Alert" box pops up to all
We understand that HRH Queen Elizabeth II is now reading articles on Slashdot. If you would please stand to show your respect.
.... and all across the world, random geeks in offices, cubicles, living rooms, and dens stand up, looking a bit embarrassed and uncomfortable because everyone's looking at them as they stand at attention with a tinny little MIDI coming from their computer.
Everything gets all wavy and misty again, indicating the end of the day-dream sequence.
-=-=-=-=-
-=-=-=-=-
My mom's going to kick you in the face!
You could also try using StackGuard to make your daemons more secure; it looks quite cool.
-- Ed Avis ed@membled.com
It helps if the interviewee reads /. and thinks it's cool :-)
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
A question shoulda been, "France is considering making Open Source mandatory for its government's use. Does that make it less desirable for the UK ?
In retrospect, I wish I /had/ chosen OpenBSD ;-)
Why is that? I mean, I know most of the standard arguments for BSD over Linux but why would you, a Linux user even at home, now make that statement? The standard security reasons? Was there a specific incident (that you can talk about) that triggered that statement? Had you later tested the BSD / Apache combination and achieved better performance?
Bleh!
I think the is the best interview I've read in a long time. Here is the key that jumped out at me.
This is a real world HIGH end test of linux in the server arena. No 8 way procs boxen running the web site. Not even load balancing. Look how it handles under the load. Wonderfully. This is what I want to see more of when people mention benchmarks. I want High profile real world examples. Thank you for a great interview and the positive support of GNU/Linux
"We hope you find fun and laughter in the new millenium" - Top half of fastfood gamepiece
"Fighting the underpants gnomes since 1998!" "Bruce Schneier knows the state of schroedinger's cat"
The benchmarking tools can be found here.
Http_load can be found here.
- I find it extremely amazing that Slashdot now has the weight (readership, etc.) behind to be able to interview just about anybody it wants
Interview: Bill Gates AnswersNow that's a slashdot interview I'd like to see. I just think that we would need more moderation points. A lot more.
-Denor
I was not overjoyed to notice comments on /. of the form "whoo, so the Royal Web site has moved to Linux. I've got a rootkit with your name on it" (you know who you are). Consider. I have just moved some high profile web sites to the OS of choice to you readers. You want to see that OS taken seriously.
/. too. You don't think all these "First Post!" morons actually have a clue do you?
/. crowd and point out that not everyone who is a /. fan is a open-source/linux fan.
Hey, script kiddies read
I just wanted to exonerate the Linux-lovin'
/. is bigger than that.
---
- Give a man a fire and he's warm for a day, but set him on fire and he's warm for the rest of his life.