Slashdot Mirror
Slashdot's Setup, Part 2- Software
The software side of Slashdot takes over at the point where our load balancers -- described in Friday's hardware story -- hand off your incoming HTTP request to our pound servers.
Pound is a reverse proxy, which means it doesn't service the request itself, it just chooses which web server to hand it off to. We run 6 pounds, one for HTTPS traffic and the other 5 for regular HTTP. (Didn't know we support HTTPS, did ya? It's one of the perks for subscribers: you get to read Slashdot on the same webhead that admins use, which is always going to be responsive even during a crush of traffic -- because if it isn't, Rob's going to breathe down our necks!)
The pounds send traffic to one of the 16 apaches on our 16 webheads -- 15 regular, and the 1 HTTPS. Now, pound itself is so undemanding that we run it side-by-side with the apaches. The HTTPS pound handles SSL itself, handing off a plaintext HTTP request to its machine's apache, so the apache it redirects traffic to doesn't need mod_ssl compiled in. One less headache! Of our other 15 webheads, 5 also run a pound, not to distribute load but just for redundancy.
(Trivia: pound normally adds an X-Forwarded-For header, which Slash::Apache substitutes for the (internal) IP of pound itself. But sometimes if you use a proxy on the internet to do something bad, it will send us an X-Forwarded-For header too, which we use to try to track abuse. So we patched pound to insert a special X-Forward-Pound header, so it doesn't overwrite what may come from an abuser's proxy.)
The other 15 webheads are segregated by type. This segregation is mostly what pound is for. We have 2 webheads for static (.shtml) requests, 4 for the dynamic homepage, 6 for dynamic comment-delivery pages (comments, article, pollBooth.pl), and 3 for all other dynamic scripts (ajax, tags, bookmarks, firehose). We segregate partly so that if there's a performance problem or a DDoS on a specific page, the rest of the site will remain functional. We're constantly changing the code and this sets up "performance firewalls" for when us silly coders decide to write infinite loops.
But we also segregate for efficiency reasons like httpd-level caching, and MaxClients tuning. Our webhead bottleneck is CPU, not RAM. We run MaxClients that might seem absurdly low (5-15 for dynamic webheads, 25 for static) but our philosophy is if we're not turning over requests quickly anyway, something's wrong, and stacking up more requests won't help the CPU chew through them any faster.
All the webheads run the same software, which they mount from a /usr/local exported by a read-only NFS machine. Everyone I've ever met outside of this company gives an involuntary shudder when NFS is mentioned, and yet we haven't had any problems since shortly after it was set up (2002-ish). I attribute this to a combination of our brilliant sysadmins and the fact that we only export read-only. The backend task that writes to /usr/local (to update index.shtml every minute, for example) runs on the NFS server itself.
The apaches are versions 1.3, because there's never been a reason for us to switch to 2.0. We compile in mod_perl, and lingerd to free up RAM during delivery, but the only other nonstandard module we use is mod_auth_useragent to keep unfriendly bots away. Slash does make extensive use of each phase of the request loop (largely so we can send our 403's to out-of-control bots using a minimum of resources, and so your page is fully on its way while we write to the logging DB).
Slash, of course, is the open-source perl code that runs Slashdot. If you're thinking of playing around with it, grab a recent copy from CVS: it's been years since we got around to a tarball release. The various scripts that handle web requests access the database through Slash's SQL API, implemented on top of DBD::mysql (now maintained, incidentally, by one of the original Slash 1.0 coders) and of course DBI.pm. The most interesting parts of this layer might be:
(a) We don't use Apache::DBI. We use connect_cached, but actually our main connection cache is the global objects that hold the connections. Some small chunks of data are so frequently used that we keep them around in those objects.
(b) We almost never use statement handles. We have eleven ways of doing a SELECT and the differences are mostly how we massage the results into the perl data structure they return.
(c) We don't use placeholders. Originally because DBD::mysql didn't take advantage of them, and now because we think any speed increase in a reasonably-optimized web app should be a trivial payoff for non-self-documenting argument order. Discuss!
(d) We built in replication support. A database object requested as a reader picks a random slave to read from for the duration of your HTTP request (or the backend task). We can weight them manually, and we have a task that reweights them automatically. (If we do something stupid and wedge a slave's replication thread, every Slash process, across 17 machines, starts throttling back its connections to that machine within 10 seconds. This was originally written to handle slave DBs getting bogged down by load, but with our new faster DBs, that just never happens, so if a slave falls behind, one of us probably typed something dumb at the mysql> prompt.)
(e) We bolted on memcached support. Why bolted-on? Because back when we first tried memcached, we got a huge performance boost by caching our three big data types (users, stories, comment text) and we're pretty sure additional caching would provide minimal benefit at this point. Memcached's main use is to get and set data objects, and Slash doesn't really bottleneck that way.
Slash 1.0 was written way back in early 2000 with decent support for get and set methods to abstract objects out of a database (getDescriptions, subclassed _wheresql) -- but over the years we've only used them a few times. Most data types that are candidates to be objectified either are processed in large numbers (like tags and comments), in ways that would be difficult to do efficiently by subclassing, or have complicated table structures and pre- and post-processing (like users) that would make any generic objectification code pretty complicated. So most data access is done through get and set methods written custom for each data type, or, just as often, through methods that perform one specific update or select.
Overall, we're pretty happy with the database side of things. Most tables are fairly well normalized, not fully but mostly, and we've found this improves performance in most cases. Even on a fairly large site like Slashdot, with modern hardware and a little thinking ahead, we're able to push code and schema changes live quickly. Thanks to running multiple-master replication, we can keep the site fully live even during blocking queries like ALTER TABLE. After changes go live, we can find performance problem spots and optimize (which usually means caching, caching, caching, and occasionally multi-pass log processing for things like detecting abuse and picking users out of a hat who get mod points).
In fact, I'll go further than "pretty happy." Writing a database-backed web site has changed dramatically over the past seven years. The database used to be the bottleneck: centralized, hard to expand, slow. Now even a cheap DB server can run a pretty big site if you code defensively, and thanks to Moore's Law, memcached, and improvements in open-source database software, that part of the scaling issue isn't really a problem until you're practically the size of eBay. It's an exciting time to be coding web applications.
Could we have a better run-down of what unpatched software is running on the server? It would really help. Thanks.
Yay for segregated caching, where one machine gets data before the others do...
If I have nothing to hide, don't search me
Seems like only a matter of time before someone could nail you with an sql injection attack. Maybe I will download your code...
Sadly, where I work, they just hope for the best and throw more hardware at the problem. We're running a new site, hosted off-site, which is killing our network bandwidth. Not my choice. I just shake my head.
A feeling of having made the same mistake before: Deja Foobar
I've always wondered, what's happening system-wise when we see "nothing for you to see here" vs. "page not found"?
stuff |
c) We don't use placeholders. Originally because DBD::mysql didn't take advantage of them, and now because we think any speed increase in a reasonably-optimized web app should be a trivial payoff for non-self-documenting argument order. Discuss!
I guess speed might be one consideration. Generally I like to use place holders because it adds simplicity when passing some types of queries. If there's one thing I've seen a problem with, it's failing to properly sanitize incoming information that is passed to the database. A LOT of php code out there is rather easy to blow a hole through due to this. It also simplifies a lot of junk I'd rather not deal with like quoting and such. In either case you're an idiot if you don't sanitize everything first anyway, but my mantra is safety first. I actually loath doing many applications where I can't use them (like the ruby database libraries).
Well that's my take anyway. There's some rather nice code in slash that taut me some better methods in perl, and I'd say you guys are way above my level.
It would be nice if a tech site like slashdot supported IPv6
Comment removed based on user account deletion
But what version of Windows Server are you running all this on? *duck*
I remember when MOD was an audio format, and DOS wasn't a network attack....
One question. When someone logs in, is there a way to login through https? Because it doesn't matter if you get to https AFTER login - the login procedure can be sniffed anyway :-/
i also got this bug once this morning. first time i open an article there were no comments (even though the index page said there were about 100), then i refreshed and there about 100. i presume this was introduced with the changes made today (the "Loading" ajax message, the lightgrey already-read-this-comment color)
(Didn't know we support HTTPS, did ya? It's one of the perks for subscribers: you get to read Slashdot on the same webhead that admins use, which is always going to be responsive even during a crush of traffic -- because if it isn't, Rob's going to breathe down our necks!)
Yes, it's a nice perk, but a even nicer perk would be to let everybody at least login through HTTPS. Weren't we bashing companies earlier for not using SSL by default for logins?
Your hair look like poop, Bob! - Wanker.
OS/2. To The Maxxxxx.
If this is the place for suggestions and complaints for Slashdot, may I put in my two cents? Sounds like people have been suggesting a number of new mods and changes to the old ones. Could this be revised? For example:
... but maybe we'll take it one step at a time.)
- separating +1 Funny into "+1 Funny-Raise Karma" and "+1 Funny-Karma Unaffected"
- combining -1 Flamebait with -1 Troll
- adding -1 Wrong (or -1 Misinformed), the opposite of "+1 Informative"
- combining +1 Interesting with +1 Insightful
(In fact, we can have diametrically opposed mods, like Informative/Misinformed, Underrated/Overrated, Insightful/Flamebait, Funny/Rude, etc.
And, yes, put in my vote for logging in via HTTPS.
404555974007725459910684486621289147856453481154 in hex is "You sank my Battleship?"
[GPG key in journal]
Just curious if there is another release of Slash coming anytime soon?
Also is there an up-to-date list of other sites running the code? The Slashcode sites list is sadly empty these days (which I hope doesn't mean there aren't any other sites that run Slash)
If CPU is a bottleneck, have you thought about using Sun's Niagra-based servers at all? They have CPU (and RAM) to spare.
It seemed an oddity to me that the old /. code base used FrontPage Extensions. Why did you use it and what did you replace it with when you dropped it?
+0.5 Somewhat Useful Kvetching
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
Honestly, I never bookmark anything on slashdot because it's news (and so is in passing) - but I bookmarked this. This is a resource. I do perl application coding for the web, and while I don't pretend to be an expert in any form, this could be really useful if I ever need to scale up performance - thanks for posting this!
So where is the process that slashdot attempts to my port 80 periodically.
Is that your abuse system?
'cause I see no reference to IIS, SQL Server, .NET and I just read a Microsoft press release that says high-volume web portals can't provide high-availability service w/o them??
Last time I checked (which was like 10 years ago), .shtml stood for Server Side Includes (SSI) HTML, which are definitely not static.
.htmls?
Wouldn't it have been better to choose an extension/term not already used, such as
GLaDOS for President 2016! "Well here we are again. It's always such a pleasure." -- GLaDOS, 2011
Isn't the standard thing to do to append the source IP you (pound or whatever) see to the existing contents of the header (if the header exists) separated by commas? There should be no need for a separate header. This works fine using F5 Load Balancers certainly, although I haven't used Pound myself.
From wikipedia:
X-Forwarded-For: client1, proxy1, proxy2
Cfr. Subject.
Because Boa is just about the best thing ever at serving static content.
Caesar si viveret, ad remum dareris.
for when us silly coders deci
When WE silly coders. It sounds goofy if you say when us decide, doesn't it?
From the article: If this reference appears in an HTTPS page, the mixed content warning will appear. How to craft a reference that works for both? The answer is again relative URLs, but using a more obscure syntax:
Happy moony
So you run 1 box to share all of the data to the webservers? Is this kinda dangerous in the fact that losing that box knocks out all your webservers? I would expect /. to run some sort of clustered NFS service or have some redundancy build in. I know RHEL runs a cluster GFS config that allows for multiple boxes to run NFS servers.
Slashdot: Failed Car Analogies. Amateur Lawyering. Anecdote Battles.
What annoys me in particular about the latter is that sometimes these implementations evolve from fragile to usable or good enough or whatever. But some admins seem to be unable to comprehend that things improve and stick to some home grown process without looking to see what has changed outside their world. They could spend less time re-applying bandages and more time doing fun things. [Like log processing with Hadoop. :) ]
Of course, there are the usual complaints about NFS security from folks who didn't know that NFS had a spec for Kerberos support or their OS-of-choice didn't support it. All hail NFSv4 for making Kerberos support a requirement...
As I read, non-character or non-integer data is escaped. Is that the reason why we cannot use Unicode characters? (Sorry, not a computer major). At least the characters you can type on a keyboard with non-US layout (for eg. I can press Alt Gr+E and it will print a euro sign), should be supported.
I know, I know, Slashdot is US centric... but...
MySQL supports [named] placeholders just fine. It is PHP that does not. PHP is the suck for databases unless you use PDO. See examples 1737 for named placeholders and 1738 and 1739 for prepared statements.
I recommend prepared statements with positional placeholders (the '?' style) as the exact same sytax is used with Java, PHP, and Perl.