Slashdot Mirror
Slashdot's Setup, Part 2- Software
The software side of Slashdot takes over at the point where our load balancers -- described in Friday's hardware story -- hand off your incoming HTTP request to our pound servers.
Pound is a reverse proxy, which means it doesn't service the request itself, it just chooses which web server to hand it off to. We run 6 pounds, one for HTTPS traffic and the other 5 for regular HTTP. (Didn't know we support HTTPS, did ya? It's one of the perks for subscribers: you get to read Slashdot on the same webhead that admins use, which is always going to be responsive even during a crush of traffic -- because if it isn't, Rob's going to breathe down our necks!)
The pounds send traffic to one of the 16 apaches on our 16 webheads -- 15 regular, and the 1 HTTPS. Now, pound itself is so undemanding that we run it side-by-side with the apaches. The HTTPS pound handles SSL itself, handing off a plaintext HTTP request to its machine's apache, so the apache it redirects traffic to doesn't need mod_ssl compiled in. One less headache! Of our other 15 webheads, 5 also run a pound, not to distribute load but just for redundancy.
(Trivia: pound normally adds an X-Forwarded-For header, which Slash::Apache substitutes for the (internal) IP of pound itself. But sometimes if you use a proxy on the internet to do something bad, it will send us an X-Forwarded-For header too, which we use to try to track abuse. So we patched pound to insert a special X-Forward-Pound header, so it doesn't overwrite what may come from an abuser's proxy.)
The other 15 webheads are segregated by type. This segregation is mostly what pound is for. We have 2 webheads for static (.shtml) requests, 4 for the dynamic homepage, 6 for dynamic comment-delivery pages (comments, article, pollBooth.pl), and 3 for all other dynamic scripts (ajax, tags, bookmarks, firehose). We segregate partly so that if there's a performance problem or a DDoS on a specific page, the rest of the site will remain functional. We're constantly changing the code and this sets up "performance firewalls" for when us silly coders decide to write infinite loops.
But we also segregate for efficiency reasons like httpd-level caching, and MaxClients tuning. Our webhead bottleneck is CPU, not RAM. We run MaxClients that might seem absurdly low (5-15 for dynamic webheads, 25 for static) but our philosophy is if we're not turning over requests quickly anyway, something's wrong, and stacking up more requests won't help the CPU chew through them any faster.
All the webheads run the same software, which they mount from a /usr/local exported by a read-only NFS machine. Everyone I've ever met outside of this company gives an involuntary shudder when NFS is mentioned, and yet we haven't had any problems since shortly after it was set up (2002-ish). I attribute this to a combination of our brilliant sysadmins and the fact that we only export read-only. The backend task that writes to /usr/local (to update index.shtml every minute, for example) runs on the NFS server itself.
The apaches are versions 1.3, because there's never been a reason for us to switch to 2.0. We compile in mod_perl, and lingerd to free up RAM during delivery, but the only other nonstandard module we use is mod_auth_useragent to keep unfriendly bots away. Slash does make extensive use of each phase of the request loop (largely so we can send our 403's to out-of-control bots using a minimum of resources, and so your page is fully on its way while we write to the logging DB).
Slash, of course, is the open-source perl code that runs Slashdot. If you're thinking of playing around with it, grab a recent copy from CVS: it's been years since we got around to a tarball release. The various scripts that handle web requests access the database through Slash's SQL API, implemented on top of DBD::mysql (now maintained, incidentally, by one of the original Slash 1.0 coders) and of course DBI.pm. The most interesting parts of this layer might be:
(a) We don't use Apache::DBI. We use connect_cached, but actually our main connection cache is the global objects that hold the connections. Some small chunks of data are so frequently used that we keep them around in those objects.
(b) We almost never use statement handles. We have eleven ways of doing a SELECT and the differences are mostly how we massage the results into the perl data structure they return.
(c) We don't use placeholders. Originally because DBD::mysql didn't take advantage of them, and now because we think any speed increase in a reasonably-optimized web app should be a trivial payoff for non-self-documenting argument order. Discuss!
(d) We built in replication support. A database object requested as a reader picks a random slave to read from for the duration of your HTTP request (or the backend task). We can weight them manually, and we have a task that reweights them automatically. (If we do something stupid and wedge a slave's replication thread, every Slash process, across 17 machines, starts throttling back its connections to that machine within 10 seconds. This was originally written to handle slave DBs getting bogged down by load, but with our new faster DBs, that just never happens, so if a slave falls behind, one of us probably typed something dumb at the mysql> prompt.)
(e) We bolted on memcached support. Why bolted-on? Because back when we first tried memcached, we got a huge performance boost by caching our three big data types (users, stories, comment text) and we're pretty sure additional caching would provide minimal benefit at this point. Memcached's main use is to get and set data objects, and Slash doesn't really bottleneck that way.
Slash 1.0 was written way back in early 2000 with decent support for get and set methods to abstract objects out of a database (getDescriptions, subclassed _wheresql) -- but over the years we've only used them a few times. Most data types that are candidates to be objectified either are processed in large numbers (like tags and comments), in ways that would be difficult to do efficiently by subclassing, or have complicated table structures and pre- and post-processing (like users) that would make any generic objectification code pretty complicated. So most data access is done through get and set methods written custom for each data type, or, just as often, through methods that perform one specific update or select.
Overall, we're pretty happy with the database side of things. Most tables are fairly well normalized, not fully but mostly, and we've found this improves performance in most cases. Even on a fairly large site like Slashdot, with modern hardware and a little thinking ahead, we're able to push code and schema changes live quickly. Thanks to running multiple-master replication, we can keep the site fully live even during blocking queries like ALTER TABLE. After changes go live, we can find performance problem spots and optimize (which usually means caching, caching, caching, and occasionally multi-pass log processing for things like detecting abuse and picking users out of a hat who get mod points).
In fact, I'll go further than "pretty happy." Writing a database-backed web site has changed dramatically over the past seven years. The database used to be the bottleneck: centralized, hard to expand, slow. Now even a cheap DB server can run a pretty big site if you code defensively, and thanks to Moore's Law, memcached, and improvements in open-source database software, that part of the scaling issue isn't really a problem until you're practically the size of eBay. It's an exciting time to be coding web applications.
Could we have a better run-down of what unpatched software is running on the server? It would really help. Thanks.
Yay for segregated caching, where one machine gets data before the others do...
If I have nothing to hide, don't search me
I've always wondered, what's happening system-wise when we see "nothing for you to see here" vs. "page not found"?
stuff |
c) We don't use placeholders. Originally because DBD::mysql didn't take advantage of them, and now because we think any speed increase in a reasonably-optimized web app should be a trivial payoff for non-self-documenting argument order. Discuss!
I guess speed might be one consideration. Generally I like to use place holders because it adds simplicity when passing some types of queries. If there's one thing I've seen a problem with, it's failing to properly sanitize incoming information that is passed to the database. A LOT of php code out there is rather easy to blow a hole through due to this. It also simplifies a lot of junk I'd rather not deal with like quoting and such. In either case you're an idiot if you don't sanitize everything first anyway, but my mantra is safety first. I actually loath doing many applications where I can't use them (like the ruby database libraries).
Well that's my take anyway. There's some rather nice code in slash that taut me some better methods in perl, and I'd say you guys are way above my level.
But what version of Windows Server are you running all this on? *duck*
I remember when MOD was an audio format, and DOS wasn't a network attack....
You have to be a subscriber, and go directly to https://slashdot.org/login.pl.
If this is the place for suggestions and complaints for Slashdot, may I put in my two cents? Sounds like people have been suggesting a number of new mods and changes to the old ones. Could this be revised? For example:
... but maybe we'll take it one step at a time.)
- separating +1 Funny into "+1 Funny-Raise Karma" and "+1 Funny-Karma Unaffected"
- combining -1 Flamebait with -1 Troll
- adding -1 Wrong (or -1 Misinformed), the opposite of "+1 Informative"
- combining +1 Interesting with +1 Insightful
(In fact, we can have diametrically opposed mods, like Informative/Misinformed, Underrated/Overrated, Insightful/Flamebait, Funny/Rude, etc.
And, yes, put in my vote for logging in via HTTPS.
404555974007725459910684486621289147856453481154 in hex is "You sank my Battleship?"
[GPG key in journal]
How about not putting ads on a login page?
I think your justification for not using placeholders is rather, uh, wrong. I agree that the argument order thing can be an issue, but that's what *named* placeholders are for. The major benefit of placeholders is not speed, it's absolute resistance to SQL injection. You may be diligent in quoting, but standard software development wisdom is that it's always better to eliminate the possibility of a bug than rely on programmer checking all the time, due to Murphy's Law of course. Also, the admittedly small speed benefit of placeholders when used with precompiled statements is going to be on the DB side, not the web side, which can make a bigger difference (as the DB is harder to scale).
main(c,r){for(r=32;r;) printf(++c>31?c=!r--,"\n":c<r?" ":~c&r?" `":" #");}
'cause I see no reference to IIS, SQL Server, .NET and I just read a Microsoft press release that says high-volume web portals can't provide high-availability service w/o them??
Please do download our code (and email us at security@slashcode.com if you find any bugs)
Nah. I've submitted bugs to slashcode at sourceforge in the past. Actual bugs. They are always closed with snarky remarks about how it's not a bug. You should add a status "Will Not Fix" or "Cannot Fix". At least admit where your site is broken and maybe give a little explanation as to why.
The most obvious of these is how nested mode is horribly broken in stories with a lot of comments or a comment that receives a large number of replies such that it is beyond the comments per page threshold. If you try to go to the next page, you get the same page of comments. Same with the next page and the next. A side effect is that many comments are completely lost in this mode. This has been a bug for many years.