Slashdot Mirror
Assorted Slashdot Updates
As you probably noticed, the Server really bogged
down today. The reason is quite simple: Katz's story,
with about 500 comments weighed in at about a meg.
70 httpd's trying to serve a one meg file (which takes 2-3 minutes
for most of you to download) perhaps a dozen times a second.
Do the math *grin*.
So I have a server level setting that will enforce Comment Limits
when we get bogged. This will annoy the heck out of
some how you, but it vastly speeds up page loads.
After that mess, I'm glad to have some good news: I
brought the 2500 stories online that I yanked awhile back when we were
getting overloaded (mostly stories from late 97 early 98)
so you can search for them again. More random musings
are attached below.
I changed the numbered links on the homepage (and
the word 'XX Bytes in Body') to link to the article instead of
to the comments page, so you can cleanly use them instead of
the 'Read More' link to browse at -1 or 'No Comments' mode,
and still read the article contents. Thanks to the 8
billion of you who asked for that one :)
I added a Default Comment Limit now. Originally the limit was 50,000 (in other words, never :) but I've changed that to 100. This won't happen much, but it will be a good line of defense when those stories get really huge. You can still change that number in your user preferences if it annoys you. Your user preference will always take priority, unless the server goes into Overload mode.
Finally, I wanna strut a bit: Since we brought the old archives back online, Slashdot now has over 5400 stories in the database. Over 3300 of them were posted by yours truly, which strangely seems to explain my lack of a social life at this point :) Anyway, I just thought that was cool.
Why not serve the topic images from the same server as the html, i.e. good old sebastian.slashdot.org. Ever since slashdot started getting really popular in late 1998 images.slashdot.org has been getting even slower than slashdot. That's where the topic images come, so rendering of web pages gets delayed too unless you browse with images turned off.
No. Say there are 300 comments. You will first go to a page with the 30 top comments (depending on how you have them sorted -- I use highest score first). At the bottom of the page you will see something like:
1 2 3 4 5 6 7 8 9 10
Meaning you are on the first page of 10, each with 30 comments.
I and others have said it before, but it seems to me it would hugely reduce Slashdot bandwidth if the comments were accessible via NNTP. Perhaps we could persuade Rob to work on that rather than submitting so many stories? Rob, I promise I'll buy a Slashdot T-shirt if you do...
If the stories themselves were only accessible via WWW, the banner hit count shouldn't be changed much. (I'm kind of surprised I haven't seen any rude remarks about the recent Ziff-Davis banners, BTW...:-)
Ooh, a sarcasm detector. Oh, that's a real useful invention.
slashdot has been bad for a while now, even while viewing the static pages (index.shtml). cachedot was much better for that. Please fix the dns entry. Thanks CT.
The problem isn't the OS, the problem is users on modems downloading large files. The way Slashdot's pages are put together, a caching proxy in front of the server (like cachedot) doesn't work, because even pages that really don't need any variance (or much variance) in content among users still have differences that keep caching from happening. My username (tgd) shows up on the page, even though it doesn't really need to be there.
Best way to improve performance other than using a more robust database and multiple servers that can generate dynamic pages is to restrict the number of possible combinations that a subpage can fall into, get rid of the stuff thats different between two users who have the same page settings (ie, get rid of the user-specific stuff), and make sure whatever differentiates it is in the PATH_INFO part of the URL so it caches, and the caching proxy doesn't think its a form submission. I bet Slashdot's got enough clout to get a donation of little ones from Corel or some place like that to throw Squid on.
Only the homepage really needs to be different per-user. The rest of the site should be comment pages in various combinations (for each of the sort methods, and a handful of comment options... moderator pages could be served by bypassing the cache servers... Even if every page was served to the proxy with a two or three minute timeout, you'd get good performance, and could spread the load on long-download pages among a few more servers... getting a few hundred simultaneous downloads instead of 70 (which seems low anyway... Apache can usually grok more than that, although maybe the mod_perl stuff can't...)
You silly doof! Don't let your expensive mod_perl+mysql processes sit there pushing bytes down 28.8 & 56K pipes. Use squid in reverse proxy mode to buffer the output of mod_perl and then let squid, which has extremely cheap threads, twiddle its thumbs waiting for the client to receive.
This is all well documented in recent discussions on mod_perl@apache.org. See also the mod_perl guide and the mod_perl mailing list archives.
See, even the mighty Linux can be brought down :)
;P)
(but then again, so can freeBSD, my OS of choice, or so I've been told
rob, have you considered using a multithreaded server like Xitami? That might improve performance somewhat.
Three Step Plan:
1. Take over the world.
2. Get a lot of cookies.
3. Eat the cookies.
How will we be able to tell if the server's in overload mode?
Oh. Never mind. We just hit overload mode, and it says there on the dividing bar.
Question: It says the comment limit is 30. Does this mean I'll miss all but the 30 most recent comments posted?
If so, that really sucks... I'll have to reload about every two minutes to avoid missing anything.
That Katz article hit a nerve. It got linked to from other sites. My girlfriend had been depressed about the shootings, so I thought I'd email her a link to the Katz story (not that it would lift her spirits or anything), but she had already read it! My girlfriend, the English major, had read an article on Slashdot without me pointing it out (which, if I recall, I've never done before anyway).
Has JonKatz's article surpassed the Linux 2.2 announcement to become Slashdot's busiest article ever?
In my opinion, it was the BEST article I've read on the Littleton, Colorado subject; the BEST article I've read by JonKatz; and the BEST news-related article I've read on Slashdot. We all give JonKatz hell for articles like his Linux newbie article... but this one was extremely well-written and hit a nerve with all of us. (Probably us "geeks" and "nerds" moreso than others. As well as those of us like myself, fresh out of high school and into college.)
Ryan
Um, folks -- with all due respect to BSD,
and using squid to reverse-proxy, what I
think Rob may have been getting at is this:
1 Mb file, served 12 times/sec.:
1,000,000 x 12 x 8 =~ 100 Mbit/sec. Last
time I checked, a T-3 was approx. 45 Mb/s.
You could be running a Cray, and as long
as your pipe ain't large enough, your pages
are gonna be slow.