Slashdot Mirror

← Back to Stories (view on slashdot.org)

We're Experiencing Technical Difficulties (Again)

Posted by CmdrTaco on from the please-bash-my-head-in-with-a-brick dept.

Proof once again that I shouldn't be allowed anywhere near a root password, Slashdot's httpd has begun crashing. It dies about every 4 minutes for no apparent reason. Nothing shows up in any of the logs. I haven't changed a single thing on this damn machine since last wed, and this started yesterday, so its either gremlins or script kiddies. Anyhoo, please hang in there- I'm working as fast as I can. I'm going to be shuffling around some hardware soon (including a much faster box for Slashdot) so hopefully that will help. This puts a delay on the new moderation system (grr) but I'll get to it. I'll be a bit balder but I'll get to it. I gotta hire a sysadmin. Ugh. Update: 03/08 01:15 by CT : Please stop sending resumes!

46 of 105 comments (clear)

  1. btw by CmdrTaco · · Score: 2

    Wrong on both counts.
    Rob "CmdrTaco" Malda
    Pants are Optional

    --
    Pants are still optional, but recommended for you.
  2. redhat by alexandre · · Score: 1

    wasnt that near the 500k hit/day?

    ---

  3. something wrong with linux/apache? by sar · · Score: 1

    Linus was just a small tot when OSS was concieved.

    --
    .
  4. heh by drwiii · · Score: 1

    Look on the bright side.. At least you're not administrator of an NT box.

  5. Cachedot Curiosity by jbrw · · Score: 1

    I noticed that cachedot accesses http://www.slashdot.org/ and not http://slashdot.org/

    cachedot is evil - *eeeeevviiiiiiilllll*

    or maybe not

    ...j
    (I hope this was setup by a non-BSI chap/chapette)

  6. Cachedot Curiosity by jbrw · · Score: 1

    My (not at all interesting) point is that normally people 'round here get hyper about publishing the URL as http://www.slashdot.org/ and not http://slashdot.org/

    Interesting, huh?

    ...j

  7. redhat by gavinhall · · Score: 1

    Posted by neuralfraud:

    Doesnt slashdot run redhat?

    I knew that dist was funky, I using stampede have never experienced a httpd failure, but then again 250 hits/day doesnt compare to 25000 :)


  8. Something to try by Rasmus · · Score: 1

    Rob, is it individual httpd's crashing, or is the main root-owned process disappearing on you? Regardless, try attaching gdb to one of your processes and see what happens when it goes down.
    You may be able to catch a seg fault or a bus error and then get a backtrace to get some idea of where things are going wrong.

  9. sysadmin by alta · · Score: 1

    Exactly where do I send my resume?

    --
    Do not meddle in the affairs of sysadmins, for they are subtle, and quick to anger.
  10. No Subject Given by tjones · · Score: 1

    Update: 03/08 01:15 by CT: Please stop sending resumes!

    Hmmm... Why not start (yet another) resume and job opening listing service?

    "Slashjobs, Jobs for Nerds, Pay that Matters"

    Nevermind.

  11. Microsoft.com by Matts · · Score: 1

    The key to Microsoft.com's stability is clustering. BIGTIME clustering. I think with a cluster a 99.5% uptime isn't that great - it means that at some point in time, for a few hours in the year, all the servers in the cluster are down. Not good.

    --

    Matt. Want XML + Apache + Stylesheets? Get AxKit.
  12. something wrong with linux/apache? by dangermouse · · Score: 1

    Yeah, run Slashdot on your NT box. Go for it.

  13. Where do I send my resume? by planet_hoth · · Score: 1

    Hopefully you don't want it in .doc format...
    (that's for the sysadmin job, btw)

    --

  14. Pepsi? by planet_hoth · · Score: 1

    It's my statement on rampant consumerism. The irony
    is that I don't even really LIEK pepsi. Go figure.

    --

  15. Stop using one machine to run Slashdot by cbj · · Score: 1

    I think the most long term solution at this point
    is to start thinking server farm as opposed to a
    single server and a 'cache' server.

    *shrug*

    The problem with httpd crashing is nifty. You'll have to get a stacktrace or other information in order to fix it though unless it gets fixed through blind luck.

    Brian

  16. Stop using one machine to run Slashdot by cbj · · Score: 1

    Doubt this will get read, but there are things like a Cisco redirector which will redirect a single ip to multiple private server machines. Load balancing solutions exist which can tie into an OS's system load to more intelligently balance but I'm not sure if anything supports the Linux kernel.

    Brian

  17. microsoft.com has a ton of Servers.. /. has 1 by CrAlt · · Score: 1

    micosoft.com uses a ton of NT servers to keep it running...when one crashes the rest take up the slack...also, I dont think microsoft.com runs the stock NT. They seem to have their own version. Ever Queso them?

    slashdot.org is 1 server (well...there is cachedot, but i dont think that does much other then cache).

    --
    I have to return some videotapes...
  18. Cachedot? by Puff · · Score: 1

    What is Cachedot? I have never heard of it before.

  19. The reciprocal Slashdot effect by Lettuce+B.+Qrious · · Score: 1

    This makes a lot of sence! A reciprocal Slashdot effect, costing Rob his hair...

  20. MY RESUME by unitron · · Score: 1

    You forgot to mis-spell "request"


    --

    I see even classic Slashdot is now pretty much unusable on dial up anymore.

  21. At least you aren't running NT by unitron · · Score: 1

    Are you sure that those 200 people would have gotten through if he had been running NT?


    --

    I see even classic Slashdot is now pretty much unusable on dial up anymore.

  22. No doubt by unitron · · Score: 1

    No doubt the site is groaning under the burden of grateful slashdotters everywhere constantly flooding Slashdot.org with messages of appreciation for such a fine and free service to us all and of admiration for all those who make it possible. Yeah, that's gotta be it!

    --

    I see even classic Slashdot is now pretty much unusable on dial up anymore.

  23. Rob, say it ain't so... by Anonymous+Commando · · Score: 1

    Rob, please tell us that you weren't actually wearing a Microsoft shirt. Tell us that it was digitally "airbrushed" in... tell us that aliens kidnapped you and put it on you, and the picture was taken just as you were coming to... tell us that isn't a disguised Monica Lewinsky in the other picture...
    ________________________

    --
    Corporate Jenga: You take a blockhead from the bottom and you put him on top...
  24. Did someone say "job?" by Signal+11 · · Score: 1



    Where can I send my resume? >:)



    --

  25. Blah - forget Linux alltogether by JB · · Score: 1

    Why use Linux? Just switch to FreeBSD...or at the very least give it a try.

    JB

  26. Slashdot slashdotted? by jhage · · Score: 1

    I know! It's the ghosts of all those servers that have melted down over the months come back to wreak their revenge! It makes perfect sense.

    Anyway, I still love ya'll. I can accept some rocky roads in this affair.

  27. Cachedot Curiosity by hagan · · Score: 1

    so shoudl this be changed?

  28. Something to try by orabidoo · · Score: 1

    and make sure you allow the damn thing to dump core, and know hwere the cores will go. as a last resort, strace -p to random httpd processes (not the master one, unless you have 10x the cpu /. currentyl takes!) see if you can catch where the fault happens.

  29. hire me!! please! by azonic · · Score: 1

    please.. must... move... somewhere... else!

    any format you prefer for applications? :)

    I only live like 3.5 hours away, (ever heard of Boyne City, MI)

  30. Are you logging port accesses? by mattc · · Score: 1

    Also that program really starts thrashing around when you are getting a lot of accesses. It would kill slashdot's performance.

  31. Are you logging port accesses? by Cadaver · · Score: 1

    Perhaps you should try out iplog. Although the logs for /. would probably be horrendously huge.

    --
    I ate something that disagreed with me. Maybe I should have cooked him first.
  32. Are you logging port accesses? by Cadaver · · Score: 1

    In case you hadn't noticed, something is already killing /.'s performance, every four minutes ;-)

    --
    I ate something that disagreed with me. Maybe I should have cooked him first.
  33. I log on as root all the time! by Stardate · · Score: 1

    Since I first started playing with linux in '95... of course my machine is definately NOT production and I've broken it many times, but I really like that feeling of power! :)

    --
    "... I declare our city to be a free and independent state to be named Tri-Insula!" --Fernando Wood, Mayor of NYC 1861
  34. network unreachable? by redhog · · Score: 1

    I think I recognize that small text :)
    Just got it on my own screen, or, at least, a similar one...

    --
    --The knowledge that you are an idiot, is what distinguishes you from one.
  35. At least you aren't running NT by bsandlin · · Score: 1


    If you were running NT, I would have to look at 200 people bitching about how bad NT is.

  36. If them be script kiddies... by Bryan+Andersen · · Score: 1
    If you find out it's hacking, I'd sugest a nice sturdy firewall built on a 2.2 kernel with IPCHAINS and IPPORTFW. After going live on the net I had some problems with hackers. Placing an IPCHAINS/IPPORTWD based firewall inbetween my systems and the world has really helped secure my home network. It really narrowed down the number of configuration files I have to set security up in. Now if I don't want X site to be able to access my network, I just deny it in the IPCHAINS config. I've got it enabled on a couple of sites due to their sysadmin being in denial or totally clueless. Currrently I only have a couple of ports open, and can open/close ports on an as needed basis. The rest of them are denied without even a NACK.

    It's interesting to see all the accesses against services with known problems. I'm surprised how many times someone tries to use a socks proxy server on my firewall when there isn't one available. The other fun thing is the reactions of sysadmins to my telling them their system was compromized. Currently I log all SYN connection request packets, and all packets to some ports. All logging goes via klogd/syslogd so it can be remotely logged on a log host.

    As for speed, it seams to be keeping up nicely with a DSL link to the outside, and transfers from my local net to the DMZ net over 100mbit connections.

    It may not be a panacea, but it's cheep, and can run on an antiquated system. I'm using a P-100 with 4 PCI slots and 24M Ram, and a 100MB HD.

  37. NACK and TCP?? by Bryan+Andersen · · Score: 1

    As I said, I only had a couple of ports open, Auth(113) is one of them...

  38. A Tracing Approach to Deterring Attacks by Bryan+Andersen · · Score: 1
    I've often though it would be nice to have a site for tracking sites where attacks have come from. One problem is determining if an attack is using spoofed addresses or not. There are other issues like falsified reports. As for addressing some of the problems I see, requireing posters to use verifiable PGP sdignatures would help with some of the integrity issues, but wouldn't fix them all.

    I know the probes I've see on my system come from many different systems, and only probe one port per system probing. I'd like to know if X system is doing probes against other systems.

  39. something wrong with linux/apache? by Nassah+The+Zerg! · · Score: 1

    Try running ANYTHING on an NT box for a week.
    And then tell me if you had no problems. I will buy an NT license the next day!
    Btw, anything is anything worthy of running.
    I have not seen Office run all day without by end of day tell me I don't have enough space to save a 10 page .doc even though I had 1gig of space!

    --
    The kernel needs a Gtk/Gnome-based post-install device configuration tools "a la" make xconfig. (Better sig coming soon
  40. Ain't payback a b*tch? by Nassah+The+Zerg! · · Score: 1

    I don't know if you would believe it. BUT I don't hate MS. I hate Windows. I love Visual Studio. I hate Windows, 95/98/NT. They aren't reliable. PERIOD>

    --
    The kernel needs a Gtk/Gnome-based post-install device configuration tools "a la" make xconfig. (Better sig coming soon
  41. Tired of Windows whiners about Linux! by Nassah+The+Zerg! · · Score: 1

    Do the following:
    (1) Erase disk!
    (2) Make a clean install of Windows.
    (3) Download Regclean.
    (4) Run Regclean
    (5) Fix Registry errors!!!!!!!!!!!!!!

    --
    The kernel needs a Gtk/Gnome-based post-install device configuration tools "a la" make xconfig. (Better sig coming soon
  42. something wrong with linux/apache? by Willy+K. · · Score: 1

    yeah...Richard Stallman would
    be pissed at the idea that Linus
    invented the OSS concept. :)
    Someone didn't read that article posted
    this weekend of his interview!

  43. Are you logging port accesses? by i+ronin · · Score: 1

    Perhaps a general intrusion detection system would be a good approach if you're concerned that it might be script kiddies. ISS makes a good one, but then I'm biased :-) Network Flight Recorder would probably also be a good one though I have no direct experience with them.

    I think that you can download an evaluation copy of ISS' RealSecure from http://www.iss.net. Or,
    NFR is at http://www.nfr.com. They say that they have eval copies for download.

    Good Luck

  44. Just switch to NT. by Loof · · Score: 1

    Yep, he'll just shoot himself...
    His problems will ALL go away :P

    Actually, NT is convenient... With loads like Slashdot it reboots all by itself every few hours... So it eliminates these problems :)

  45. It's there in all it's glory..... by KlTheKiten · · Score: 1

    Just as any follow the leader organization, RH has mirrored the post on it's front page....

    --

    ...some days you're the dog, some days you're the hydrant...
  46. where do we send resumes? by trey · · Score: 1

    where do we send resumes? :>

    --

    he who has the fastest cart always has the best lie.