Slashdot Mirror

← Back to Stories (view on slashdot.org)

Running a UDP Remote Console with Linux 2.6

Posted by simoniker on from the so-there dept.

Bruce Perens writes "Many system admins have learned how to use syslog to log events of remote systems. But when the kernel crashes, its final messages never get to syslog. If you don't have a remote console manager, you won't see them, unless you run netconsole."

39 comments

  1. First Post? by extremescholar · · Score: -1, Offtopic

    This is my second shot at a first post! Hey, someone's gotta do this...

    --
    Using the Freedom of Speech while I still have it.
    1. Re: First Post? by pizza_milkshake · · Score: 5, Funny

      did you send them via UDP? perhaps your first attempt hasn't made it yet.

    2. Re: First Post? by Bruce+Perens · · Score: 0

      A lot of folks here wouldn't get the joke. :-)

      --
      Bruce Perens.
  2. Tempting... by dmayle · · Score: 1

    I've got to say, it looks really tempting. Things don't usually go that far south, but when they do, it's definitely an advantage to have every piece of information you can get. Especially the kernel messages. Unfortunately, until the clietn daemon can be fixed to work with the latest version of netconsole, this is pretty much worthless, because I can imagine few people willing to leave netcat running on a production machine 24-7...

    1. Re:Tempting... by Bruce+Perens · · Score: 3, Funny
      Writing another client daemon would not be a big job. This is at most a 200-line program. No doubt a Perl hacker could do it in one line - but I might lose my lunch from looking at that :-)

      Bruce

      --
      Bruce Perens.
    2. Re:Tempting... by Anonymous Coward · · Score: -1, Flamebait

      Please what that, you could cost yourself some consulting contracts and that would be a Bad Thing.

      Bruce Berens, #1 in self-promotion and charging money for the work of others. What have you done, in the past 1000 days, that improved Linux? Anything? And no, a distro to replace RedHat/Suse because they won't pay your consulting rate isn't doing something positive, it is just being petty.

    3. Re:Tempting... by hattmoward · · Score: 4, Informative

      It looks like the packet format is syslog compatible in the recent 2.6 kernels. I think the work is already done. =)

    4. Re:Tempting... by Bruce+Perens · · Score: 1
      My syslog daemon (in Debian sarge) doesn't have a switch to tell it what hosts are OK to listen to. It just has a big on-off switch for accepting connections from the network (which seems to mean anything but localhost).

      Bruce

      --
      Bruce Perens.
    5. Re:Tempting... by hattmoward · · Score: 3, Informative

      Neither does the syslogd provided with slackware. The snippet below will allow you to inclusively list log hosts.

      iptables -A INPUT -s $LOGHOST -p udp --destination-port 514 -j ACCEPT
      iptables -A INPUT -p udp --destination-port 514 -j DROP

      You can repeat the first line as many times as needed, replacing $LOGHOST with an IP or a resolvable domain name. Just make sure the DROP rule goes last.

    6. Re:Tempting... by Bruce+Perens · · Score: 1
      I posted a comment about this below the article on technocrat, thanks.

      I must confess to being surprised that syslog can't just be told to accept packets only from a list of IPs.

      Bruce

      --
      Bruce Perens.
    7. Re:Tempting... by hattmoward · · Score: 4, Interesting

      It may not be as surprising when you note that sysklogd was written in a time where the network was considered more trustworthy, though there is another reason, which is mentioned below. You should also find the syslog-ng package available on your Debian system, which can filter by sending host, although it will spend more CPU time to do the blocking at that level. Both solutions work, though.

      The reason you don't see this feature made more prominent is that UDP packets are easily spoofed, since there is no handshake required to get the payload through. With TCP, someone has to be at the "sending" host to reply to the handshake, while UDP simply accepts what it receives in the single packet (barring the sender being a local-net host, with strict ARP checking on the receiver). So anyway, filtering hosts doesn't completely cover you from log spoofing or DOSing, but it's still a good idea.

    8. Re:Tempting... by Bruce+Perens · · Score: 2, Interesting
      That's a good answer, thanks! I had not thought about the ease of spoofing a UDP packet. I'm actually surprised that so far I am not losing any of them between Texas and California.

      If I had another machine in the same data center, I'd not be sending UDP over the internet.

      Thanks

      Bruce

      --
      Bruce Perens.
    9. Re:Tempting... by bill_mcgonigle · · Score: 1

      The reason you don't see this feature made more prominent is that UDP packets are easily spoofed

      I've had good luck with syslog-ng's TCP support. That's what brought me to syslog-ng in the first place.

      Or is that what you were getting at?

      --
      My God, it's Full of Source!
      OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
  3. Huh? by Anonymous Coward · · Score: -1
    What? Lunix doesn't crash! This Pernes guy is obviously another Micro$oft astroturfer.

    So technocrat.net is still running, huh? I thought it had been completely colonized by the ship-jumping Slashdot Journal crowd...

    1. Re:Huh? by Bruce+Perens · · Score: 3, Funny
      Well, I've got to say that I switched from the stable 2.4 kernel to the unstable 2.6 one to use netconsole, and so far 2.6 hasn't crashed and I haven't been able to see how far into the panic I'll get the messages. The system is being slashdotted now, maybe that will help.

      Bruce

      --
      Bruce Perens.
    2. Re:Huh? by consolidatedbord · · Score: 0, Troll

      Lunix?

      You're obviously a BSD astroturfer. :)

      --
      while true ; do echo this is my sig; done
  4. some conditions.. by molo · · Score: 4, Informative

    This only works/is helpful in certain situations:

    1. network driver is initialized
    2. kernel messages are what you want to capture
    3. the kernel doesn't get itself foobarred so badly that it can't send packets
    4. interrupts are enabled (right? can't talk to the network card otherwise)

    Also, if you change mac addresses or network cards of the first hop between the sender and receiver, this all needs to be reconfigured.

    This can certainly be useful, but recognize its limitations.

    -molo

    --
    Using your sig line to advertise for friends is lame.
    1. Re:some conditions.. by Bruce+Perens · · Score: 2, Interesting
      Yes. I really want a console server. I can't get one without paying a lot more or getting less for what I'm paying.

      Bruce

      --
      Bruce Perens.
    2. Re:some conditions.. by molo · · Score: 4, Informative

      Yes, serial console servers are quite useful for remote administration.

      Another option might be to build your own - take an older spare machine and stick a PCI USB controller in it and add a bunch of USB-to-serial dongle devices (possibly with a USB hub). These show up as /dev/ttyUSB#.

      You can then ssh in and use minicom or whatever. You get added security since many terminal servers don't support ssh, plus you have a fully configurable system to manage stuff. You could write a program to page/email you on certain kernel events for example. It could be pretty flexible if that is what you need.

      Just my two cents.

      BTW, Bruce, thanks for your contributions to the community and to Debian. We all appreciate it.

      -molo

      --
      Using your sig line to advertise for friends is lame.
    3. Re:some conditions.. by Paul+Jakma · · Score: 4, Informative

      1. Correct
      2. Correct, the purpose of netconsole being to capture kernel messages that otherwise would not be sent due to a hung kernel - all other messages, syslog can take of.
      3. Unlikely. If the kernel can panic, it can use netconsole. Used with nmi_watchdog, you can even get a stack trace from a stuck interrupt handler
      4. No, netconsole uses polling-mode and drives the nic driver directly.

      Note that netconsole can also dump the entire contents of RAM.

      --
      I use Friend/Foe + mod-point modifiers as a karma/reputation system.
    4. Re:some conditions.. by molo · · Score: 1

      3. Ok, seemingly hard lockups are out - but these are more and more uncommon with 2.6 - but nearly everything else is dumpable, unless it screws with the NIC driver itself? I'll have to check out that watchdog, that sounds useful.

      4. Thanks for the info, I'll have to read up on that.

      -molo

      --
      Using your sig line to advertise for friends is lame.
    5. Re:some conditions.. by andreyw · · Score: 1

      Interrupts are used to *receive* - i.e. they provide the kernel with the notification that *something* happened to the NIC.

    6. Re:some conditions.. by Paul+Jakma · · Score: 1

      3. Ok, seemingly hard lockups are out - but these are more and more uncommon with 2.6 - but nearly everything else is dumpable, unless it screws with the NIC driver itself? I'll have to check out that watchdog, that sounds useful.

      What do you mean by "hard lockups" exactly? NMI cant be disabled AIUI, so unless hardware were hung, there's a good chance you can get a panic.

      --paulj

      --
      I use Friend/Foe + mod-point modifiers as a karma/reputation system.
    7. Re:some conditions.. by emag · · Score: 2, Insightful

      3. Ok, seemingly hard lockups are out - but these are more and more uncommon with 2.6 - but nearly everything else is dumpable, unless it screws with the NIC driver itself?

      Well, a friend of mine using 2.6 is experiencing random lockups (we suspect either hardware problems, or ghosts), with nothing ever logged. However, he can still ping the machine even after it's locked up, but the console also doesn't respond. I've pointed him at this article, and he's going to try enabling netconsole to see if anything from the kernel is getting lost. Hopefully it'll help figure out what the problem is.

      --
      "The urge to save humanity is almost always a false front for the urge to rule." --H.L. Mencken
    8. Re:some conditions.. by Zork+the+Almighty · · Score: 1

      Offtopic, but I love your use of when in the article summary :)

      --

      In Soviet America the banks rob you!
    9. Re:some conditions.. by Bruce+Perens · · Score: 1

      Of course, it's always a hardware problem :-)

      --
      Bruce Perens.
    10. Re:some conditions.. by molo · · Score: 1

      If he can ping the machine, the kernel is still functional. If he can ssh in, then userspace is still functional and X or the console is locked up.

      Is he using X, a framebuffer console or a standard VGA console? I've had plenty of times where X locked up on me and I wasn't able to recover the video device. This is not uncommon on some hardware. Best bet at that point is to ssh in and reboot.

      If X locks up, there is nothing to be logged. The kernel sees it as just another userspace process.

      -molo

      --
      Using your sig line to advertise for friends is lame.
    11. Re:some conditions.. by emag · · Score: 1

      X isn't running on that particular system. Existing ssh sessions freeze. New ssh sessions can't be initiated. FWIU, no new TCP sessions can be initiated. No clue if it's framebuffer or VGA console, but apparently C-A-# works to switch consoles, though the rest of the keyboard doesn't respond at all. So like I said, it sounds like some sort of bad hardware, or ghosts. I should also mention this only started happening with 2.6. Under 2.4 it was rock solid.

      --
      "The urge to save humanity is almost always a false front for the urge to rule." --H.L. Mencken
  5. Same thing as sending syslog to remote loghost? by menscher · · Score: 2, Insightful
    I really don't see much benefit here. Seems like someone just took syslogd and made it a kernel module. Just one more piece of the kernel to crash. Personally I'd rather keep the kernel as small as possible. Less to go wrong that way.

    While on the topic, though, it would be nice if Linux did something similar to IRIX, where a crash would save the kernel coredump to the swap partition. Yes, I know the Linux Kernel Crash Dump project does this, but they're not exactly stable yet (hell, their website won't even load right now).

    1. Re:Same thing as sending syslog to remote loghost? by drdink · · Score: 4, Insightful

      This is a useful tool for those of us who have a slew of machines on a single network segment and don't want to have an expensive KVM or 47 monitors. The ability to put your consoles over the same wire you put your normal network traffic over is great. It is like those cable companies that also provide phone service over your cable line, sort of...
      IRIX isn't the only operating system that does crashdumps. I've found them to be an invaluable tool for debugging FreeBSD. It has supported them for quite a long time. The ability to gdb your kernel and the vm after you reboot and see exactly where things went wrong is great. I still don't understand why Linux lacks this.

      --
      Beware, Nugget is watching... See?
    2. Re:Same thing as sending syslog to remote loghost? by Thing+1 · · Score: 0, Offtopic

      A slashdotting of one?

      --
      I feel fantastic, and I'm still alive.
  6. For FreeBSDers, try ethercons by drdink · · Score: 4, Interesting

    If you use FreeBSD and find this interesting, you should take a look at rwatson's ethercons patches. Basically, it is this but supports bidirectional communication. That means you can run a getty on it and login over ethernet console. Rather slick.

    --
    Beware, Nugget is watching... See?
  7. linux doesn't crash by Anonymous Coward · · Score: -1, Troll

    I'm confused... the article says something about the kernel crashing. This is linux, the kernel doesn't crash! Now make one of these for Windows so I can debug some BSODs and NOW we're talking!

  8. Blantant link whoring. by Anonymous Coward · · Score: -1, Offtopic

    Unacceptable. Mod parent down.

    1. Re:Blantant link whoring. by Anonymous Coward · · Score: 0

      It is in his sig, give me a break.

  9. Oh I'll see them... by rthille · · Score: 1

    remote console manager, you won't see them,

    My console is a 300 baud teletype you insensitive clod!

    --
    Awesome furniture, accessories and cabinetry in Santa Rosa, CA: http://humanity-home.com/
    1. Re:Oh I'll see them... by Bruce+Perens · · Score: 4, Funny
      My console is a 300 baud teletype you insensitive clod!

      THERE'S STILL LOTS OF LIFE LEFT IN MY 5-LEVEL BAUDOT MACHINE.

      BRUCE

      --
      Bruce Perens.
  10. SNMP trap? by professorfalcon · · Score: 1, Insightful

    Why not just send an SNMP trap?

  11. That's A Phrase You Never Hear by RogL · · Score: 2, Funny

    Never heard this before, probably never hear it again:

    "The system is being slashdotted now, maybe that will help."