Is There A Network Equivalent Of Alt-SysReq?

Random Q. Hacker asks: "Alt-SysReq has saved me from filesystem damage and runaway processes more than once. Unfortunately, several of the machines I admin are thousands of miles away in data centers, and it takes 15 minutes for data center personnel to go to our cage, hook up a monitor to the right system, and call back for interactive troubleshooting. I have played with snmpd, but it's a userspace daemon, and most of its functionality involves executing external programs and accessing files. Sometimes a system gets hung so bad (say, on root becoming unavailable, or memory becoming completely full) that the only thing still working is the kernel itself. Is there a kernel backdoor (as in a patch) that could let me have (secure, authenticated) SysReq functionality through the network?"

Comments

Not quite what you asked for, but ...

[John Jorsett]

I've had similar problems in the past, though in my case it was that I didn't have security clearances high enough to get into the facility where the machine I developed resided. What we ended up doing was to set up an encrypted phone line on which I could dialup the remote system and connect to the console port. In my case, if the system had to be reset, I had a user at the other end do it. However, there are devices which allow you to do power cycling or equipment reset via the telephone (there's one called Power Mate over at Blackbox Catalog). If your remote site can dedicate a phone line to the equipment, an encrypted modem call and a remote reset capability might be the way to go.

Another choice might be to insert a terminal server over at the remote end, connected to the lan on one side and the server's console port on the other ("10/100 Serial Server" over at Blackbox). It wouldn't give you a remote reset capability, but you'd be able to control the server no matter what state it was in, short of total unresponsiveness.

Re:Not quite what you asked for, but ...

[dubl-u]

I'm also not quite sure what the original poster wants, but when a box in a cabinet gets so unhappy that it's not controllable via the network, I use APC's MasterSwitch, which is basically and eight-plug power strip with an ethernet jack and the ability to turn the outlets on and off individually.

I have also heard interesting things about the watchdog cards that have Linux drivers in the kernel, although I've never used them myself.

Both these solutions strike me as ugly, but I guess it beats driving to a colloc in the middle of the night.

Break

[The Madpostal Worker]

If its PC hardware, sending a break over the serial port does the same as SysRq, so if you got a console server(these let you use serial ports over the network, you usually telnet into them and then pick a port) you could do all the Magic SysRq stuff remotely... Combine this with a somthing to do remote poweroffs, a modern bios that does serial console stuff (lets you config the bios), a properly configured bootloader, and a properly configured kernel and you'll never need a physical console (excluding hardware failures).

/*
*Not a Sermon, Just a Thought
*/

Re:I never knew what SysReq did anyway.

[coyote-san]

The easiest way to answer this is to compare it to the "break" signal on serial ports, or the hook "flash" on telephones. The SysReq key is (or is supposed to be) an equivalent "out of band" signal from the keyboard and should always be recognized even if the keyboard buffer is full, hosed, or otherwise unusable.

What do you do with "SysReq"? Anything you want. On some systems, particularly in a "secure" environment, the "SysReq" key is how you get a login prompt because it is how you can ensure that you're seeing the real login program, not a password sniffing userspace front-end. That's the thinking behind WinNT using "Ctl-Alt-Del" to bring up the login screen.

On Linux, the kernel can be configured to bring up a very small "monitor" that allows you to perform a few tasks (e.g., sync'ing the hard disks and performing a clean shutdown) when all else fails.

I don't believe any handlers are installed for Windows non-NT or DOS.

Re:Serial console

[awx]

Ahh, that would be the RealWeasel: http://www.realweasel.com/intro.html

Reports i'm hearing say it's absolutely fantastic, especially if you're on an non-top end server without serial-line BIOS availability. Downside? It's a bit pricy for the average user, but if you can afford a colo, you can probably afford one of these too...