Is There A Network Equivalent Of Alt-SysReq?

Random Q. Hacker asks: "Alt-SysReq has saved me from filesystem damage and runaway processes more than once. Unfortunately, several of the machines I admin are thousands of miles away in data centers, and it takes 15 minutes for data center personnel to go to our cage, hook up a monitor to the right system, and call back for interactive troubleshooting. I have played with snmpd, but it's a userspace daemon, and most of its functionality involves executing external programs and accessing files. Sometimes a system gets hung so bad (say, on root becoming unavailable, or memory becoming completely full) that the only thing still working is the kernel itself. Is there a kernel backdoor (as in a patch) that could let me have (secure, authenticated) SysReq functionality through the network?"

Comments

Not quite what you asked for, but ...

[John Jorsett]

I've had similar problems in the past, though in my case it was that I didn't have security clearances high enough to get into the facility where the machine I developed resided. What we ended up doing was to set up an encrypted phone line on which I could dialup the remote system and connect to the console port. In my case, if the system had to be reset, I had a user at the other end do it. However, there are devices which allow you to do power cycling or equipment reset via the telephone (there's one called Power Mate over at Blackbox Catalog). If your remote site can dedicate a phone line to the equipment, an encrypted modem call and a remote reset capability might be the way to go.

Another choice might be to insert a terminal server over at the remote end, connected to the lan on one side and the server's console port on the other ("10/100 Serial Server" over at Blackbox). It wouldn't give you a remote reset capability, but you'd be able to control the server no matter what state it was in, short of total unresponsiveness.

Re:I never knew what SysReq did anyway.

[coyote-san]

The easiest way to answer this is to compare it to the "break" signal on serial ports, or the hook "flash" on telephones. The SysReq key is (or is supposed to be) an equivalent "out of band" signal from the keyboard and should always be recognized even if the keyboard buffer is full, hosed, or otherwise unusable.

What do you do with "SysReq"? Anything you want. On some systems, particularly in a "secure" environment, the "SysReq" key is how you get a login prompt because it is how you can ensure that you're seeing the real login program, not a password sniffing userspace front-end. That's the thinking behind WinNT using "Ctl-Alt-Del" to bring up the login screen.

On Linux, the kernel can be configured to bring up a very small "monitor" that allows you to perform a few tasks (e.g., sync'ing the hard disks and performing a clean shutdown) when all else fails.

I don't believe any handlers are installed for Windows non-NT or DOS.

Re:Serial console

[awx]

Ahh, that would be the RealWeasel: http://www.realweasel.com/intro.html

Reports i'm hearing say it's absolutely fantastic, especially if you're on an non-top end server without serial-line BIOS availability. Downside? It's a bit pricy for the average user, but if you can afford a colo, you can probably afford one of these too...