Run Your Firewall Halted for Extra Security

n8willis writes: "There's a great article over at the SysAdmin magazine site that presents a unique approach to improving network security: run your firewall in a halted state. This means runlevel 0; no processes running and no disks mounted, but with packet filtering still on. The author heard a rumor of this capability in the 2.0 series kernels, and he's managed to get it working in 2.2 as well."

Comments

Works for me...

[Rorschach1]

Though I usually just use the power switch. Can't beat a powered-off firewall for security.

...and? We do this all the time

[TicTacTux]

See floppyfw. Does not even *need* a HD to boot. Floppies can be write-protected. Even an old 486 will do that trick.

Amazing what queer pranks people invent in place of a rather obvious solution...

Re:...and? We do this all the time

[moonbender]

This solution is more secure than a floppy would be. Write-access to the flopy can be forbidden, but all the other processes and security holes remain as possible targets.

Re:...and? We do this all the time

[TicTacTux]

Why? That whole system runs in runlevel 0. No user accounts, no login, no getty, no daemons - just the bare kernel plus an open console.

Disadvantage: You must employ your weary bones and walk over to do some maintenance...

Re:...and? We do this all the time

[gid]

Dang, why stop there, write something in ASM and flash it right on the bios. Wasn't there a linux bios project awhile back? :) I wonder how much progress they've made.

Re:...and? We do this all the time

[tricorn]

Y'all know that a halt() call that specifies a non-powerdown halt is basically a no-op, right? I mean, it does check that you're running as root, but then it does nothing:

From linux-2.4.14/arch/i386/kernel/process.c

void machine_halt(void)
{
}

(the machine_power_off routine also does nothing unless there is a poweroff function enabled; since, the halt isn't actually doing a poweroff (or it won't be doing a good job of routing after that) it presumably isn't set).

All that's being done by the process desc ribed in the article is a) bring down all running services except for the network; b) make init wait to die (I haven't looked at init to see if it waits for a normal ctrl-alt-del interrupt, then do a reboot, or if it just sets the kernel to handle ctl-alt-del and reboot all by itself). It doesn't even kill all other running processes that might be running, just the ones that were started by init (e.g. getty) and processes that remain in those process groups. If someone starts up a background job, it will still be there. There's nothing magic about run level 0 - it is just a convention that init uses.

Not all drives are unmounted - the root device will still be mounted read-only, maybe /proc and other pseudo-file systems will still be there. init is still running. This is a lower level of security than starting the kernel up by running a shell script that enables the network and ipchains, then sleeps forever. You could even have that shell script leave you in a plain old shell if you want to do be able to do something else (you'd have to know how to mount filesystems and remember to unmount them (since init won't be running, you can't just do a simple "shutdown" anymore), etc).

Here's an example script that might be used. Start it by passing "init=/netstart" (or wherever you put the script) to the kernel when you reboot:

#!/bin/sh
mount proc /proc -t proc
mount -o ro /usr
/etc/rc.d/init.d/network start
/etc/rc.d/init.d/ipchains start
umount /usr
exec /bin/sh

If there's a hole in the TCP stack itself, or in the IPchains routing, that opens up some vulnerability, it doesn't really matter if there's a simple /bin/sh running or not (if you really want to be secure, have the shell script exec a simple C program that checks to make sure it is PID 1; does a kill(-1, SIGKILL) and then sigsuspend() in a loop; now you're safe unless someone's hacked your kernel, libc or compiler).

I note that in 2.2, if process 1 exits, it kills the kernel; in 2.4, it gives a "Kernel panic: Attempted to kill init!". In either case, the effect is the same: ctrl-alt-del is disabled, and the kernel stops routing packets.

Another interesting consept: Invisible Firewall

[bob@dB.org]

introduction from http://www.openlysecure.org/openbsd/how-to/invisib le_firewall.html

In the stone age of firewalling, a firewall was a fairly complicated device that was less-than-trivial to factor into your network. It needed an IP address on it's outside, and another on the inside. This immediately created subnetting problems, forcing wasted IP allocation and overall disquietude amongst the cognoscenti. It also meant that your firewall was very visible to the world, and its function was rather obvious and easy to deduce. There had to be a better way. And now there is...

Dream with me for a moment: A black box, shimmering in the soft LED-green glow of the network cabinet. You take the network cable from your router, which previously went into a switch, and stick it in one of the snappy plugs in the back of the box. There's one more plug on the black box, so you grab another cable and hook the box up to the switch. You step back, and suddenly: everything looks the same. You go to all your computers. As far as you can tell from inside and outside the network, the box doesn't exist. It does nothing. A few minutes later, you have a monitor and keyboard hooked up to the back of the box. You quickly and easily begin to tweak a file that gives you fine grained control over access to your network. You shut off all access to your mailserver from the outside world except on port 25.

Re:Another interesting consept: Invisible Firewall

[douglips]

The problem with that is your LAN needs to be all on routable addresses. NAT won't work if the firewall has no IP address...

For exposed servers this would work well, not so much for the 18 Windows boxen you have Joe Salesguy using.

Re:Another interesting consept: Invisible Firewall

[Chang]

Who says the firewall and the nat have to be the same box? Just because this is usually one device doesn't mean it must always be that way

Logging?

[JimR]

Bit of a shame if you want to log any attacks
on the firewall though.

With no disks mounted where can you log it to?

Re:Logging?

[booyah]

Same place I log anyways

Dot matrix printer on LP1 :-)

try to erase THAT log :-)

-booyah

Re:Logging?

[Stary]

If you're seriously paranoid about it, get an old dot-matrix printer and hook it up as the logging output device. You'll have a logging device that nobody could alter no matter how compromised the system gets, without having to mount anything read-write.

Re:Logging?

[doorbot.com]

Bit of a shame if you want to log any attacks
on the firewall though.

If you can get this to work with IPFilter/PF you could use the "dup to" method to send the packet out a third NIC to a packet logging machine. Now you can have a transparent firewall, which is not accessible at all, but you can still have some logging features. One possible design for this would be two SBCs in a 1U case... one is the "halted" firewall and the other is the logging machine.

I don't know if this is possible with NetFilter or IPChains, however.

I figured this out in a much different way

[Binestar]

We were offering a linux-firewall, VPN solution to a hospital and turns out one of the machines we sent out had a bad token ring card, that mixed with an obscure bug in the Token ring driver, (Which has since been fixed) it would cause linux to die quite often. To the point where the computer would accept no keyboard input, would cease logging, and for all intents and purposes was a dead box. Yet the machine continued to route traffic and continued to function as a VPN.

Any listening ports were ignored, but the routing still took place. I never thought of it as a way to make the router more secure, but I guess in that way it makes since. It would really suck to have to completely shutdown everything to have to malke routing changes though, so I'm not sure this is the best solution for high availability router use.

NOT a recommended config

[peril]

The most interesting thing I see with this is why?

It can't be managed. It can't be monitored. It can't be logged.

This may be fine as a novelty, but running a network secured with such a hack is silly.

Let's talk about shutting down all userspace processes on a box except syslog and snort and I say you've got an interesting box.

OTW - it seems just like a game.

--Adrian

brilliant!

[Mr. Slippery]

This is a very interesting idea.

It only works with firewalling that's inside the kernel - packet filtering and NAT. But what if kernel modules were added to handle some of the features now run in user space?

Indeed, what if kernel modules were added to handle non-firewalling tasks instead? Could a kernel module provide a useful network service? You start the machine up, it loads the kernel and "halts" but still provides the service. Something goes wrong? Just power cycle; there's no disk access, no way for an attack or malfunction to make a persistant alteration in the machine.

Re:brilliant!

[swb]

Indeed, what if kernel modules were added to handle non-firewalling tasks instead? Could a kernel module provide a useful network service? You start the machine up, it loads the kernel and "halts" but still provides the service. Something goes wrong? Just power cycle; there's no disk access, no way for an attack or malfunction to make a persistant alteration in the machine.

I think you just reinvented the embedded system.

Re:Er, aren't there better ways to do this?

[Junta]

Of course mounting drives read-only, operating out of a ramdisk is still not as secure as this approach if you can afford a very static set of firewall rules.

In this state, the system is incapable of even offering a shell without a full reboot. Once you give it the ability to offer a running process to hijack and potentially have a shell open, the read-only mount only lasts until the equivalent of mount -o remount rw is executed, and then all bets are off. Same of a ramdisk (unless the ramdisk has no ide driver or whatever). But in any case a danger in having it so that you can change firewalling rules is that if they get in, *they* can change rules too.... So, at least in theory, a halted firewall is more secure than even the most anal tactics along the lines you describe. Of course, the chances of an exploit being so severe as to offer root shell is remote, but you can never be too paranoid in some cases.

All this being said, if you have a system dedicated to firewalling by itself, and you are worth your salt as a network administrator, setting up a tight firewall is child's play. If it is coming down the input chain and it is not coming from a specific address range over the correct interface, drop. Maybe allow ssh if you critically need remote admin capabilities, but if something goes wrong with your firewall, you probably need to be there in person anyway.

And if someone untrusted can get into your network and come over the trusted interface into your firewall, well, your network has a lot more problems than a less than perfect firewall...

It's a bridge

[booch]

This method causes the firewall to act as a bridge instead of a router. The advantage is that the firewall is not IP-addressable. To hack it, you'd have to go down to the MAC layer, which is generally only possible if you're on the same network segment.

I read the SysAdmin article a month ago and thought the same thing. The OpenBSD Invisible Firewall is a much better solution -- you can't hack it from the outside, but you can still make any changes necessary without causing downtime.

More Secure Solution

[davidu]

A much more secure solution would be to have a firewall with no IP addresses.

Memoirs of an Invisible Firewall

Using OpenBSD (and linux could work too) it is possible to create a bridging firewall with no IPs that simply scrubs packets as they come through the interface.

One could always add a dialup modem to the machine in case remote access was neccessary but when you have two nics neither of which have IP addresses or running services it makes a machine a whole hell of a lot more useful then a linux machine in halted mode which could EASILY run into weird memory/timing issues. (which the author didn't bring up)

-davidu

Why run an OS at all?

[Mignon]

Now that you've read my provocative title, I'll tone it down a bit:

There have been some great comments to this article (which I haven't read) but I got to wondering: if you're going to run in a sort of comatose state where your only ability to change the system is to reboot it, why bother booting in the first place?

My idea was to use the Linix BIOS or something similar, and run your packet filtering from there. Then you can forget the hard drive and floppy (though you'd probably want that floppy to be able to flash your BIOS with updates and the like.)

Does this make sense to anyone? Or is there something I'm overlooking like maybe that while running as a BIOS, Linux wouldn't be able to talk to the network interfaces, say?

I guess if you're going to go to that kind of trouble, you might as well have an embedded system, or run from flash RAM, as others have mentioned. Still, it's always fun to get hardware and software to do things beyond what they were designed to do.

Halted firewall vs bridge

[chrysalis]

This is roughly equivalent to a briding firewall with no assigned IP address. No one can ever connect remotely.

A bridging firewall as the advantage of still being administrable from the local console.

overkill?

[anthony_dipierro]

If your data needs that much security, you shouldn't have it connected to the internet in the first place.

SunScreen...

[Psarchasm]

SunScreen has been doing this for quite some time.

Read about it here

er..

[einstein]

I had my firewall's IDE controller go up in smoke, but the Linux kept chuggin' along. I couldn't log in or do anthing to the box, and the display was full of errors, but it still was routing for the rest of the network. now that was security.
---

Solving the rebooting "problem"

[vrmlguy]

There seem to be a lot of comments along the lines of "Yeah, but you have to reboot to change the config, and that takes down your firewall". I see this as an opportunity, not a problem.

As the article points out, the kernel continues to run when halted, so the first part of the solution is to signal the kernel to transition out of run-level 0 in a safe way. There used to be ISA cards for debugging that had a push-button at the end of a cable; when pushed, an interrupt was triggered to invoke the debugger. I can't see any reason why the Linux kernel couldn't be patched to watch for that interrupt while halted and restart the boot process, say from the point where a boot disk is mounted. The second step would be to modify the init.d scripts affecting the IP stack to abort if the NICs are already configured.

The end result would be a firewall with a button that, when pressed, would cause the system to "wake up" and allow configuration changes to be made. When you're all done, just do another "init 0". To guard against forgetful netadmins, you may want a watchdog process that also does an "init 0" fifteen minutes after the system comes up.

I can't see any show stoppers to this idea. What do you think?

Re:Solving the rebooting "problem"

[lkaos]

Or one could use the MagicSys keys built into the kernel and simply use a standard keyboard :)

Re:Solving the rebooting "problem"

[cgleba]

Just kill every process except init and unmount filesystems.

Bind "init 3" to a key-combination in inittab and voila!

I'm pretty sure it will work however I have no interest in testing it.

Correction

[mmurray]

Hi all...

As the author of the article being discussed, I wanted to point out one of my own errors. I discussed the lack of swap-space as a limitation to the setup; however, the linux kernel isn't pageable, so swap space would have no effect on the performance of the firewalling code.

I've had a few people point that out, so I wanted to post that correction publically.

Feel free to email me at mmurray@ncircle.com if you have questions or commments... :)

Mike

Use hardware remote access

[ErikTheRed]

Frankly, with all of the discussion centered around administering a machine that's at runlevel 0 or fully stealthed with no IPs, etc., I'm surprised that no one (so far) has mentioned hardware-based remote access products such as Compaq's Remote Insight boards (many other server vendors have similar products).

For ~$500 you get a board that replaces your keyboard, mouse, and video controllers, has its own built-in ethernet adapter (that is invisible to the rest of the computer - it's dedicated to remote access) and an SSL-secured web server. You can completely control the machine via a java applet. You can even cold-boot it if it's in a hung state (and, of course, view any errors on the screen while the machine's in a hung state). Other features include a virtual floppy drive that allows you to copy data to and from the machine (you can even boot off of the virtual floppy). There's plenty of additional coolness; the only downside is that Compaq cards only work in Compaq Proliant servers, HP cards only work in HP servers, etc...

More secure...?

[st0mp]

Aren't we forgetting the most important security feature of a firewall...? There's no logging! This is fly by the seat of your pants security if you ask me. You gonna hang a lucky rabbit's foot over the thing?

Swap

[srichman]

The other consideration is that with drives unmounted, all swap space is removed from the machine. This shouldn't be difficult in a machine that is handling even large amounts of traffic, given sufficient amounts of memory. However, in an older machine with fewer resources, it is possible to experience performance issues with extremely large amounts of traffic.

Kernel memory doesn't swap in Linux. So, even if you could have swap space in a halted firewall box, it wouldn't be used at all