Run Your Firewall Halted for Extra Security

n8willis writes: "There's a great article over at the SysAdmin magazine site that presents a unique approach to improving network security: run your firewall in a halted state. This means runlevel 0; no processes running and no disks mounted, but with packet filtering still on. The author heard a rumor of this capability in the 2.0 series kernels, and he's managed to get it working in 2.2 as well."

Comments

Works for me...

[Rorschach1]

Though I usually just use the power switch. Can't beat a powered-off firewall for security.

...and? We do this all the time

[TicTacTux]

See floppyfw. Does not even *need* a HD to boot. Floppies can be write-protected. Even an old 486 will do that trick.

Amazing what queer pranks people invent in place of a rather obvious solution...

Re:...and? We do this all the time

[moonbender]

This solution is more secure than a floppy would be. Write-access to the flopy can be forbidden, but all the other processes and security holes remain as possible targets.

Re:...and? We do this all the time

[TicTacTux]

Why? That whole system runs in runlevel 0. No user accounts, no login, no getty, no daemons - just the bare kernel plus an open console.

Disadvantage: You must employ your weary bones and walk over to do some maintenance...

Re:...and? We do this all the time

[gid]

Dang, why stop there, write something in ASM and flash it right on the bios. Wasn't there a linux bios project awhile back? :) I wonder how much progress they've made.

Re:...and? We do this all the time

[Deadplant]

you miss the point, this is halted system, not just a system with no drives mounted. I other words no user processes can start at all. It is therefore impossible for a virus to run, the only kind of exploit possible would be one that directly exploits a flaw in the kernel (a fairly rare thing). and beyond that, it would probably have to be an exploit designed for this kind of system specifically since most exploits assume it's possible to execute a process (like a shell, or an 'rm -rf' or something)

Re:...and? We do this all the time

[tricorn]

Y'all know that a halt() call that specifies a non-powerdown halt is basically a no-op, right? I mean, it does check that you're running as root, but then it does nothing:

From linux-2.4.14/arch/i386/kernel/process.c

void machine_halt(void)
{
}

(the machine_power_off routine also does nothing unless there is a poweroff function enabled; since, the halt isn't actually doing a poweroff (or it won't be doing a good job of routing after that) it presumably isn't set).

All that's being done by the process desc ribed in the article is a) bring down all running services except for the network; b) make init wait to die (I haven't looked at init to see if it waits for a normal ctrl-alt-del interrupt, then do a reboot, or if it just sets the kernel to handle ctl-alt-del and reboot all by itself). It doesn't even kill all other running processes that might be running, just the ones that were started by init (e.g. getty) and processes that remain in those process groups. If someone starts up a background job, it will still be there. There's nothing magic about run level 0 - it is just a convention that init uses.

Not all drives are unmounted - the root device will still be mounted read-only, maybe /proc and other pseudo-file systems will still be there. init is still running. This is a lower level of security than starting the kernel up by running a shell script that enables the network and ipchains, then sleeps forever. You could even have that shell script leave you in a plain old shell if you want to do be able to do something else (you'd have to know how to mount filesystems and remember to unmount them (since init won't be running, you can't just do a simple "shutdown" anymore), etc).

Here's an example script that might be used. Start it by passing "init=/netstart" (or wherever you put the script) to the kernel when you reboot:

#!/bin/sh
mount proc /proc -t proc
mount -o ro /usr
/etc/rc.d/init.d/network start
/etc/rc.d/init.d/ipchains start
umount /usr
exec /bin/sh

If there's a hole in the TCP stack itself, or in the IPchains routing, that opens up some vulnerability, it doesn't really matter if there's a simple /bin/sh running or not (if you really want to be secure, have the shell script exec a simple C program that checks to make sure it is PID 1; does a kill(-1, SIGKILL) and then sigsuspend() in a loop; now you're safe unless someone's hacked your kernel, libc or compiler).

I note that in 2.2, if process 1 exits, it kills the kernel; in 2.4, it gives a "Kernel panic: Attempted to kill init!". In either case, the effect is the same: ctrl-alt-del is disabled, and the kernel stops routing packets.

Er, aren't there better ways to do this?

[oGMo]

This might be fine for a firewall whose rules never change, but if you want to actually do something to it, you can't have it halted, and rebooting will just cause downtime.

Why don't you just run with drives mounted read-only, or have everything copied to a ramdisk? Surely you can verify a rule to allow remote access from only certain locations if this is your problem, or even require serial console access.

Re:Er, aren't there better ways to do this?

[gid]

Well if you have drives mounted, even read only, then someone could potentially get in and bounce off the firewall to attack another machine on the inside of your firewall using processes and utilities on the drive that you mounted. Whereas if you didn't have the drive mounted to begin with, then no one could get in, even you! And that's the point.

This definitely isn't a method for everyone, it's for really paraniod, stable networks that don't change. If your network doesn't fit this description, then your way might be the next best thing.

Re:Er, aren't there better ways to do this?

[gorilla]

You could have two firewalls, one with the current configuration, and one being preped for the next configuration. When you are ready to change them, just flip them.

Re:Er, aren't there better ways to do this?

[Junta]

Of course mounting drives read-only, operating out of a ramdisk is still not as secure as this approach if you can afford a very static set of firewall rules.

In this state, the system is incapable of even offering a shell without a full reboot. Once you give it the ability to offer a running process to hijack and potentially have a shell open, the read-only mount only lasts until the equivalent of mount -o remount rw is executed, and then all bets are off. Same of a ramdisk (unless the ramdisk has no ide driver or whatever). But in any case a danger in having it so that you can change firewalling rules is that if they get in, *they* can change rules too.... So, at least in theory, a halted firewall is more secure than even the most anal tactics along the lines you describe. Of course, the chances of an exploit being so severe as to offer root shell is remote, but you can never be too paranoid in some cases.

All this being said, if you have a system dedicated to firewalling by itself, and you are worth your salt as a network administrator, setting up a tight firewall is child's play. If it is coming down the input chain and it is not coming from a specific address range over the correct interface, drop. Maybe allow ssh if you critically need remote admin capabilities, but if something goes wrong with your firewall, you probably need to be there in person anyway.

And if someone untrusted can get into your network and come over the trusted interface into your firewall, well, your network has a lot more problems than a less than perfect firewall...

Re:Er, aren't there better ways to do this?

[dpilot]

Or just run "lcap CAP_SYS_ADMIN" to drop mount
capability. Unfortunately, you drop a bunch of other capabilities too. While you're at it, run "lcap CAP_SYS_BOOT" to prevent reboots, and "lcap CAP_LINUX_IMMUTABLE" to prevent those immutable and append-only attributes from getting changes, "lcap CAP_SYS_MODULE" to lock your modules in place, and prevent further loading, and "lcap CAP_SYS_RAWIO" to close down /proc/kmem. There are more things you can remove from the kernel capabilities bounding set to lock your system down.

Of course, by the time you're done, your system is about as easy to administer as if it were halted. Not to mention log rotation requiring a reboot.

Replace INIT

[Skweetis]

I used to run my ipfwadm firewall kind of like this. I replaced INIT with an ipfwadm replacement I wrote that read ipf style rules from a config file and applied them to the kernel chains, then suspended. This was all on a ramdisk. It worked pretty well, although the ramdisk had to be replaced to change rules, which was kind of inconvenient.

Another interesting consept: Invisible Firewall

[bob@dB.org]

introduction from http://www.openlysecure.org/openbsd/how-to/invisib le_firewall.html

In the stone age of firewalling, a firewall was a fairly complicated device that was less-than-trivial to factor into your network. It needed an IP address on it's outside, and another on the inside. This immediately created subnetting problems, forcing wasted IP allocation and overall disquietude amongst the cognoscenti. It also meant that your firewall was very visible to the world, and its function was rather obvious and easy to deduce. There had to be a better way. And now there is...

Dream with me for a moment: A black box, shimmering in the soft LED-green glow of the network cabinet. You take the network cable from your router, which previously went into a switch, and stick it in one of the snappy plugs in the back of the box. There's one more plug on the black box, so you grab another cable and hook the box up to the switch. You step back, and suddenly: everything looks the same. You go to all your computers. As far as you can tell from inside and outside the network, the box doesn't exist. It does nothing. A few minutes later, you have a monitor and keyboard hooked up to the back of the box. You quickly and easily begin to tweak a file that gives you fine grained control over access to your network. You shut off all access to your mailserver from the outside world except on port 25.

Re:Another interesting consept: Invisible Firewall

[douglips]

The problem with that is your LAN needs to be all on routable addresses. NAT won't work if the firewall has no IP address...

For exposed servers this would work well, not so much for the 18 Windows boxen you have Joe Salesguy using.

Re:Another interesting consept: Invisible Firewall

[Chang]

Who says the firewall and the nat have to be the same box? Just because this is usually one device doesn't mean it must always be that way

Re:Another interesting consept: Invisible Firewall

[Phil Gregory]

Tom Limoncelli presented a paper on this exact subject a few years ago. It's entitled Tricks You Can Do If Your Firewall Is a Bridge and covers a number of useful things you can do with bridging firewalls.

--Phil (My firewall is just an OpenBSD box. I like it that way.)

Logging?

[JimR]

Bit of a shame if you want to log any attacks
on the firewall though.

With no disks mounted where can you log it to?

Re:Logging?

[booyah]

Same place I log anyways

Dot matrix printer on LP1 :-)

try to erase THAT log :-)

-booyah

Re:Logging?

[Stary]

If you're seriously paranoid about it, get an old dot-matrix printer and hook it up as the logging output device. You'll have a logging device that nobody could alter no matter how compromised the system gets, without having to mount anything read-write.

Re:Logging?

[doorbot.com]

Bit of a shame if you want to log any attacks
on the firewall though.

If you can get this to work with IPFilter/PF you could use the "dup to" method to send the packet out a third NIC to a packet logging machine. Now you can have a transparent firewall, which is not accessible at all, but you can still have some logging features. One possible design for this would be two SBCs in a 1U case... one is the "halted" firewall and the other is the logging machine.

I don't know if this is possible with NetFilter or IPChains, however.

Re:Logging?

[Foxman98]

would be fairly easy...

see we have this thing these days....

it's called "fire"

i have portable "fire creation device".

commonly called a "lighter"

;-p

I figured this out in a much different way

[Binestar]

We were offering a linux-firewall, VPN solution to a hospital and turns out one of the machines we sent out had a bad token ring card, that mixed with an obscure bug in the Token ring driver, (Which has since been fixed) it would cause linux to die quite often. To the point where the computer would accept no keyboard input, would cease logging, and for all intents and purposes was a dead box. Yet the machine continued to route traffic and continued to function as a VPN.

Any listening ports were ignored, but the routing still took place. I never thought of it as a way to make the router more secure, but I guess in that way it makes since. It would really suck to have to completely shutdown everything to have to malke routing changes though, so I'm not sure this is the best solution for high availability router use.

NOT a recommended config

[peril]

The most interesting thing I see with this is why?

It can't be managed. It can't be monitored. It can't be logged.

This may be fine as a novelty, but running a network secured with such a hack is silly.

Let's talk about shutting down all userspace processes on a box except syslog and snort and I say you've got an interesting box.

OTW - it seems just like a game.

--Adrian

Re:NOT a recommended config

[SilentChris]

Break a Furby apart to learn its internal electronics (and try to reprogram its main chip to speak curse words) = A hacking "experiment" with "great knowledge" learned by the experimenter.

Creating an invulnerable firewall that only needs to be configured once and will forever be secure = A "game".

?

brilliant!

[Mr. Slippery]

This is a very interesting idea.

It only works with firewalling that's inside the kernel - packet filtering and NAT. But what if kernel modules were added to handle some of the features now run in user space?

Indeed, what if kernel modules were added to handle non-firewalling tasks instead? Could a kernel module provide a useful network service? You start the machine up, it loads the kernel and "halts" but still provides the service. Something goes wrong? Just power cycle; there's no disk access, no way for an attack or malfunction to make a persistant alteration in the machine.

Re:brilliant!

[swb]

Indeed, what if kernel modules were added to handle non-firewalling tasks instead? Could a kernel module provide a useful network service? You start the machine up, it loads the kernel and "halts" but still provides the service. Something goes wrong? Just power cycle; there's no disk access, no way for an attack or malfunction to make a persistant alteration in the machine.

I think you just reinvented the embedded system.

Re:You could try something even more secure...

[ajakk]

Or you could try not trolling and read the actual article. The author
talks about whether or not this could be done in OpenBSD as well because
of its good kernel level firewall utilities.

Not much of a Firewall

[FreeLinux]

This isn't much of a firewall when there is no logging or alerting. With this approach you basically have a solid state packet filter and if that's what you want, a Linksys box would still be a better solution.

There are a lot of things that can be done but, one has to ask: Other than doing it for the sake of doing it, what's the point?

Re:Difference betwwen this and Read only media?

[anti-snot]

Difference is, even with read only media, an intruder can still do what he wants in memory (although rebooting will wipe it clean). With this, there is nothing running but the kernel and kernel memory... not even a ramdisk to fiddle with.

OpenBSD

[Motheius]

IPF can run in stealth mode. In this mode a packets TTL isn't decreased when the packet traverses the firewall. It is invisible to the netork at large.

What is the benefit of this?

[SClitheroe]

The kernel is still executing, as is IPChains (obviously)..If IPChains has any exploits, what is to stop a hacker from being able to modify the firewall configuration in memory, thus punching holes in the firewall?

Re:What is the benefit of this?

[entropi]

ipchains isn't running. it's used before the halt to set the rules, and those rules simply aren't cleared out when the system switches to halted mode. but to make changes you'd need to run ipchains again, which would be another process, which can't happen with this configuration.

Re:Logging CAN be done in this setup.

[BlueUnderwear]

Wouldn't this still need a locally running klogd to forward the messages to the remote syslogger?

It's a bridge

[booch]

This method causes the firewall to act as a bridge instead of a router. The advantage is that the firewall is not IP-addressable. To hack it, you'd have to go down to the MAC layer, which is generally only possible if you're on the same network segment.

I read the SysAdmin article a month ago and thought the same thing. The OpenBSD Invisible Firewall is a much better solution -- you can't hack it from the outside, but you can still make any changes necessary without causing downtime.

My way or the HIGHWAY BUB!

[aphor]

This is kind of ridiculous because it requres you to take the whole firewall down to make rule changes. Why not just blackhole all packets with the firewall's interface addresses as destination, and leave a local serial console with password authentication and idled running?

Alternately, this is kind-of a solution for embedded firewalls or something.. maybe for tiny flash-card bootable network probes (remember firewalls are routers and can be made to do nasty things too) that you boot and then eject the flash media...

More Secure Solution

[davidu]

A much more secure solution would be to have a firewall with no IP addresses.

Memoirs of an Invisible Firewall

Using OpenBSD (and linux could work too) it is possible to create a bridging firewall with no IPs that simply scrubs packets as they come through the interface.

One could always add a dialup modem to the machine in case remote access was neccessary but when you have two nics neither of which have IP addresses or running services it makes a machine a whole hell of a lot more useful then a linux machine in halted mode which could EASILY run into weird memory/timing issues. (which the author didn't bring up)

-davidu

Best way I've found for secure firewalls...

[Macphisto]

At home, I run two servers - one inside the firewall with NFS services and so on, and the firewall itself, which is a self-sufficient machine except during boot, where it netboots from the NFS server. After that point, its two NICs are only used for routing packets. All administration takes place through the serial port, which init respawns /bin/sh onto. The system runs off of a ramdisk loaded during the netboot. I've found this to be a nice solution, and it rules out any possibility of breach since the machine is only accessible through a serial port hardwired to the server - ethernet interfaces are configured to only route packets.

Re:Why use a whole computer?

[EllF]

The LinkSys DSL Routers, while nice, are not perfect solutions. I run one on the inside of my network, and I've noticed that it passes along packets that I have not explicitly told it to pass. One of the features that doesn't seem to be touted these days is that it "senses" specific types of packets and "intelligently" allows them through. The trouble with this is that the router is not doing stateful packet inspection - it's just guessing based on port number. Not a *huge* deal for most people, but I most certainly would not trust such a system to be my primary border defense.

Cheap and easy, but not as good as the real thing. When it comes to security, I'd rather have a locked-down debian box that I built from source than a piece of plastic that I bought at Staples.

Re:Logging? - syslog

[Anonymous Coward]

There's no syslog running anymore.

Dumbass.

Why run an OS at all?

[Mignon]

Now that you've read my provocative title, I'll tone it down a bit:

There have been some great comments to this article (which I haven't read) but I got to wondering: if you're going to run in a sort of comatose state where your only ability to change the system is to reboot it, why bother booting in the first place?

My idea was to use the Linix BIOS or something similar, and run your packet filtering from there. Then you can forget the hard drive and floppy (though you'd probably want that floppy to be able to flash your BIOS with updates and the like.)

Does this make sense to anyone? Or is there something I'm overlooking like maybe that while running as a BIOS, Linux wouldn't be able to talk to the network interfaces, say?

I guess if you're going to go to that kind of trouble, you might as well have an embedded system, or run from flash RAM, as others have mentioned. Still, it's always fun to get hardware and software to do things beyond what they were designed to do.

Halted firewall vs bridge

[chrysalis]

This is roughly equivalent to a briding firewall with no assigned IP address. No one can ever connect remotely.

A bridging firewall as the advantage of still being administrable from the local console.

overkill?

[anthony_dipierro]

If your data needs that much security, you shouldn't have it connected to the internet in the first place.

SunScreen...

[Psarchasm]

SunScreen has been doing this for quite some time.

Read about it here

er..

[einstein]

I had my firewall's IDE controller go up in smoke, but the Linux kept chuggin' along. I couldn't log in or do anthing to the box, and the display was full of errors, but it still was routing for the rest of the network. now that was security.
---

But...

[Klowner]

Then how would I telnet to my firewall from school?

*dodges flying shoes*

;)

Klowner

Solving the rebooting "problem"

[vrmlguy]

There seem to be a lot of comments along the lines of "Yeah, but you have to reboot to change the config, and that takes down your firewall". I see this as an opportunity, not a problem.

As the article points out, the kernel continues to run when halted, so the first part of the solution is to signal the kernel to transition out of run-level 0 in a safe way. There used to be ISA cards for debugging that had a push-button at the end of a cable; when pushed, an interrupt was triggered to invoke the debugger. I can't see any reason why the Linux kernel couldn't be patched to watch for that interrupt while halted and restart the boot process, say from the point where a boot disk is mounted. The second step would be to modify the init.d scripts affecting the IP stack to abort if the NICs are already configured.

The end result would be a firewall with a button that, when pressed, would cause the system to "wake up" and allow configuration changes to be made. When you're all done, just do another "init 0". To guard against forgetful netadmins, you may want a watchdog process that also does an "init 0" fifteen minutes after the system comes up.

I can't see any show stoppers to this idea. What do you think?

Re:Solving the rebooting "problem"

[lkaos]

Or one could use the MagicSys keys built into the kernel and simply use a standard keyboard :)

Re:Solving the rebooting "problem"

[cgleba]

Just kill every process except init and unmount filesystems.

Bind "init 3" to a key-combination in inittab and voila!

I'm pretty sure it will work however I have no interest in testing it.

Cheap rackmount solutions?

[iamsure]

Okay, so do anyone know of a cheap place to get rackmount 1U's, like, with just a pentium, network card and cdrom?

Take this idea, a cheap rackmount, a readonly boot medium, and you have a very very secure device.

Possibly the ultimate in security? :)

Re:Logging? - syslog

[gaj]

no need for it. the fw is acting as a *client* for syslog. syslog would be running on the log server.

Correction

[mmurray]

Hi all...

As the author of the article being discussed, I wanted to point out one of my own errors. I discussed the lack of swap-space as a limitation to the setup; however, the linux kernel isn't pageable, so swap space would have no effect on the performance of the firewalling code.

I've had a few people point that out, so I wanted to post that correction publically.

Feel free to email me at mmurray@ncircle.com if you have questions or commments... :)

Mike

Use hardware remote access

[ErikTheRed]

Frankly, with all of the discussion centered around administering a machine that's at runlevel 0 or fully stealthed with no IPs, etc., I'm surprised that no one (so far) has mentioned hardware-based remote access products such as Compaq's Remote Insight boards (many other server vendors have similar products).

For ~$500 you get a board that replaces your keyboard, mouse, and video controllers, has its own built-in ethernet adapter (that is invisible to the rest of the computer - it's dedicated to remote access) and an SSL-secured web server. You can completely control the machine via a java applet. You can even cold-boot it if it's in a hung state (and, of course, view any errors on the screen while the machine's in a hung state). Other features include a virtual floppy drive that allows you to copy data to and from the machine (you can even boot off of the virtual floppy). There's plenty of additional coolness; the only downside is that Compaq cards only work in Compaq Proliant servers, HP cards only work in HP servers, etc...

More secure...?

[st0mp]

Aren't we forgetting the most important security feature of a firewall...? There's no logging! This is fly by the seat of your pants security if you ask me. You gonna hang a lucky rabbit's foot over the thing?

Swap

[srichman]

The other consideration is that with drives unmounted, all swap space is removed from the machine. This shouldn't be difficult in a machine that is handling even large amounts of traffic, given sufficient amounts of memory. However, in an older machine with fewer resources, it is possible to experience performance issues with extremely large amounts of traffic.

Kernel memory doesn't swap in Linux. So, even if you could have swap space in a halted firewall box, it wouldn't be used at all

Hrm, there is also SPARC

[clump]

Management is one reason I like SPARC so much. IE, connect a serial line to even a decade+ old Sun Pizzabox and manage via OpenPrompt. Even if the operating system is hosed you can still reach the OK prompt and reboot, or perform other tasks. Further, you can use the OK prompt to do network boots out of the box.

The Compaq cards do have an upside, however, in that the way you describe them it seems that you can foobar the 'parent' machine and still use the card. It does use power from the parent machine, so both SPARC's OpenPrompt and this card will be null if the host machine loses power.

I really wish x86 engineers would develop something similar to a server-class BIOS implementation. Don't get me wrong, modern x86 CPUs are great, but the BIOS has got to grow up.

Versus an empty "netstat"?

[PhotoGuy]

Why not just avoid starting all services, and verify with "netstat" that there's no processes listening.

If there's not a single process listening on sockets (shown by netstat), then only the kernel is around doing it's masquerading, routing, etc., just as per runlevel 0.

While it's a cute trick, why limit *all* your potential functionality (flipping on ftp for 15 minutes, to let a buddy upload a file).

-me

Solving the RAM issue

[Com2Kid]

Some people so far have mentioned that even having anything loaded in RAM creates a danger.

Would it be possible to create a system that uses Static RAM (Smart cards, whatever) and have two sticks of it inserted?

One that you boot from and read the firewall rules and such out of, and another 8 meg or so stick that is used just for proccessing things.

Some types of flash cards now days have read write tabs on them, after your OS has loaded and all and you have got things how you want it, halt it, and then switch the tab on your main memory stick so that it is read only.

The 8 meg stick would have previously been assigned to be used for working with packets and such that the firewalls needs to do.

Even better of course would be if you could get a CPU with a nice sized Cache on it (hmmmm, K6-3s, hehe. External cache up to what, 8 megs? :) :) :) ) and just run almost everything directly out of that after booting. Though that would require some really nifty programming more then likely. :) It would DEFINTLY not be something that any random hacker would suspect, hehe.