Firewall Failover With pfsync And CARP

Daniel Hartmeier writes "OpenBSD developer Ryan McBride explains the new firewall redundancy features in the upcoming OpenBSD 3.5 release in his article Firewall Failover with pfsync and CARP. CARP (Common Address Redundancy Protocol) is a free alternative to the patent-encumbered VRRP, responsible for electing masters in a firewall cluster, while pfsync syncronizes packet filter state information among nodes. The combination allows to replace single-point-of-failure firewalls with clusters of two (or more) nodes, which continue to filter ongoing and new connections when nodes fail. Additional features like arpbalance allow one to share a single IP address for multiple servers, transparently balancing load among them, and adapting to servers failing. Pre-order for OpenBSD 3.5 has started, CDs will ship May 1st."

Comments

That's really cool

[Anonymous Coward]

I think my office implemented such functionality for like $120k, and it doesn't even work too well.

Re:That's really cool

[hdw]

Yup, we have something like that too.

Except that our 50.000USD firewall solution fails to handle state sync (they've got problems enough with rules sync) and the the failover works so bad that the dudes that run it have failed over to manual fail over :)

I've been _soo_ tempted to suggest to replace the all the gunk with OpenBSD, since it has all the stuff we need, and it works ...

And it is a little bit cheaper. // hdw

HSRP

[bolix]

I love sniffing the Cisco equivalent to CARP. Lots of HSRP calls to 224.0.0.224 with no security built in. A simple ARP poison will fuck the switch. More advanced attack methods can be found c/o Phenolit

I wonder...

[Yarn]

What hardware would I need to do this on my 1000SX uplink. Admittedly, I've only peaked at 80Mbit/s so far, but I think even handling that will take some beefy hardware.

Re:I wonder...

[dhartmei]

Filtering ordinary traffic (not extreme test-cases of minimal packets, average number of packets/connection) statefully at 100Mbps doesn't require much hardware. Even little Soekris boxes (embedded 486 133MHz) can do that.

For Gbps, the limiting factor is the NIC and its driver. Some cards/drivers are reported to reach more than 70% of the maximum throughput. The reason they don't (yet) go further is not packet filtering, though.

If you want specific names/models, the mailing list archives contain the reports.

Re:I wonder...

[hdw]

I'm running an OpenBSD 3.4 firewall on a PII-400 with a 100Mb/s Internet feed.

And I know that I've reached over 40Mb/s without any sign of problem with the firewall.

So unless you're running lots of IpSec stuff or have a high rate of connects I don't think the firewall (or OpenBSD) will be the problem.

I think the selecting a good NIC is more important. // hdw

Re:I wonder...

[Homology]

I wonder how a "little" p2 can filter 40MB/s of packets. when it seems like the same p2 will bog down in other stuff (im not talking about a gui)

can you explain this?

The grandparent wrote 40Mb/s, like in 40 mega bit, and a PII can handle this. However, you should have a good NIC and not one of those pisspor Realtek that offloads the work to the CPU.

Re:I wonder...

[hdw]

yup, I can.

First of all, I said Mb, not MB, call me conservative but I'm used to count bandwidth in bits, not bytes.

Second, as I stated, check your NIC and the drivers.
It means a lot when it comes to network handling.

(I remember how out old VAX 11/785 reacted when it shared an non-switched net with 2 sparc servers, the poor VAX was down on it's knees just by trying to ignore the traffic :)).

And as a wider note, the performance of a system isn't only down to processor speed. There's tons of parameters, both hard and soft, that come into play.

Figuring out _what_ parameter to fiddle with is regarded as voodoo :)

/ hdw

ps
No, I'm no speed guru, neither did I wave a dead chicken and dance backwards while installing that firewall.
I asked for a raw Internet feed to the labnet, and at that time we didn't have anything less then 100Mb/s to hand out. And the server played it nice.
ds.

Re:Mailto link?

[dhartmei]

@openbsd.org addresses are already readily available for harvesters through cvsweb, mailing list archives and usenet gates, putting one in a /. posting couldn't make things any worse.

The upside is that after a certain amount of spam received, people get really good at filtering it. That's where the motivation behind some of the anti-spam features in OpenBSD comes from, I guess :)

Re:This is awesome

[dhartmei]

What CARP/pfsync does is transparent balancing on IP level. Each client connection is redirected to an arbitrary available server. This works for applications where each server can independantly handle a client request, like serving stateless HTTP or DNS from multiple servers.

For SQL, clustering is much more involved. One client might insert data that must propagate to the other server, or locks across all servers must be obtained, etc. This cannot be done transparently on IP level, the servers themselves must support it.

Search for replication, clustering or redundancy together with postgresql, you'll find erserver etc. Except for very special cases (like read-only databases), this way beyond IP level packet filtering ;)

Sad.

[MisterP]

It's kinda of sad that something this cool gets so little discussion on a site like Slashdot. I guess it will be news when CARP gets ported to linux and iptables gets ip state sync'ing across hosts.

Re:Sad.

[hdw]

Userland CARP is already ported to Linux.

http://www.ucarp.org

/ hdw

I'm a firewall admin amongst other things..

[harikiri]

...and this looks really attractive to me. Our environment comprises of Nokia IPSO-based firewalls running Checkpoint, so I'm very familiar with VRRP.

However, as excellent as this looks, I can only shudder in horror at the thought of migrating any of our existing rulesets across to openbsd/pf, let alone distributed management of policies across several 'clusters' of firewalls we have.

Yes my friends. I'm asking for a GUI. FW Builder is a good start, but it still needs work (porting to Windows would be a good start). Migration tools from Checkpoint (or other commercial firewalls) would be another good addition.

PS, I ask for Windows support not for my sake, but so that my co-workers would be able to use it. However, this criticism is levelled at FW Builder.

OpenBSD/pf/CARP has provided a brilliant technical starting block, but it needs these additional tools to make inroads into enterprise organisations.

Re:I'm a firewall admin amongst other things..

[harikiri]

I'm very aware that I could put together my own 'deployment' script with a combination of ssh/scp and rsync.

If you need a GUI and FW admin is your day job, I have to wonder why you're bothering with FW admin.

I do not need a GUI. My colleagues do not need one either (we previously used PIX... shudder). But when you start dealing with a large number of firewalls (we have over 25 deployed), and not simply firewall rules, but NAT, PAT, authentication and VPN's - having a GUI frontend that ties all that information up together and provides it in an easy to manage way, is a lot better than grepping and trawling through long configuration files to make additions or changes.

Yes any capable firewall admin should be able to implement rules once they read documentation for ipfilter/iptables/pf/ipfw/etc - but they shouldn't necessarily have to. The people I work with aren't stupid, they just don't want to have to work at the command-line across multiple systems to implement a single rule.

Re:I'm a firewall admin amongst other things..

[hdw]

PF is not hard to understand and distributing common rules and specific rules is super easy and secure, with tools that come with a default install of OpenBSD (scp).

I have no problem understanding pf rules or distribution via scp (or cvs, works very well).

But it's not about understanding pf rules, it's about keeping track of, often hairy, network and system topology, of various security policies and in many cases a horde of users that need authentication (and that forget their PINs, break their tokens, move between sites ...).
All perfectly possible to handle by editing the rules by hand and push out with scp but only together with hordes others docs keeping track of all the needed fluff.
Then add that changes to the ruleset should be fully traceable and often have to pass thru several pairs of hands and eyes before we even reach the firewall admin. So we really need something easier to the eye than pf rules.

A good, database driven, firewall admin GUI is a very good thing, and it a vital part of enterprise security.

Any enterprise which hires network or firewall admin staff who can't understand pf.conf after reading the fine docco, needs to look into why their hiring policies are such a failure, so as to allow them to hire a fraud.

Oh, come on, step down to the land of the living.

People get shifted around at every reorganisation, suddenly all security is in one global department, 6 months later it's back to the local sites, then it's outsourced, then it's insourced again and 'firewall admins' aren't just carefully selected high profile security pros, they come from all over the place.

// hdw
ps.
I think I'll go back and look one of my old projects again, OpenBSD/pf/altq/carp is really getting ready for primetime.
ds.

Re:I'm a firewall admin amongst other things..

[sirket]

There is absolutely no benefit to a GUI at all

This is a idotic comment. I've been a firewall admin for years. I admin CheckPoint, PIX, NetScreen, ipfw, ipf, and pf firewalls.

Have you ever tried to configure a fully meshed VPN topology between 30 sites by hand? Are you really going to sit there and write 900 rules by hand and expect to do it without making a mistake?

What about defining a group of objects on one firewall (say a cluster of web servers) and then going to implement a rule on a different firewall that uses that web server group? With a central GUI, you can define the object once and not worry about changing it in 5 places or making a mistake when you copy it over to another firewall. (Yes this can be done with scripts but if you are going to write a whole management interface, why not stick a GUI on top of it to make browsing rules easier?)

What about when you need to print out the rule sets for a compliance officer or your CEO?

What about when you have have 25 firewalls and you forgot to backup the rule set on a firewall that just died. Wouldn't it be nice to have a management box with all the rule sets stored locally?

There are about 50 good reasons to have a GUI and very few reasons not to have one. As long as you can configure the boxes from the command line and the GUI doesn't generate gibberish rules, then it is an excellent addition to a great firewall package.

-sirket

Re:I'm a firewall admin amongst other things..

[sirket]

So what you're saying is "I don't want to do my job, cause that's too much work."?

No. What he is saying is that unlike you, he is not an idiot. He recognizes how easy it is to make a typo when you have to enter the same rule and object definition on 25 firewalls. He recognizes the security advantages of a simple clean way to view firewall rules to help avoid a mistake in the ruleset.

The biggest information security threat to any company is the arrogance of its admins. Instead of bitching about a GUI a good firewall admin would welcome additional tools to help manage his or her firewalls. As long as the GUI doesn't stop you from editing rules by hand, why not make use of its ability to display your rules in a different way?

-sirket

Re:I'm a firewall admin amongst other things..

[Bensmum]

I don't recognize a good thing because its not a good thing. This isn't a difficult concept. Just because you make the claim that a GUI is somehow required and you can't function without one doesn't make it so. If you insist on claiming that open source firewall solutions aren't good enough because they don't provide a GUI, how about you back it up with some facts, instead of just insulting the people who are giving you this stuff *for free*. Talk about a "world owes me" attitude.

And its not that I am so arrogant that I never make a mistake, its that I *test* changes to see if they work, the new rulest is applied for 30 seconds to see if it works, and automatically reverted to the old rule set after that. If it did work, I update it for real. A GUI isn't going to help with this.

CARP also works on Linux, NetBSD and OpenBSD 3.5

[chrysalis]

Try UCARP a portable userland implementation.

CARP/pf song for 3.5 Release

[RupertJ]

In keeping with OpenBSD's promo songs, the 3.5 release features a Monty Python-style sketch and song about CARP/pf and VRRP etc. Very funny stuff indeed. Lyrics and links to download the songs in MP3/OGG format at http://www.openbsd.org/lyrics.html

Conterpoint: Cisco PIX

[^BR]

Cisco PIXes are configured the old way thru SSH (ok, there's a Web interface, never heard of anyone using it) and they sell pretty well. Cisco do have a (laughable) management solution that includes a GUI but almost nobody use it as it plain sucks (simply installing it is a nightmare, plen,ty of dependencies...). The nice thing is that it provides a nice market for third party solutions to do that job...

So having a GUI is not a prerequisite for enterprise acceptance. Even if being Cisco sure helps...

Re:Conterpoint: Cisco PIX

[sirket]

I configure PIX's all day long and I love the simplicity of a PIX config file. That said, Cisco has been losing market share for years because they don't have a GUI. Ever try to set up a ton of VPN's through the command line? Doable? Certainly. Fun? Not a chance.

-sirket

Reading isn't that hard.

[Bensmum]

Seriously, aren't listening. You don't have to enter the same rule and object definitions over and over, that's exactly what I am saying. You make a single template, and then any firewall from there is just changing some variables like $ext_if or $local_net. Plus there are lots of things you don't have to do with pf, like making a whole set of rules to stop spoofing, with pf you can just do antispoof on $ext_if. I am not complaining about a GUI tool, I am saying the parent poster is dumb for complaining about the lack of a GUI, when he hasn't even bothered to learn how the thing works, to see if he even needs one.

Re:Reading isn't that hard.

[sirket]

You're assuming I want a simple rule set that can be templated. That isn't how most firewalls work. They share objects, but rarely do they share rules. Can this be done through macros and from the command line? Of course it can. The problem is that when you are updating your firewall during the 1 hour 3 am maintenance window it is easy to make a mistake that you just overlook because you've been staring at rule sets all day. Different data representations (A GUI) are critical to making sure that you understand exactly what your rule set is doing. A GUI is also useful for building intial rule sets and for prototyping changes. Finally, a GUI prevents you from making a typo (at least in terms of syntax). It's not a big deal if you verify your rule sets each time (good advice no matter what) but a GUI won't let you make these mistakes in the first place.

As long as the GUI doesn't prevent you from editing the raw rules, then it should be a welcome addition to any admins toolkit.

I am saying the parent poster is dumb for complaining about the lack of a GUI, when he hasn't even bothered to learn how the thing works, to see if he even needs one.

You don't know anything about the parent poster. You've never met him and you don't know what he or she knows and doesn't know. For all you know you've been insulting Bill Cheswick. Or perhaps he is just one of the many overworked admins out there who would like to see a tool that would make his job just a tiny bit quicker so that he can go home on time and actually see his family before sunset.

-sirket

Interview with Ryan McBride

[dhartmei]

Jeremy Andrews from kerneltrap.org has just published an Interview with Ryan McBride, which makes for an excellent read on CARP and pfsync.