Changes in the Network Security Model?

Kaliban asks: "As a Sysadmin, understanding network security is clearly an important part of my skillsets so I wanted to get thoughts on a few things that I've seen recently after some discussions with co-workers. Are network services becoming so complicated that application level firewalls (such as ISA Server) are absolutely necessary? Is the simple concept of opening and closing ports insufficient for networking services that require the client and server to open multiple simultaneous connections (both incoming and outgoing)?This leads me to my next question: has the paradigm of 'if you offer external services to the Internet then place those machines onto a perimeter network' been eroded? Are application level firewalls sophisticated enough to allow machines on your internal network to advertise services to the Internet? When is it alright to 'poke a hole in the firewall' to allow this? Personally, I think the answer is 'Never!' but perhaps I'm out of touch with current network security models."

Comments

Try a three-tiered approach

[Eponymous Cowboy]

There are three disparate levels of security you need to consider, and it is advisable to take a three-tiered approach to the problem.

First, for employees and others who have trusted access to your network, the answer is not to poke holes in your firewall. Rather, the answer is simple, just three letters. VPN. By setting up a secure, encrypted, authenticating channel, you bring your trusted users into your network. From your point of view and theirs, it is as if their machines were physically located on the other side of your firewall--just like having the machines right in your building.

Second, for business partners and contractors who need limited access to a subset of services, but whom you do not trust fully, the answer is quite likely also a VPN, but not directly into your network. For services provided to these people, you want everything from your end first going through application-level firewalls, and then through the VPN, over the Internet, to them.

Using a VPN in these cases prevents random hackers from entering your network on these levels.

Finally, for the general public who simply need access to your web site, the ideal situation is to simply host the web site on a network entirely separate from yours--possibly not even in the same city. Use an application-level firewall to help prevent things like buffer overflows. Then, if your web server needs to retrieve information from other systems on your network, have it communicate over a VPN, just like the second-level users mentioned above--that is, through additional levels of firewalls to machines not directly on your primary network. (Basically, you shouldn't consider your web servers as trusted machines, since they are out there, "in the wild.")

By following this approach, you expose nothing more than is necessary to the world, and greatly mitigate the risk of intrusion.

Re:Try a three-tiered approach

[redhog]

One thing that I need to consider at my current job is that you can NOT trust employees computers at home, even if you can trust employees - if they are running Windows, they are potential virus and worm vectors, and needs to be shielded off, so a simple VPN-solution is no solution.

We've solved the most immediate problem by allowing only ssh, and giving employees with Windows a copy of WinSCP (an excelent, two-pane Windows-FTP-client-look-a-like front-end to scp), which they have had no problems using (they did not have any oportunity to work from home before, so they don't complain :).

We also plan to later on introduce AFS and allow remote AFS mounts, and VNC remote-desktops.

Locally, we have a simple port-based firewall, basically walling off all inbound traffic except ssh and http (and allowing nearly all outbound traffic), and keep our OpenSSH and Apache servers updated (have you patched the two ssh bugs reported on /. on your machines yet?).

So, my advice is - keep it simple. Do not trust a too complicated system. And keep your software patched for the latest bugs - keep an eye on the security-update-service for your distro/OS and bugtraq.

Re:Try a three-tiered approach

[kennyj449]

In my opinion, between the danger of worms transmitted above the application level and the existence of uneducated users (in many cases, uneducatable) as well as the whole physical security issue, even an internal network is not to be trusted (though few are actually worse than the Internet, except for pervasive wireless networks that don't use a strong, non-WEP encryption solution.) VPNs can definitely be very useful, but placing using them only at the outer edges of your network (e.g. internet-based links) leaves you wide open to any form of attack that originates from inside, which is always a danger no matter how good your external defenses are.

Personally I don't think that physical seperation is necessary if you're going to be using a strong VPN, because of the fact that you can make it so that the only traffic that passes back and forth is through a VPN and is then no less secure (if anything more secure, except for the purposes of physical security) than if traffic were being passed over the internet. You also get the advantage of increased throughput, a single (or fewer) physical sites to manage, and lower bandwidth costs. Every little bit helps...

In any case, it is my opinion that any computer which can communicate with others on the internet, no matter how well-restricted such communications are, should itself be considered non-trustworthy. It might be safer for being behind a firewall, but it can still grab a trojan or worm either through accidental or intentional means and become a staging point for internal attacks. It is for this reason that I personally believe that it is imperative to ensure that every computer on a network is secure and has personal firewalling of some form installed (if you're dealing with *nix workstations this is a no-brainer for a competent admin; Windows boxen will benefit greatly from simple solutions such as Tiny Personal Firewall.)

This all goes double for boxen which are physically located outside of the network and which VPN inside (this is the reason for that last paragraph's worth of rambling.) A certain amount of distrust should be exercised for computers which can find themselves poorly protected from the dangers of the internet at times, and as such it is not only necessary to keep such boxes under close scrutiny and send their traffic through a decent firewall, but also to either educate users (as well as possible) on good security or require as a matter of policy that they utilize certain security measures (a personal firewall combined with a regularly-updated antivirus application is a potent combination that goes a long way towards keeping a computer clean.) Assuming that a VPN is a safe connection is a recipe for disaster; it prevents others from listening in but otherwise it is no better than any other old TCP/IP connection.

VPNs, of course, can be quite useful on an internal network. Packet sniffers tend to have difficulty picking up on SSH as it is, but put that through a 1028-bit encrypted tunnel and it become exponentially more difficult to crack apart (and such layering protects you from vulnerability as there are now *two* effective locks which must be picked in order to gain entry.) It isn't going to make a difference between two servers connected with a crossover cable and which enjoy strict physical security, but when traffic is being passed over a network with old windows 95 boxen running Outlook, it pays to be prudent. Such encrypted seperation, when used intelligently, can often eliminate the need to physically seperate network segments when connectivity can be useful.

Oh, one last point: if you're using a WLAN, it's only logical that unless it's strictly for visitors doing web surfing and chatting on AIM, a VPN is useful there as well. WEP is both less useful and far less effective.

As for a good VPN technology to use for any application, IPSEC is always handy (and enjoys excellent and robust out-of-the-box support in the more recent revisions of... almost everything.)

Sorry if this seems a bit unclear, but I've had a long day. :)

Immature Technology

[John Paul Jones]

Are application level firewalls sophisticated enough to allow machines on your internal network to advertise services to the Internet?

Nope. That should never happen.

The problem here is that application-level firewalling is fraught with problems. The lack of intuitive management for this type of firewalling is a problem that quite a few companies are trying to solve -- with limited success, so far. The problem is that as you move up the OSI layers, the variables increase exponentially. If you think that 65,536 is a big number, try writing an application-level script that permits "acceptable" MAPI requests while denying "unacceptable" MAPI requests. How do you determine that this NFS packet is good, and this one is bad? From the same host to the same server? How about X11? SSH? Oh, and don't break anything while you're at it. Lions and tigers and bears, Oh my!

These are the problems of an immature technology. As time passes, these issues might be somewhat mitigated, but there are plenty of "network administrators" that haven't fully grasped the concept of IP, and struggle with L3/L4 firewalling, to say nothing of moving up the stack.

Here's a tip, though; look for Bayesian filters in firewalls in a few years. That will be a trip.

vpn is NOT a magic word

[smitty45]

VPNs are great until you realize that they provide only *temporary* access to your office network. What happens to those road warrior's machines when they're not vpn'd but still on the internet ?

are they firewalled properly ?
are their virus definitions updated ?

if no or "don't know" to either of those, then having a VPN will compromise any amount of safety it could bring. in other words, it's possible that the lastest and greatest worm that wasn't able to penetrate your office network until you patch is now vulnerable due to the work-at-home employee who VPNs in, and is now infecting everyone.

a bottom line is to have a well thought out security policy and PROCESS....and that only comes with training, more training, and training. Some education would help, too. Even people like Mudge and Dan Greer don't stop learning.

and for those who would call your questions stupid...they are the folks who are afraid to ask the stupid questions.

ideal vs practical

[vt0asta]

You're going to get a lot of answers on how in the perfect world there will be DMZ this, several layers of routers that, firewalls in between them all, VPNs, NIDs,and a whole bunch of other things that may not be applicable.

The answer really depends on what you are protecting and whether or not the security required to protect it is worth the cost.

The only way application aware firewalls CHANGE the paradigm of network security models is for a certain class of protection. Usually that line of protection is or train of thought is "we would like something slightly better than nothing".

If you need protection more than that, it sounds like you already know what is best practice. That hasn't changed, and you are not wrong to suggest to your co-workers otherwise.

Think of it along the lines of what the military would do. Just because there is some new whiz bang motion tracking CCTV x10 ninja thing that shoots lazers. You better believe they are still going to have soldiers with rifles in watch towers, soldiers walking the perimeter, and 20ft of dead man zone and razor wire fences surrounding, along with the whiz bang consolidating gadget.

Some add'l tidbits

[Anonymous Coward]

First off, remember - you won't be able to think of everything. No security model is complete without behind-the-wall systems, be they basic monitoring systems up through more sophisticated custom snort or proprietary IDS. It all depends on your paranoia level.

There are a few ways to handle the bane of netadmins - 'I wanna get to my files!' VPN, as suggested, is one solution - but not without problems. Recent issues with X.509, OpenSSH hacks for IP-over-SSH, etc. You can mitigate the danger by using a set of consistent criteria for each of your requirements, like a checklist. For example:

1) Is the service mission-critical? (BOFH them if no!)
2) Can the service be offered through a less-vulnerable channel? NFS mounts moved to SFS, perhaps, or encrypted AFS as mentioned above.
3) Is there a way to move the service into a perimeter network (or outside entirely)? Even if this means synchronizing a set of data to an outside machine via cron, if the data on the machine is less important than the internal network security, this can help.
4) Once the user is connected, authenticated and accessed, *THEN* what can go wrong? What could they do maliciously? What could they do accidentally?

Personally (and this is just me talkin', no creds here) I tend to reflexively say "NO!" until convinced otherwise. I know that there are services which *must* be available through the wall, but I want the requestors to have to work to convince me. Closed systems are more secure.

Also, don't be afraid to investigate low-tech but simple and effective means of circumventing problems. First thing I ask users who want to get an occasional file home - "Can you mail it to yourself?" Second thing: "Would you be able to use a 'public folder' that I have synch to an accessible box, say, every half hour?"

I second the opinion of iptables. It's a sharp tool, so be careful - but correctly applied, it kicks pants off most application or appliance firewalls. Invest the time to learn the sharp tool, and you'll realize that most of what you pay for on big expensive firewalls is manageability (i.e. Java GUIs, wizards, databases, multiple systems preconfigured - IDS, firewall, proxy, etc). Do the work.

Good luck. Don't listen to people who berate you for 'not knowing things.' Attempting to learn them in advance - due diligence - is a sign of a good admin. Be thorough. And above all, find a friend who does the same kind of work, and check each other. Probe each others' networks. Try exploits posted on the net.

Final, and most important - software updates. The boring part, but the most critical.

Cheers.

I depends on the size of your network

[egarland]

There is no one answer. If security is your only concern you should have as many layers of security as possible with firewalls between each layer locked down as tight as possible. That said, security is never your only concern. Cost, ease of maintenance, performance, and flexibility are all important in a network design. After all, the purpose of your company is probably to get something accomplished, not to avoid getting hacked. There are times when every different network configuration is appropriate from super secure to a cable modem router to a windows box right on the internet. There is no one answer.

Application layer firewalls are another layer above port filtering. They can increase security and could, in theory, make it worthwhile to share a service hosted on a machine that is inside your network. I would only do that if you trusted the security of your internal network. Most network designs assume that once you get in to the "internal network" there is no more security and all your deepest company secrets are available to anyone browsing around. If this is true, you've probably made some bad decisions somewhere along the way and you should address those before you open any holes. If you are willing to maintain strict security on your internal network then the added simplicity of allowing Internet access to machines on it can be worth the risk. This can be a lot easer than setting up a dmz.

Usually layers do make sense though, even if one of the layers is just a Linux box doing firewalling, routing and serving some services. One thing I like to do is to mix operating systems at different layers. That way if you get a worm of some kind that gets into one layer it won't penetrate to the layer behind. For example, internet facing servers are Linux based, desktops are Windows based.

Another thing I have done when I absolutely needed a Windows based web server is to setup Apache as a reverse-proxy only forwarding requests to a particular subdirectory to the Windows server. This filtered out all the standard buffer overload attacks since none of them referred to that subdirectory name. It also made sure the requests were relatively well behaved and buffered outgoing data for the Windows box, reducing connection counts when it was under high load. This is an easy way to do an application layer firewall and if you are firewalling with a Linux box you can do it right on the firewall.

Are you NUTS?!

[TheDarkener]

Is the simple concept of opening and closing ports insufficient for networking services that require the client and server to open multiple simultaneous connections (both incoming and outgoing)?

I am the head sysadmin for a company that has many Linux, Windows, and Solaris servers, and other specialty systems such as Cobalt Raqs, proprietary satellite equipment like IP enabled RF Modems, MUXes, IPEs and a shitload of high-bandwidth routers in multiple POPs around the world. If you think that a firewall to protect your network is insufficient, especially for a network with mixed OSes and such, you are terribly wrong. Imagine working in an ISP. You have your private workstations, then your servers (DNS, MXes, etc.), then your colocation equipment. Put it all on the same network? Suuuuure!! WHOOPS! Someone hacked into a colo box and then used his r3wt account on that box to scan your internal network for other vulnerable boxes (all at the same time, using your T1/T3/OC-192 for hosting the world's biggest movie IRC bot). You didn't have a firewall and/or IDS to detect the initial portscan on the colo box, and now you don't know that he/she is sucking up your bandwidth and scanning your entire internal (well, to you it's internal, external, whatever) network for more boxes to royally *$#! up. Trust me. Once a box is rooted, you take it of as SOON AS POSSIBLE and reinstall. It's a shitty feeling knowing that someone owned YOUR network and now you have a shitload of crappy work to do over the weekend. Not to mention downtime, customer/employee complaints, fielding the hundreds of "I CAN'T CHECK MY E-MAIL!!! BOO HOO!" calls, and general feeling that maybe...just maybe there's a box that got 0wnz0r3d that you might not know about.

The moral of this story, boys and girls, is that FIREWALLS ARE GOOD. Intrusion detection systems are GOOD. NAT is GOOD. TCP syncookies are GOOD. Everything on the Internet is vulnerable by default unless YOU TAKE THE TIME TO SECURE IT YOURSELF. Keep the colo systems on their own subnet. Shit, keep each SYSTEM on it's own 2 port VLAN with the uplink. Keep your servers on a DMZ. Keep your internal workstations on a TRUSTED, PRIVATE, NATted network. Close every damn port besides the ones that are used by servers. Do not open ANY ports to your trusted, internal network. If someone roots a box, at least they can't load an SSH trojan on port 2000 with no password and automatic root access to get back in later.

Who cares about the network?

[rc.loco]

Firewalls are great at slowing down intrusions. However, without proper application security architecture and host-level security hardening, you cannot really protect a network-accessible resource. Often times, the only resource (network, application, host) that we can control 100% of the time so that it can be trusted is the host.

Besides, the bulk of compromise situations occur INTERNALLY. Is that PIX on your WAN router really going to stop disgruntled Gary down in QA from trying out across 5 subnets the latest script kiddie tool that his roommate hooked him up with. If you spend quality time hardening your hosts, chances are you may really not lose more than a few hosts at a time during a significant compromise at the application-layer (e.g., a remote root sendmail hole, a bug in BIND). I think we need to revive the popularity of security "tuning" on the host side - a lot of people forgo it for strong network security but I think that the latter is a much more difficult perimeter to maintain.

I've seen others post about the dangers of VPNs. I totally agree, they are conduits for information loss, but are likely to be mostly self-generated (internal). Example: Disgruntled Gary in QA sucks down the product roadmap details off the Intranet before giving his 2 weeks notice and starting to work for a competitor.

Apologies to Gary's everywhere. ;-)

The Internet Will Become Port 80

[ObligatoryUserName]

Sad to say, but in the future, the only reliable port will be 80. All clients will have all ports except 80 blocked by default (right now this seems like wishful thinking!) and no one will open any other port (it will give them a scary security warning!), and even if they wanted to, they might be blocked from doing so by their ISP.

We're already seeing shades of this, but it hasn't reaced the majority of Internet users yet. Back in late 90's my company rolled out a product for schools that to be retooled when it was realized that many schools were firewalling everything except port 80. (They added a mini proxy server to the product that sent everything over 80.)

I have a friend that's a sysadmin for a medium sized insurance company - and they had all their internal applications break a couple weeks ago when a MS worm started bouncing around the Internet. However, the problem wasn't that they were using Windows machines (I think all their servers were AIX...)- the problem was that their ISP (the regional phone company) had blocked off the port that all their applications used because it was the same port that the worm used to get into systems. Last I heard, the phone company was refusing to ever re-open the port. (The phone compnay made the change without even informing anyone at the insurance company, everything just stopped working and from what I heard it took them a day to figure out why their data wasn't getting through. I believe they were resigned to changing all their programs to work on a different port.)

So, we've already come to the point where connections on other ports seem strongly subject to the winds of fate, and I see no reason the situation won't get worse. In most environments 80 is the only port that people would notice if it was blocked, and there are too many sysadmins out there who don't know any better. Right now, if I was developing an application that needed to communicate on the Internet, I would only trust that it could use port 80, and I wouldn't even bother looking at anything else. You can even see application enviornments starting to spring up now (Flash Central) where it's assumed that most applications will just share a port 80 connection.

It sure is a sub-optimal situation, but I don't know what can be done to stop the trend. Ironically, such a situation makes simple port-blocking firewalls useless because all applications will be running on port 80 anyway.

How you will get r00ted

[btg]

I have been involved with lots of different bits of security for a few years now, and quite a few people seem to think I know what I'm doing.

Playing the "security component Lego" game is great fun, and a little intelligent thought will soon see you set up with a nice, best-practice architecture. This is how it will then fail.

1. You will have unpatched machines which will be trivially rooted with a script-kiddie exploit. You will know that you should have patched, but you won't have the time, manpower, or authority to ensure the patches are in place.

2. You will misconfigure something, and then miss the problem in reviews because you didn't get peer or professional verification of all your configs.

3. You will get owned by an internal employee who has exactly the level of trust that you planned for, but abused it.

4. Someone will walk in with a clipboard, bamboozle the secretary and walk out with your fileserver.

5. You will create a whole bunch of really cool procedures, but the CIO / CTO won't back them when the first departments complain about lost productivity - this will undermine the whole thing and you will be back at square one.

6. You will give someone VPN access, and they will connect their virus and worm ridden home machine. It will infect your network, and their kids will surf pr0n and share mp3s on your dime.

7. Your backups will have some unforseen problem, your restore procedures won't work right because they aren't tested, and you will lose much company data (and your job).

8. Your users will deliberately download trojan-ridden, virus infected, IE Object Overflow infested garbage, despite clear, explicit orders to the contrary being sent to them twice a day. They will do this because dancing rabbits are somehow more compelling than 'all those emails from the grumpy tech guys'.

When we talk about the 'current paradigms', I don't even think about fancy technology, I think about these obvious threats that always apparently only happen to other people, because some wiseguy always knows better. "IF you do blah blah, like we do..."

Your "paradigm" wish might be: "I want a network where every single part is doing as best it can to defend itself against the threat at the keyboard as well as the threat from external attack - not a perimeter, not 'tiers', but every part."

Re:Encapsulating protocols is a "bad thing"

[oniony]

Layering came about because of the inflexibility of systems administrators to react to the need for new services to be accessible. HTTP is one of the few protocols that are allowed through firewalls because of over-zealous blocking. Because of the need for applications to work, people have realised that the only way forward is to get their protocol to run over HTTP, hence SOAP and the rise of XML. I've experienced this need to layer first-hand on many occasions.

Developers tend to do the least work necessary to achieve the result they desire. The fact that so many protocols run over HTTP now indicates that the developers of the applications that use these protocols have been unable to persuade the systems administrators to open ports so that they could their necessary applications to work. Instead they resorted to the harder task of layering to avoid the blocks.

The sysadmin that said "I like people to do some work to convince me" says it all. The attitude is that of a power-monger. A pragrmatic sysadmin would work with the applications developers to find a solution. Maybe they frown upon opening ports for applications, but they should at least put the effort in to explore the options otherwise we'll always end up with this layering effect for every networked application. I wonder how long it takes before we end up with protocols running over HTML/HTTP to avoid the application firewalls that start blockting non-HTML HTTP traffic.