Protecting Your Enterprise Network from Vendor App Servers?

anomaly wonders: "I work for a company with a large IT infrastructure. We have lots of applications in our environment. For a number of applications, vendors provide the apps, and provide core support to those app servers. Our vendors are notorious for demanding superuser access to the boxes that support their applications. To protect our enterprise network from attacks allowed in by well-meaning but less-than-perfectly-competent vendors, we have set up a quarantined network for each vendor. This works well when the model is ASP-like and all of the components live on a single box, but fails when the application needs to be connected to one or more enterprise applications (RDBMS, smtp, they want backup, etc) or when it needs to be connected to lots of target systems inside our environment on lots of different ports. How can I restrict a vendor/application server's access to our enterprise network while still providing platforms to make the applications productive for our user community?" "Frequently vendors can't restrict their applications to run on a limited set of ports. Most of the time they stare blankly when we want their application to run as something less than superuser.

Our biggest challenge is keeping track of all of the dependencies and managing what ports need to be allowed to which destinations. Of course, when security is tight our business-types say 'you're breaking my application.'

What can you suggest about how to provide access to applications, patch/protect the OS on the app server, and protect the enterprise network? What does your organization do?"

Comments

Layer 7 Firewall

[passthecrackpipe]

You need a Layer 7 Firewall, a.k.a. an application level firewall. Something like Zorp is a good start, but you probably need something with a bit more intelligence about the applications you are talking about though.

L7 Firewalls usually get a bad rap because they tend to be pretty fussy in setup, something you can't really avoind with this kind of stuff. Also, if I was in your shoes, I would learn to stop worrying and start loving tight-ass SLA's...

Re:Layer 7 Firewall

[Mattcelt]

Application proxy firewalls work just fine - Raptor makes an enterprise-class AP firewall that does a fantastic job. They require more resources to manage than simple stateful-inspection firewalls, but they are much more secure when managed properly.

As far as Mandatory Access Control (MAC) goes, it is even more difficult to manage than an AP firewall, and a terrible pain to implement - ever tried to make a MAC model work in an Active Directory environment? Not easy...

And even if you just choose to implement the RBAC part of a MAC model, how do you define roles? Unless you have a very stratified and well-defined role structure for the people who work in the enterprise, it is a daunting task to set up roles. In one place I worked, there were ~10,000 employees - and 4800+ job titles. Not exactly conducive to role assignment, to be sure.

Consultancy

[KontinMonet]

Well... don't get EDS to work on it!

Tell them to screw off..

[Heem]

Seriously.

I built a network for vendor lab equipment that was it's own netowrk, connected to the main network via a file server with 2 nics and NO routing. Therefore, file could be transfered from the lab network to the file server, and then from there to any of the main networks, of course after they had been scanned for viruses and such, since anti virus software usually would not run on their equipment. My rules were firm and backed by higher-ups - if you couldnt get your equipment to work in the enivroment given, or with only minimal flexibility from me (for example i'd write scripts to move their files automatically to the main, backed up network or something simple like that) - then we will find another vendor to provide us a solution that fits. Period. I never had any problem with a vendor once they heard the terms. They won't make money off us if they can't make the system work in our environment.

Re:Tell them to screw off..

[ischorr]

No, he just knows how to actually do the job he's being paid for. Once you graduate high school and eventually enter the *real* world of IT, you might understand this.

Re:Tell them to screw off..

[Jonathan the Nerd]

"I often reflect that if 'privileges' had been called 'responsibilities' or 'duties', I would have saved thousands of hours explaining to people why they were only gonna get them over my dead body."

-- Lee K. Gleason, VMS sysadmin

Switch vendors

[31415926535897]

Our vendors are notorious for demanding superuser access to the boxes that support their applications.

There are a few things you can do in this case:
1. Get a different vendor (if the application truly does require su for vendor support).
2. Reqest different support staff from the vendor (do this if the app doesn't require su but the support staff is too lazy).
3. Learn how to support the app in-house

I am very serious on these points. You do not want to give root access to people that should not need it. If they say they need it, they have an awful application.

The place I work has a vendor area fenced off from the rest of the server room, and the vendors only have access to what they need. If they need more privileges, the IT guys watch them like a hawk until they're done.

Re:Switch vendors

[Eudaemonic Pie]

Exactly right. Vendors never *need* root on our box. They often *want* it because it makes their job and their life easier. With properly applied permissions, there is very little a vendor cannot do just using the application owner id. The exception being if their app server binds to ports 1024 and they need a restart. Anything else, like oh adjusting permissions of files they don't own, applying OS patches, rebooting the box, killing processes they don't own, etc, etc aren't things I want my vendors doing *anyway*. Depending on the size of your shop, you may have controls or processes in place that require approvals before any of that works gets done, so why let the vendor go around your process if you can't?

Re:Switch vendors

[ischorr]

"If they need more privileges, the IT guys watch them like a hawk until they're done." ...And hopefully your staff *is* there for them when they need other access, questions about other pieces of the environment, etc.

I've worked escalation support for a large storage vendor for a number of years, and while I completely understand (and agree with!) limiting vendor access, I find that IT departments tend to be *very* unresponsive when the vendor support folks need their help.

It's not exactly efficient, for example, when I have to wait 7 hours for logs to be uploaded, 5 hours for the "network guy" to respond to a page and fix the duplex mismatch he created that's causing 50% of packets from our systems to clients to be dropped, weeks for the system admin to stop piddling around and get Sun on the phone when we have valid interoperability issues (Sun won't talk to most other vendors except through the common customer), etc.

In the meantime, in my experience, the CIO tends to lambast the vendors for "poor responsiveness" and "terrible support" - though the smart ones eventually realize that the IT staff is shooting themselves in the foot, and do something about it.

IT departments that are responsive to vendors, on the other hand, are a breeze to work with, and issues are typically resolved many times faster.

Re:Switch vendors

[theonetruekeebler]

Sometimes switching vendors just isn't an option. There are lots and lots of niche markets where there's that one tool that everybody has to run. For example, at a motorcycle dealerships most parts fiche CDs will only work with a specific parts/sales software tool---which runs on NT and "needs" to run as Admin, even though it is only an elaborate piece of middleware connecting a database to a few desktops running the application. It is (as of mid-2003) also a slow, buggy piece of shit.

When the only alternative to required software is working by hand (or a major reverse-engineering project), you just gotta suck it up and figure out how to protect the rest of your systems from their arrogance.

Generally speaking, it's a clear sign of laziness or incompetence on the part of a half-assed programmer to think he needs root for everything. Hell, Oracle doesn't need root to run, and it's a mighty damned complicated RDBMS suite. If you're stuck with one of these vendors, I urge you to make it clear to them that you are using their software because you are stuck with it, that given the chance you will jump ship in a heartbeat, and that the reason you'll never buy any of their other products is that their claim to "need" root is a sign of either ineptitude or a cavalier attitude towards their customers.

Re:Switch vendors

[cduffy]

There's a different kind of situation:

That where the vendor's software is on a "black-box" machine that only they have administrative access to, and which runs no other software.

I'm (senior deployment engineer at) one of those vendors. Not only is our app fairly complex to administer, but every server that goes out contains a copy of the "secret sauce" -- not our application itself (which our bigger competitors could probably recreate in 9 months or less), but the data behind it (much more expensive and difficult to recreate). Consequently, our management is paranoid -- the servers are rigged to self-destruct (wipe the keys that allow them to decrypt the partition where all data is stored) in the event that any attempted tampering is detected, and can only be reenabled by the very small subset of our staff w/ access to the private keys.

Right now our app has components that run as 4 different users (to isolate breakins to the component where they occur), and includes a firewall and a VPN solution (both of which need root for obvious reasons).

Since it's our box that's running the app, and that same system does nothing else -- why restrict our access further? Absolutely, firewall us off from anything we don't need -- but restricting access to our box seems silly (and is something we'd consider only for a very large customer -- which is just as well, since the small folks we're selling to right now haven't had a problem).

In any event, I'm curious to hear how you'd respond.

Don't let them

[illumin8]

Frequently vendors can't restrict their applications to run on a limited set of ports. Most of the time they stare blankly when we want their application to run as something less than superuser.

Simple answer: Don't let them. This is standard operating procedure in the financial services industry. Do you think that a bank would EVER let a non-employee or non-contractor access any bank system whatsoever? Especially remotely? If these companies want to do business with you, they WILL play by your rules or you'll pick one of their competitors products. In my experience, the only companies that required remote access to their systems were ones that A. Didn't have a fully working product, and B. Had to have the developer log in constantly to patch binaries with the latest bug fixes just to get a semi-working product. These are not the type of companies you want to do business with in the first place.

Speaking as a sysadmin for a smallish financial services company that processes around 1.2 million transactions a month, I would NEVER allow any vendor remote access to our network. It just wouldn't happen. Even if I did want to give them access, there are rules and regulations that strictly forbid my giving them access. If they give you a hard time, make something up about a security audit or something.

Seriously, you are asking for trouble if you let them have access. Who's going to take the blame when one of their developers logs in and wipes out all of your company's data? Chances are they'll blame it on you and you'll be in trouble for their mistake.

My suggestion

[oexeo]

Our vendors are notorious for demanding superuser access to the boxes that support their applications. To protect our enterprise network from attacks allowed in by well-meaning but less-than-perfectly-competent vendors, we have set up a quarantined network for each vendor.

What can you suggest?

Find some better vendors?

Vendors are asshats

[Anonymous Coward]

Seriously ... . I'm a sysadmin for a medium sized accountancy company, and you wouldn't believe the number of vendors I've had to beat off. I end up talking them down to lower levels of access, all the while listening to the lusers talk down to me as if they knew more about running it on *my* network than the local sysadmin.
In the long run though, I've figured that these lusers are going to be more trouble than they're worth, and am talking the boss into replacing the NetApps and Junipers (routers) with Gentoo Linux boxen. So in short, don't protect, just reject!

Simple use a Firewall

[cybrchld]

I have a similar configuration between two in house networks I use the firebox X700 in routing mode it allows me to open only the ports needed to be routed between the networks and also allows me to monitor all traffic to keep an eye on the test network side.

VPN

[Julian Morrison]

Create a dedicated VPN. The misbehaving app server sees only this VPN. Everything it needs to access has a presence thereon, carefully firewalled to allow the relevant ports to open. Everything it doesn't need to access, is not even on the network.

VPNS are handy...

[eldub1999]

We force the vendors to enter via VPN. we use the VPN gateway to restrict each vendor account's access to only the IP addresses of the systems they need access to. Further, we occasionally use a packet sniffer to watch certain vendors.

We disable the account by default and require them to contact us and tell us what they are doing (change control) before letting them in.

Works for us.

They can screw up without root

[Burdell]

I work for an ISP, and we do not give anyone outside the system/network administration team root access to anything if at all possible. We have given network vendors (Redback and Ascend/Lucent) remote access to a device a couple of times (we changed the passwords on the particular device before and after), and we have one web application that we had to give the vendor root access for setup (but that was on a dedicated server).

Our antispam server is a dedicated server, but it still was screwed up once by the vendor. We were having a problem (only half our mail was actually being processed by the filters), and management directed that I go ahead and give the vendor support person access to the server (as the user the software runs under). He logged in to look at it, and within minutes the system was broke and mail was queueing up because the antispam software shut down and wouldn't restart. He had seen something unrelated to our problem that he thought didn't look right and decided to "fix" it for us. As soon as I got him to change it back, I told him to log out and removed his access (and they won't ever get it again).

After that, the only other time we considered allowing a vendor access was on a problem case that was over a year old that we had to have fixed ASAP. However, in that case, we were NOT going to allow remote access; the vendor was going to have to send somebody to us and we'd sit him down in front of the console (which would be logging) with us looking over his shoulder.

The only people that know your systems and network are your people (and you'll make enough mistakes). Vendors should not have access to change things at any point; at most they should get a "view" type account (they can look all they want but they can't touch). If something needs changing, they need to tell you and you can make the change (after evaluating it and having a back-out plan). For complex systems, you never end up installing software exactly the way the vendor specified; they are not going to know or understand changes you made for your local configuration (and how such changes affect other services and systems). Even a well-meaning "fix" can cause serious problems, and since it'll be your job to fix them, you better know what was done.

I go through this all of the time

[flying_monkies]

From the vender side.

First: It's amazing how many people haven't got a CLUE. "Don't give it to them or get another vender" isn't an option. When you're running a 24x7x365 system and expect sub 1 hour response time from your venders (our customers do) you're going to have to give some.

All of the boxes we hook up have phone lines into them. If security is an issue on dial-in access, we set it up as follows:

Modem is enabled dial-out only, no dial in access.
If the box dials home, we contact the administrator, identify who we are and which box has troubles and have them enable access to the server/hardware and reset the root password temporarily on the box. This occurs only if it's something we haven't already configured to work with sudo.
We then keep logs of the entire session, then email it to the customer when we're done.
When we're done, we have them reset the root password and disable dial-in on the modem again.
If we require access to a network resource from the machine, we're onsite with the customer shoulder surfing.

We do this at secure military sites, financial institutions, etc. and have yet to have an issue in 20 years.

Several different possible solutions

[postbigbang]

1) write in to the SLA policy-- a SU agreement with liablity
2) use (as mentioned) a L7 switch (Telena has one, too)
3) give temporary access, and make sure you check for root kits everytime with a script
4) tell your management just how expensive it is to have so many vendor's spoons in the soup and how potentially destabilizing it is to do this
5) use smart token card access coupled to your own CA; Tie the proximity of the card via RFID to a pacemaker attached directly to the aorta. If they lose it, they die. Simple.
6) partition roots across servers. Get an SNMP trap when they logon to keep track of them. Set a script against cron to send an additional alarm when they're on for more than a few minutes or upload more than a few megabytes through specific ports (indicating massive changes rather than remote control screen delta)
7) ask for one of their children for hostage use

Check-out the FreeBSD jail facility

[Diomidis Spinellis]

[I know this will cost me karma points.]

The FreeBSD operating system provides the jail(2) system call and the jail(1) command for imprisoning a process and its future decendants. The jail facility is based on the chroot(2) implementation, but prevents well-documented means to escape chroot confinement, offering partitioning of the file system, process, and networking namespaces. The facility removes all super-user privileges that would affect objects not entirely inside the jail.

For more information read:

• Poul-Henning Kamp and Robert Watson. Jails: Confining the omnipotent root. In Proceedings, SANE 2000 Conference. NLUUG, 2000.PDF, HTML.
• Poul-Henning Kamp and Robert Watson. Building Systems to be Shared Securely ACM Queue vol. 2, no. 5 - July/August 2004 HTML

A technique

[Gyorg_Lavode]

A technuiqe my work employed to get people to stop requesting things is to make some simple form to fill out to get what they want. But then require 2 or 3 signatures. (Their supervisor, their company sponsor or contact and their own.) Then you take 3 or 4 weeks to process any of these forms, (purposefully). And you deny half of them.

Then you make your policy strictly exclusionary. And when they say "BUT I NEED THIS!", you say, "Ok, fill out a form 23" or whatever the form is. They'll learn quickly that they aren't going to get many of them approved and they'll start putting them in only when they really need them.

Re:Shameless plug

[jellomizer]

No just threaten to call IBM global outsourcing. That usually gets the venders to play nicely. But if you Really get IBM inside your company you will reach new levels on non-productivity. Just because they manage the largest global corporations doesn't mean they are good.

Re:Find a new vendor

[Fallen Kell]

This is what we do. Seriously, we don't let ANYONE have access. If they "need" root for something, they call us up and they tell us what they need to have done and we do it.

I work at for a large government contractor, where root privilage is truely guarded to only those who need to know as there is data that absolutely need to be protected. Allowing random people from random companies access is not an option. It is actually a criminal offense to let someone else use your account who does not also have an account on the system (it is still a security violation even if they do have an account and is possibly subject to disciplanary action, up to and including criminal punishment depending on what it was being used for).

Basically, the only people who have root are the actual system administrators. And that is even locked down in the sense that we are supposed to log in with our regular account first and "su" to root, or otherwise need to call up security and let them know what system you need to log into directly as root (login and audit logs are checked on a regular basis for any discrepencies).

Basically, like I said, if they need root, they go thru one of us to do something.

Non-technical reasons!

[TheLibero]

Sometimes those a$$holes ask for such things in order to get better view of your infrastructure and update their sales account with information about you. Let me use a non-technical example here. You own a super store that is specialized in Sports clothing. You have a 5-year contract with Nike to supply you with trainers. The contract is gonna expire in 6 months. Also, you have another contract with Puma to supply you with arm bands. Recently, customers have been complaining about the quality of those arm bands, so you contacted Puma (now imagine that in IT world), Puma sales would ask you if they can send a representitive over to check the stock you've got. They do that not only to check the stock, but to check what other products you've been stocking from competitors, so they can update their accounts and have better picture of the areas they have competiton in, and against what companies.

There is big marketing battle just behind you!

heh, my experience is the opposite

[stratjakt]

As one of those vendors for governments, I have to constantly deal with some moron admins who refuse to give me any access to the machine, yet CONSTANTLY call for support.

I don't generally need superuser access on the machine, since generally the only thing that gets screwed up is the data in the RDBMS (you know, user error).

I had one propellerhead go around in my database deleting tables and columns he felt they didn't need. He told me on the phone "we don't use timestamps here". One of those slam your head on the desk conversations. These are civil servants with lifetime jobs, and maybe they knew all about VAX in 1970, but goddamn if they aren't dense.

They tend to think that RAID is a magical "never need to backup ever" solution. I just love it when they call me up after their second RAID-5 drive failed, and I ask them when they last did a backup - and they go "uhhhh we don't need to backup we gots RAID".

Then I explain how RAID has nothing to do with archival or backups, etc, etc.. And I pull out the backup I made last time they had a major upgrade and tell them they have to reenter every parking ticket for the last 8 months, and they threaten and bitch how it's my fault and I tell them I'm not their admin, and if they really want to go to their bosses and fess up how incompetent they are they can go ahead.

Frankly, I'd love for some more competent clients. Of all of them, I can think of one who has any clue what to do with a computer.

But then sometimes they call with a problem that requires fixing on the machine. I'm not going to sit on the phone talking them through shit, I'm not going to email them scripts or code, etc. More than once I've had to tell them that if they don't give me access it wont get fixed.

If it's a problem for you, give them superuser rights when they need it, when they're done doing maintainance, take it away.

As one who's worked on the vendor side...

[Eneff]

1. Smaller niche vendors usually don't have the hardware or manpower to simulate your system. You went with them because they were half the price of dealing with IBM, remember? :)

2. Often, those applications that require Superuser for the installation are third party applications. Install it yourself. You may have more trouble on the outset, but you'll know what's going on with your machine.

3. Often, vendors have their programmers as installers. The bad news is that you see the problems you do with the installations. The good news is that they'll know exactly why they need root - and they'll tell you what they need done. This might need to be a tag team installation, of course.

4. Remember, you can always invite them up there if you want to pay them for their time. Remote access is requested because it's cheaper. Alternatively, put in the contract that you want installation instructions. It will take more time, but you're always welcome to pay more.

Many of these vendor problems are reduced to cost-cutting measures. If you want to pay more, then vendors will be willing to oblige.

Re:You don't

[MattBurke]

You obviously can't have ever dealt with anything of importance then...

If you have something go down on you which is costing several people's salaries per hour in lost revenue, you don't sit around for days trying to sus it out - you get the vendor to fix it. That's why you pay them for their support. You do pay for vendor support don't you?

They know their products a hell of a lot better than you ever will, and you'll likely have a contract saying it will be fixed in a stated timescale or they'll be paying you compensation. Often vendors will courier out replacement hardware instantly before the engineers have even sat down to log into the device, and those engineers will be of the highest quality if you've flagged a critical call. If you've triggered a bug in a piece of vendor kit, how long will it be before you've diagnosed it to be a bug? Then how long will it be before you get a fix? I know of at least one large company which has compiled and installed a new software image on a customer device within hours of a critical fault call being raised. Would they do that if you weren't paying for support? I think not.

For maintainence however, you learn how to do it yourself.

Re:On demand and via a process

[SparklingClearWit]

What if the problem is caused by bogosity in a config file that only root can read?

Then it's written wrong. I want the config files to have permissions for a user named $vendor.

What if the logs produced by the application are only readable by root (or by adm)?

Then it's written wrong. I want the log files written to a location for a user named $vendor.

What if the process is running with root privileges and you need to trace/truss it to perform the diagnose it?

Then it's written wrong. I want the application to run in a chroot or jailed environment limited $vendor and $vendorapp.


There is *no* good reason for a vendor app to run as root/superuser/Administrator. It's poor coding, and poor design. If that's the best your 'cheaper' vendor can do, you'll end up paying more in the long run for support, security problems, and related issues. It's too easy "since you're root" to just change file permissions to make your app 'just work', instead of correcting the application to work within the system constraints.

On a similar note, us logging RDP possible?

[JoeShmoe]

May not apply to *nix but for Windows servers, I answer requests like this with Remote Control on a Terminal session. If a vendor needs to get "Administator" level access for some reason, I push them through Terminal Services then remote control their session so I or someone else can watch what they do. If the vendor is smart (haven't found one yet) they are aware that someone can watch what they are doing, but they aren't so I usually get to watch them do all kinds of boneheaded things while they take a guess and check approach to fixing a problem. The downside for me is I have to take notes so when the vendor finally disconnects I can undo anythign they did that was outside the scope of their repair.

Which makes me wonder why I can't record an RDP session somehow. This would serve two purposes: a) it would let me replay what a vendor had done later when it is more convenient and b) it would give me some proof if I later have to go blast a vendor for doing something absurdly stupid.

The same question for VNC or really any kind of remote management tool that is likely to be used on the Windows platform. Can any of them be logged and/or replayed?

- JoeShmoe
.