Slashdot Mirror
Protecting Your Enterprise Network from Vendor App Servers?
anomaly wonders: "I work for a company with a large IT infrastructure. We have lots of applications in our environment. For a number of applications, vendors provide the apps, and provide core support to those app servers. Our vendors are notorious for demanding superuser access to the boxes that support their applications. To protect our enterprise network from attacks allowed in by well-meaning but less-than-perfectly-competent vendors, we have set up a quarantined network for each vendor. This works well when the model is ASP-like and all of the components live on a single box, but fails when the application needs to be connected to one or more enterprise applications (RDBMS, smtp, they want backup, etc) or when it needs to be connected to lots of target systems inside our environment on lots of different ports. How can I restrict a vendor/application server's access to our enterprise network while still providing platforms to make the applications productive for our user community?"
"Frequently vendors can't restrict their applications to run on a limited set of ports. Most of the time they stare blankly when we want their application to run as something less than superuser.
Our biggest challenge is keeping track of all of the dependencies and managing what ports need to be allowed to which destinations. Of course, when security is tight our business-types say 'you're breaking my application.'
What can you suggest about how to provide access to applications, patch/protect the OS on the app server, and protect the enterprise network? What does your organization do?"
Our biggest challenge is keeping track of all of the dependencies and managing what ports need to be allowed to which destinations. Of course, when security is tight our business-types say 'you're breaking my application.'
What can you suggest about how to provide access to applications, patch/protect the OS on the app server, and protect the enterprise network? What does your organization do?"
You need a Layer 7 Firewall, a.k.a. an application level firewall. Something like Zorp is a good start, but you probably need something with a bit more intelligence about the applications you are talking about though.
L7 Firewalls usually get a bad rap because they tend to be pretty fussy in setup, something you can't really avoind with this kind of stuff. Also, if I was in your shoes, I would learn to stop worrying and start loving tight-ass SLA's...
People who think they know everything are a great annoyance to those of us who do.
Seriously. If they want root, tell them to fuck off.
Well... don't get EDS to work on it!
Did he inhale?
Seriously.
I built a network for vendor lab equipment that was it's own netowrk, connected to the main network via a file server with 2 nics and NO routing. Therefore, file could be transfered from the lab network to the file server, and then from there to any of the main networks, of course after they had been scanned for viruses and such, since anti virus software usually would not run on their equipment. My rules were firm and backed by higher-ups - if you couldnt get your equipment to work in the enivroment given, or with only minimal flexibility from me (for example i'd write scripts to move their files automatically to the main, backed up network or something simple like that) - then we will find another vendor to provide us a solution that fits. Period. I never had any problem with a vendor once they heard the terms. They won't make money off us if they can't make the system work in our environment.
Don't Tread on Me
Our vendors are notorious for demanding superuser access to the boxes that support their applications.
There are a few things you can do in this case:
1. Get a different vendor (if the application truly does require su for vendor support).
2. Reqest different support staff from the vendor (do this if the app doesn't require su but the support staff is too lazy).
3. Learn how to support the app in-house
I am very serious on these points. You do not want to give root access to people that should not need it. If they say they need it, they have an awful application.
The place I work has a vendor area fenced off from the rest of the server room, and the vendors only have access to what they need. If they need more privileges, the IT guys watch them like a hawk until they're done.
if the app NEEDS to run, you put it on a DMZ and let the world have at it. If they want internal access....make an effort to secure it and when they say they can't do that, get it in writing -- email will do if you've already got that -- and make sure you've secured everything you can. Not much else you CAN do, unless you're the boss.
FreeBSD for the impatient.
Tell them you need root on one of their machines before you'll renew the service contract. Seriously, no competant organization should be giving 3rd parties root.
Only give temporary access to allow agreed changes to take place. Superuser isn't required to diagnose problems, only to fix them. This also gives some stability as 3rd parties don't enter and alter things as they please.
The disadvantage is that there is an operational overhead -- but then there should be because if its a pain to change things then less gets changed.
Frequently vendors can't restrict their applications to run on a limited set of ports. Most of the time they stare blankly when we want their application to run as something less than superuser.
Simple answer: Don't let them. This is standard operating procedure in the financial services industry. Do you think that a bank would EVER let a non-employee or non-contractor access any bank system whatsoever? Especially remotely? If these companies want to do business with you, they WILL play by your rules or you'll pick one of their competitors products. In my experience, the only companies that required remote access to their systems were ones that A. Didn't have a fully working product, and B. Had to have the developer log in constantly to patch binaries with the latest bug fixes just to get a semi-working product. These are not the type of companies you want to do business with in the first place.
Speaking as a sysadmin for a smallish financial services company that processes around 1.2 million transactions a month, I would NEVER allow any vendor remote access to our network. It just wouldn't happen. Even if I did want to give them access, there are rules and regulations that strictly forbid my giving them access. If they give you a hard time, make something up about a security audit or something.
Seriously, you are asking for trouble if you let them have access. Who's going to take the blame when one of their developers logs in and wipes out all of your company's data? Chances are they'll blame it on you and you'll be in trouble for their mistake.
"When the president does it, that means it's not illegal." - Richard M. Nixon
What can you suggest?
Find some better vendors?
If a vendor is unsympathetic to your concerns, it's up to you to find an alternative or work around them. As you explain, the second option is not always possible when they require access to a number of services at a fundamental level. The worst cases of this occur when you have one or two vendors to pick from for a given application. My suggestion is then to design the application yourself within your security parameters and functionality requirements -- as many people do not have that capability within their own ASP (otherwise they'd do it already) you might want to use something like Sourceforge and contract a team overseas to do it cheaply, supervising the project from here and optionally open-sourcing it after it's built. Then you've got something designed to your parameters without support or upgrade costs especially if the community digs what you've built.
Try not. Do or do not, there is no try.
-- Dr. Spock, stardate 2822-3.
Can you not just write a contract that holds them completely liable for all damages, losses, downtime, etc caused by that machine? Then give them the option of whether or not they need to be a superuser that badly.
Seriously ... . I'm a sysadmin for a medium sized accountancy company, and you wouldn't believe the number of vendors I've had to beat off. I end up talking them down to lower levels of access, all the while listening to the lusers talk down to me as if they knew more about running it on *my* network than the local sysadmin.
In the long run though, I've figured that these lusers are going to be more trouble than they're worth, and am talking the boss into replacing the NetApps and Junipers (routers) with Gentoo Linux boxen. So in short, don't protect, just reject!
I have a similar configuration between two in house networks I use the firebox X700 in routing mode it allows me to open only the ports needed to be routed between the networks and also allows me to monitor all traffic to keep an eye on the test network side.
Create a dedicated VPN. The misbehaving app server sees only this VPN. Everything it needs to access has a presence thereon, carefully firewalled to allow the relevant ports to open. Everything it doesn't need to access, is not even on the network.
I'm an IT guy who does applications managment for an Air Force network and this is a problem that we have run into as well.
;)*
Exchange Service accounts, NetIQ accounts running as Domain Admins etc.
This is generally not a good idea, but for your average IT admin superuser domain level rights were the only way to have those programs function properly for the longest time. However, one simply solution that I have just recently tested with our NetIQ application managment Server is to create the local service account on say...our SMS server not as a domain admin, but just as a local admin. If I do this I can avoid giving blanket access to the entire domain to that program, and I can control which servers have that particular service account on them.
Obviously its not a total fix, although luckily for us, server 2003 has eliminated a lot of the most common "null session" issues that were such a pain to hunt down and change manually.
The only other thing I could suggest is creating a test-bed network and installing one of the suspect application servers at a time and using a sniffer capture program to see which ports have what sort of protocol activity over them. Document the information, test several application servers at a time, shut down uneeded ports and then implement it on your network. *with approval of course
I do hope that I haven't completely misunderstood your problem.
application level firewal....bleh...cover all ur servers with a shiny tin foil...be sure to replace them regularly.
What about running a Linux VPS or UserModeLinux?
We force the vendors to enter via VPN. we use the VPN gateway to restrict each vendor account's access to only the IP addresses of the systems they need access to. Further, we occasionally use a packet sniffer to watch certain vendors.
We disable the account by default and require them to contact us and tell us what they are doing (change control) before letting them in.
Works for us.
I work for an ISP, and we do not give anyone outside the system/network administration team root access to anything if at all possible. We have given network vendors (Redback and Ascend/Lucent) remote access to a device a couple of times (we changed the passwords on the particular device before and after), and we have one web application that we had to give the vendor root access for setup (but that was on a dedicated server).
Our antispam server is a dedicated server, but it still was screwed up once by the vendor. We were having a problem (only half our mail was actually being processed by the filters), and management directed that I go ahead and give the vendor support person access to the server (as the user the software runs under). He logged in to look at it, and within minutes the system was broke and mail was queueing up because the antispam software shut down and wouldn't restart. He had seen something unrelated to our problem that he thought didn't look right and decided to "fix" it for us. As soon as I got him to change it back, I told him to log out and removed his access (and they won't ever get it again).
After that, the only other time we considered allowing a vendor access was on a problem case that was over a year old that we had to have fixed ASAP. However, in that case, we were NOT going to allow remote access; the vendor was going to have to send somebody to us and we'd sit him down in front of the console (which would be logging) with us looking over his shoulder.
The only people that know your systems and network are your people (and you'll make enough mistakes). Vendors should not have access to change things at any point; at most they should get a "view" type account (they can look all they want but they can't touch). If something needs changing, they need to tell you and you can make the change (after evaluating it and having a back-out plan). For complex systems, you never end up installing software exactly the way the vendor specified; they are not going to know or understand changes you made for your local configuration (and how such changes affect other services and systems). Even a well-meaning "fix" can cause serious problems, and since it'll be your job to fix them, you better know what was done.
...you wouldn't believe the number of vendors I've had to beat off.
;)
Well, I hope you at least wash your hands after each one
Facts do not cease to exist because they are ignored. - Aldous Huxley
Call IBM Strategic Outsourcing - we do that everyday for some of the largest global corporations on the planet.
Too lazy to create a sig...
Don't think of this as a small scale issue. Design "security zones" providing the requirements for the enterprise, the vendor and external connectivity. An ASP environment is a long way toward that goal. If the company you work for is serious about security, they will be willing to foot the bill for a completely independent network security zone for the vendor application(s). Contact the most appropriate vendor for your firewall (Checkpoint comes to mind) and discuss the options with the vendor. Then, make sure you log everything important in the firewall. Another completely independent network that makes sense to have available is a management network. Separate from the "data network" and definately in a different security zone. It should be a separate "security zone" from your general data network. It might also make sense to connect to it only by VPN. This network would contain all of the SNMP and console activity. This can be built on a general network using GRE tunnels (IPSec if you really desire security) and/or extended with L2TP. At the very least, you would have sufficient logging in the VPN and system to track vendors activity. I would also consider building a completely separate non-routed network for backup. Offload all backups to a network that doesn't touch any other environment. Also, if a vendor wrote a program that can only run as su -, find another vendor. Their application was written unrealistically. So, in the end state, you'll be running multiple networks. Keep it simple, separated, secure and logged.
The VPN solution is great for the Vendor Company portion of the link, but I would also suggest a "vendor firewall" if you need to extend that service down to machines in the internal network (eg, stores in a grocery chain). The Vendor will *only* see their appserver from the outside, and their appserver will *only* see the vendor's devices internally. Good luck securing the actual end devices; unless you have each device behind a layer 2/3 firewall of some sort. Assume that the vendor can do anything with their devices, and try to limit/monitor their actions as much as you possibly can. The whole "root access" is a red herring, unless it's root access on one of *your* servers that they are requiring (which I would vigorously fight against); nmap, rdesktop, etc. all run without "root" privileges.
The wheel is turning, but the hamster is dead.
You may want to have a look at a Sybase product called Replication Server, which permits you to distribute your data in near real time.
Even though it is not a simple product, setting up a warm standby is fairly straight forward and relatively simple. By setting up appropriate firewall rules you can ensure that the connection is in one direction only. As an added bonus you are better set up in case of a desaster.
The RDBMS in question need not to be Sybase ASE. It works fine with just about any major RDBMS. In fact: There are Sybase customers that use Rep Server in order to replicate from Oracle to Oracle, since Oracle "Replication" just plain sucks!
ich bin der musikant
mit taschenrechner in der hand
kraftwerk
I'm your vendor. I need root for about a half an hour to fix some things.
Also, I need your ATM card and pin number, I need to fix that as well.
Don't forget to give me your house keys and please have your wife put on her negligee.
I'm glad we have a meaningful working relationship.
From the vender side.
First: It's amazing how many people haven't got a CLUE. "Don't give it to them or get another vender" isn't an option. When you're running a 24x7x365 system and expect sub 1 hour response time from your venders (our customers do) you're going to have to give some.
All of the boxes we hook up have phone lines into them. If security is an issue on dial-in access, we set it up as follows:
Modem is enabled dial-out only, no dial in access.
If the box dials home, we contact the administrator, identify who we are and which box has troubles and have them enable access to the server/hardware and reset the root password temporarily on the box. This occurs only if it's something we haven't already configured to work with sudo.
We then keep logs of the entire session, then email it to the customer when we're done.
When we're done, we have them reset the root password and disable dial-in on the modem again.
If we require access to a network resource from the machine, we're onsite with the customer shoulder surfing.
We do this at secure military sites, financial institutions, etc. and have yet to have an issue in 20 years.
I disagree with what you say, but I'll defend your right to say it to the death - Voltaire
We allow them onto only a pair of servers, all vendors come in this way. Then require them to use Symark's PowerBroker or PassGo's UNIX Privledge Manager (basically the same product) to handle authentication and connection to the system their software is on. Both products perform keystroke logging and other helpful features.
You can let them have their access, and watch their session in real time.
1) write in to the SLA policy-- a SU agreement with liablity
2) use (as mentioned) a L7 switch (Telena has one, too)
3) give temporary access, and make sure you check for root kits everytime with a script
4) tell your management just how expensive it is to have so many vendor's spoons in the soup and how potentially destabilizing it is to do this
5) use smart token card access coupled to your own CA; Tie the proximity of the card via RFID to a pacemaker attached directly to the aorta. If they lose it, they die. Simple.
6) partition roots across servers. Get an SNMP trap when they logon to keep track of them. Set a script against cron to send an additional alarm when they're on for more than a few minutes or upload more than a few megabytes through specific ports (indicating massive changes rather than remote control screen delta)
7) ask for one of their children for hostage use
---- Teach Peace. It's Cheaper Than War.
With the internet and remote admin tools coming / have come to full potential, vendors keep coming to ask for this more and more.
:: Speed on Highway and Number and Color of Cars traveling in pack/line. Some great laughs later over some good wine.
I have been a software vendor for many years. With software that is correctly build and tested, the need have access to a forgien box is about 0.
That is not say that there were not times I wished I had access. Mainly language translation, English-French with neither tech being able to speak the other and a translator that did not understand techincal terms. Think Baud Rate and Partity Bit
Even when I went on site, I generally never touched a machine. I always had the local operator or manager do the actual typing. This way, they learned the what and how along with me, and how to fix it.
But back to today's world, IF and I MEAN IF the higher levels of company force/black mail you into a corner , setup a connection PC. Vendor connects to there, and that is terminal of them in your system. Log every thing. Turn off the PC when you can not be watching what happening.
The FreeBSD operating system provides the jail(2) system call and the jail(1) command for imprisoning a process and its future decendants. The jail facility is based on the chroot(2) implementation, but prevents well-documented means to escape chroot confinement, offering partitioning of the file system, process, and networking namespaces. The facility removes all super-user privileges that would affect objects not entirely inside the jail.
For more information read:
A VPN can keep their connection to your box encrypted, but it doesn't control what they do TO your box through that connection -- and how it might affect the rest of your network.
Also, because VPN traffic is encrypted, you can't use a packet sniffer or intrusion detection system to watch it. Maybe you can position your sniffer or IDS to watch it after it emerges from the VPN link, but it does limit your network topology options.
Then you make your policy strictly exclusionary. And when they say "BUT I NEED THIS!", you say, "Ok, fill out a form 23" or whatever the form is. They'll learn quickly that they aren't going to get many of them approved and they'll start putting them in only when they really need them.
I do security
When the vender begins to sell you their app. Bring up these Issues about not allowing them to have super user access. It is just that simple. When then make them sign a contract that they will not use. And if they begin demmanding then the contract is null and void. Just that simple. Just be smart when buying products. Especially Canned apps.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
And have them revoke the suppliers rights to sell products for Windows, since they haven't got a clue.
I am working in a big company, and if the vendor do not spend the little money to fix their broken app, we will pick another. If they haven't got a clue about security on a network, they are are too dumb to deliver software for us.
Say you have an ERP application from one vendor, a CRM application from another, and middleware from another. On each server at your facility, run a virtual machine and run that application in the virtual machine. Connect the virtual machines through SSH tunnels that run from virtual machine on one physical machine to virtual machine on another physical machine.
This will increase processing time, necessary hardware, administration, etc., in other words, COST, but this can be considered a necessary security cost.
In the meantime, I would get a software development department, organize a SourceForge project for each application you guys need, or join an existing such project, and implement in-house free software versions of the same things. You can rest assured that other businesses have similar needs, and these free software projects will eventually replace the expensive vendor controlled versions that you run now. This will decrease costs in the long run.
...since VMS has a finer grained privilege system which has no such thing as "superuser" (though heavily priv'd users are possible). In addition there are third party and builtin controls available that allow things like limiting which files/directories/devices even super-priv'd users may get to, may allow certain programs to have privs without the maintenance accounts having them, hiding some storage from programs, etc. etc. Also while a program can be permitted access to kernel space, the privs to do that are distinct from those allowing filesystem access or network access. You can (and should) ask why any extra privs are needed, and can track their usage. Modern VMS runs only on Alpha or IA64, good performing processors but not x86. On the other hand the fact that the underlying instruction sets used are not x86 means that everyman's buffer overflow exploits tend not to be engineered for the machine. That programs typically never run in fully priv'd environments (but are given privs they need for legitimate use only) limits too what mischief buffer/heap overflows and the like may do. /dev/kmem, and expect to spend a lot longer setting all that up than you would in VMS, which ships secure out of the box.
If you need to run more safely in x86, I would suggest having a look at SELinux (from NSA) which gives some better controls than usual. Pay special attention to protections of devices, especially some of the more dangerous ones like
There is big marketing battle just behind you!
"Evil thrives when good men do nothing"
You've decided to follow the mantra of the BOFH and keep your life simple.
It's a well-used policy. =)
Lost at C:>. Found at C.
As one of those vendors for governments, I have to constantly deal with some moron admins who refuse to give me any access to the machine, yet CONSTANTLY call for support.
I don't generally need superuser access on the machine, since generally the only thing that gets screwed up is the data in the RDBMS (you know, user error).
I had one propellerhead go around in my database deleting tables and columns he felt they didn't need. He told me on the phone "we don't use timestamps here". One of those slam your head on the desk conversations. These are civil servants with lifetime jobs, and maybe they knew all about VAX in 1970, but goddamn if they aren't dense.
They tend to think that RAID is a magical "never need to backup ever" solution. I just love it when they call me up after their second RAID-5 drive failed, and I ask them when they last did a backup - and they go "uhhhh we don't need to backup we gots RAID".
Then I explain how RAID has nothing to do with archival or backups, etc, etc.. And I pull out the backup I made last time they had a major upgrade and tell them they have to reenter every parking ticket for the last 8 months, and they threaten and bitch how it's my fault and I tell them I'm not their admin, and if they really want to go to their bosses and fess up how incompetent they are they can go ahead.
Frankly, I'd love for some more competent clients. Of all of them, I can think of one who has any clue what to do with a computer.
But then sometimes they call with a problem that requires fixing on the machine. I'm not going to sit on the phone talking them through shit, I'm not going to email them scripts or code, etc. More than once I've had to tell them that if they don't give me access it wont get fixed.
If it's a problem for you, give them superuser rights when they need it, when they're done doing maintainance, take it away.
I don't need no instructions to know how to rock!!!!
You do NOT let outside users have access to production systems as root (or as any other user). When they ask, you tell them to stop smoking crack and make them tell you how to fix it. When they ask again, instead of root, give them the middle finger, and them make them tell you how to fix it.
You need to learn to do things on your own. Calling up the vendor and handing them root to fix your problems merely makes you vendor dependant, and in my opinion any SysAdmin who does that should be fired. If I were a manager, why would I continue to employ SysAdmins who only call the vendor every time there is a problem? I would be thinking to myself that not only am I paying this bozo a huge sallary, but I'll be paying for the support contract forever as well, and not just for one system, but for however many app servers you've got. Why not just hire someone for minimum wage to dial the vendor when there's a problem than pay someone who is suposedly highly trained but can do nothing for themselves?
I'm not a manager, but I am the lead SysAdmin for an Internet services company. We have about 40 servers and about half as many networking devices (managed switches, firewalls, load balancers, etc). Whenever we get a new type of device we do end up calling the vendor quite a bit, but we always make sure they teach us the solution and we impliment it. Over six months or so as I learn all the ins and outs and bugs of the device we no longer need to contact the vendor. I have a team of three people, and if one of them gave a vendor the root or enable password on any of our devices I'd have them fired for network security breach, and if they weren't taking proper steps to learn a new device on their own I'd have them fired for incompetence and replaced. There are too many smart people out there to keep employed dumb ones, especially for the price tag SysAdmins deserve.
When your vendor is wearing this t-shirt it's a bad sign.
Of course, it could be worse.
Let me qualify explain that statement. I support a small web-based application. It'll win on Unix or Windows (under Apache & PHP) I don't care about that stuff. But what does piss me off is when we have to port the app to every DB MS under the sun. Our support costs go through the roof just because the customer wants it their app DB.
Well, try explaining to IT (not a smart bunch, after all they dropped out of CS (or never even tried)) that their DB du jour is not up to the task of our RDBMS. True 90% SQL compliance is a good start, but there it is impossible to create a full-featured app and still be completely agnostic.
Examples: MySQL is the worst. The last insert ID is a horrible concept that is not portible. Try finding it in Informix, it is impossible to find, but exists. Thent here is the MySQL timestamp 'feature' - the first timestanp field (and only the first) when UPDATEd is updated to NOW() regardless of whether or not you include it in the updated feilds.
The only reason why they want the DB changed is because they have lazy IT departments that want to do nothing. Their IT staff does not understand the complexities of SQL DBs to them it is just an DB, and "thay're all compabible through SQL" yeah, right. "What about ODBC?" ODBC is fine for basic record operations, but the moment you try to do anyhing advanced, you're out of luck.
I used MySQL as an example, but I've worked with several other Databases, and there is no Holy Grail of DB abstraction. There's much more to databases than inserting updating, deleting and selecting rows, depsite what IT thinks.
Slashdot's rate-of-post filter: Preventing you from posting too many great ideas at once.
Depending on the nature of the application, you can give the vendor a UML machine and its root access. This lets them do what they want/need to, without compromising your environment.
Can You Say Linux? I Knew That You Could.
When saying NO isn't a valid answer...
1) Limit their access to their boxes using a good VPN solution. Limit their access to your secure network components to just the specific ports needed, using both source and destination IPs with a stateful firewall and good access lists.
2) Reduce their need for access. Build the servers boxes on VMware to allow you to grow the hardware without costly reinstalls by them. Costly is defined here as your $$$ and your labor. You keep a backup of the machine before they mess it up, and it also lets then do trial upgrades using real data on a clone of the server. Restore it easily.
3) Monitor it to the max. Catch their problems before they do. Save the day. Enable SNMP within the LAN, set up thresholds and alerts. Then correctly install a real IDS solution on your network. Be Big Brother on your Network
Be Proactive, by building IPsec VPNs, VLAN's with Firewalls, Monitoring & IDS for inspection, And using VMware where possible can provide many of the solutions.
Remember it is your network, and it may be your job when it all hits the fan, and
It may also be your promotion when you save the day.
The solution lyes within... OpenBSD. Very clean, lean, simple and secure, yet no mess to your existing infrastructure - worth checking out - www.openbsd.org.
In other words, VLAN + PF their asses or Transparent Bridge + PF them... So many solutions to this, you just need to look at the various options provided, within.
1. Smaller niche vendors usually don't have the hardware or manpower to simulate your system. You went with them because they were half the price of dealing with IBM, remember? :)
2. Often, those applications that require Superuser for the installation are third party applications. Install it yourself. You may have more trouble on the outset, but you'll know what's going on with your machine.
3. Often, vendors have their programmers as installers. The bad news is that you see the problems you do with the installations. The good news is that they'll know exactly why they need root - and they'll tell you what they need done. This might need to be a tag team installation, of course.
4. Remember, you can always invite them up there if you want to pay them for their time. Remote access is requested because it's cheaper. Alternatively, put in the contract that you want installation instructions. It will take more time, but you're always welcome to pay more.
Many of these vendor problems are reduced to cost-cutting measures. If you want to pay more, then vendors will be willing to oblige.
We make an "appliance" product which is a package of applications in a locked down box. The root account is disabled and the root password is scrambled. After we ship them, NO ONE (not even us) can log in as root. There are only two user accounts, one for config and the other for upgrades. Neither one has any ability to open a "normal" shell like bash.
No one has ever complained. And we do service them 24/7/sub-1 hour.
But here's something farther out: how about contracting with someone (either the vendor or some other provider) to run the apps themselves on your behalf? How does everyone feel about that? Is there an important class of applications for which this is an acceptable option?
For somebody who is simply selling you software to require that level of access to a server - ANY server - is illogical and a probable security hole. Might I suggest using the wheel?
This sig no verb.
I'm admin for a optical shop and when they asked for root, thats just what I told him.. I also said "That IF I EVER allow you root, it will be through remote secure desktop and I WILL be watching your every move! AND, if you do get stupid with that mouse I will pull the plug and call your boss, is that clear??"
They quickly agreed, since I got layered security in place (router, software, NAT, etc..), they'll have the devil's own time just making the attempt since it'll take me an half hour just to configure the network to allow for tunneled access.
Pure and simple. Oh, and forge a contract in stone saying that if you do allow access, you wash your hands of any fuckup that they perform, since it's their own dammed fault in the first place. And ah, if they do fuck it up, both contracts are broken, their boss gets a nice call from your attorneys, PLUS your own boss, etc, etc.... IE, Things MIGHT get a little ugly for the vendor in question.
IF they want your business, they will have to cooperate with you in order to maintain a good business relationship.
After all, it's a buyer's market right now.
First rule of holes; When in one, stop digging.
Just curious, having never run into the problem.
Now I'm the grandest Tiger in the Jungle!
Back when I started, working on mainframes, the standard, everywhere, was that you had a development environment, and when you were done making your changes, you passed it over to the admins, who moved it into a test environment that replicated the production environment (often with less data, but otherwise identical), ran regression tests, and *then* put it into production.
Oh, but that's mainframes, where they run whole companies, not workstations/servers that, uh, oops... Well, maybe you could lower yourself to provide a real test environment for the vendors, and y'all from the company take responsibility for final testing and promotion to production?
mark, m'frame, *and* pc, *and*
workstation/server experience
I'm senior deployment engineer at one of those vendors (well, not one of your vendors, but similar kind of deal). One of the things I've put the most time into is building a (OpenVPN-based) administration infrastructure that'll work damn near anywhere. (If we need to, we can even tunnel over HTTP -- hasn't come to that yet, though).
Our integration components are likewise designed to be flexible and nonintrusive -- as much code as possible on our server, as little as possible on the system we're integrating with.
I'd like to think that when we start deploying to larger environments, these efforts will get noticed.
I realize that my tiraid may not have been completely explained. We have no problem installing our required DBMS on whatever server or OS (customer's or provided by us), but so many times they will not install our DBMS on their production servers, so we end up with our own app server.
This is not my fault, this is theirs. They want to take no risks with disrupting other things, so the price of that is another server. They bring it on themselves.
My angst is directoed towards the ones who will not install our seotware on their servers, won't provide an app server, and demand that I MUST port it to their DB. It works for selling to large companies, but when you're talking hundreds or thousands of new isntallations a year, I just cannot port to every DBMS under the sun.
So your spp server problem is one that you can point the finger back at yourself for. You seem to know more than the company who wrote the software from the ground up. After all, we know nothing about databases, and we picked the wrong one from the start....
Slashdot's rate-of-post filter: Preventing you from posting too many great ideas at once.
but I think DMZs are overrated.
© 2004 The SCO Group, Inc. All Rights Reserved.
You're thinking of HIPAA.
Sox stands for Sarbanes-Oxley Act. It applies to every publicly-traded US company.
Yeah, right.
Your vendors can already run arbitrary code on your systems. You must trust them.
The standard practice of firewalls to restrict access, and lower priviledge accounts are all good (and important). But the proper protection should have been negotiated into the service contract in terms of access the vendor requires and liability. Durring that negotiation process the technical authority should have considered the security concerns (and added costs). If the technical authority was inept the best you can do is minimize the risks now, and use this example to raise security concerns for the next contracts.
We have one consultant/developer/sysadmin (yes, he's a special guy) that has access to one system. He has a password to go though our firewall, and one to access the system.
All the other guys have to come in person, and then our admin logs in, and stays while the guy does his job. Usually we let these guys come in to install a test machine, which we learn how to use, install and configure, and then we install the real box.
If its a unix apps there probably is a Solaris port. If its a Linux app you can run it under Solaris10 x86 in a seperate zone.
Similiar to FreeBSD Jails you can totally isolate the process but I think Solaris Zones partition a whole instance of the OS which gives it more functionality than BSD Jails. For example you can access ports lower than 1024 as root but in a seperate Solaris partition.
I am a little ignorant about this since I have just been reading about it. Anyone here know more info?
http://saveie6.com/
My company makes electronic medical records software. A few months ago it came down the pipe that we were legally required to make a specific modification to the billing codes as of midnight on the last day of the month. The patch only got final approval by our QA department almost immediately before it had to be out in the field.
Without remote access to our fielded servers, we simply couldn't have pushed it out. We don't have enough people on the deployment team (support isn't trusted with administrative access on the fielded systems, though we give them access to monitoring tools) to log into all our fielded systems and run the update manually, nor would it be as consistant as remotely applying an automated patch that's been approved by our QA department.
Mind you, when I say "our fielded servers", I mean it. We own those systems (though our customers paid for them), we have sole administrative access to them, and they run no other software. Perhaps your situation's different.
What if you set up your Linux system with User Mode Linux, or your FreeBSD system with FreeBSD jails, like modern hosting companies provide. This will provide your external customer/vendor with root access, but within a locked in virtual server. If your external vendor wants to maintain their database installation, fine: they have root on their own "machine", on their own IP, and there is very little risk to the larger system with real root.
we need a "bad, clueless, analogy" mod.
If a vender requires direct access to any of my equipment i require them to use an online meeting software. WebEX, Cisco MeetingPlace, Lotus Same time, etc. They don't have to know passwords, and i can watch every thing they do.
I'm a cucumber
It is able to do most of the things you need. It all depends on how complex solution you want.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
If these are critical applications, you should have a development/staging/production environment.
Let the vendor have access to a development or staging server that looks close to production environment. Put this server into a DMZ. Allow access only from pre-designated hosts. Let the vendor "fix" the applications on this box. Port the changes to production yourself.
Consider using a VMWare like solution for several of these virtual servers on a single hardware platform to keep hardware costs in control.
This wont fix all your problems, but will shunt off routine access requests by vendors off to a non critical replica.
(As a vendor, I can attest that this also causes the customer support staff to get trained faster :) )
The software I write requires admin privledges in order to do its job.
For example, it is not possible to get some system information from Microsoft SQL Server without using an admin account in SQL Server.
Our customers want that data and will not use our product without it.
May not apply to *nix but for Windows servers, I answer requests like this with Remote Control on a Terminal session. If a vendor needs to get "Administator" level access for some reason, I push them through Terminal Services then remote control their session so I or someone else can watch what they do. If the vendor is smart (haven't found one yet) they are aware that someone can watch what they are doing, but they aren't so I usually get to watch them do all kinds of boneheaded things while they take a guess and check approach to fixing a problem. The downside for me is I have to take notes so when the vendor finally disconnects I can undo anythign they did that was outside the scope of their repair.
Which makes me wonder why I can't record an RDP session somehow. This would serve two purposes: a) it would let me replay what a vendor had done later when it is more convenient and b) it would give me some proof if I later have to go blast a vendor for doing something absurdly stupid.
The same question for VNC or really any kind of remote management tool that is likely to be used on the Windows platform. Can any of them be logged and/or replayed?
- JoeShmoe
.
-- I wonder which will go down in history as the bigger failure: the War on Drugs or the War on Filesharing
There are two reasons for root access:
1) Full access to all files
2) Process control
No third party should have full access to all files - there's too much scope for disgruntled employees to wreak havoc. Force them to specify which files/directories they need access to then create a user or group that has rights to those areas. If they can't document the areas they need access to, find a supplier who can.
Keep a change log of their requests for access to files and ensure that no management types are allowed to screw the process up. If the specify areas like rc.x, hit them with a baseball bat until they're a lot more specific.
The ability to manipulate processes is more difficult. I would prefer to have them call in to a sys. admin. when they need access to processes outside their own group. There's scope to fsck the entire system up otherwise.
IF they really need remote access there are two possiblities - VPN or dial-up.
The best dial-up system I've seen allows incoming calls to a router/server which then calls back only to the third parties registered number. Changing the dial-back number requires a written request which goes into the change log. All connections through the dial-back server are 100% logged.
VPN software is getting pretty sophisticated these days so it's worth reviewing the commercial alternatives. We're using the Cisco offering at the moment. Again all connections through the VPN are logged.
I develop and code a web-based database application, that is installed on a dedicated server at our client's sites. In most cases, we also install some kind of remote access software for maintainance. We access this software via Modem or ISDN with or without callback, VPN or special firewall configurations via internet. Some clients even expose the server to the internet.
Our big advantage is that we need only one TCP port (80 for http or 443 for https) with one protocol (HTTP), this is rarely a problem for any kind of firewall or network admin. For remote access, we can use different software with different requirements, depending on the wishes of the clients. SMTP, POP3 and IMAP4 can be handled very similar. Other server applications can be much harder, even such "simple" things like a Windows fileserver needs various TCP and UDP ports, and firewalling such applications can be a major PITA. Even FTP needs at least two ports and contains IP addresses inside the data packages, but either passive mode or a smart firewall can help firewalling FTP.
So, if you "just" need some software based on well-known internet standard protocols, a dedicated server behind a firewall should do the trick. If the vendor wants remote access, add a protected (i.e. callback + token) modem or ISDN line. If you need propritary software, maybe even in a non-IP environment, the things become harder. You may even need to build your own firewall or proxy, perhaps based on Linux or *BSD.
Tux2000
Denken hilft.
going forward, if a vendor won't accept cost and liability to undo the damage their incompetance
causes, get a vendor that will...If they know
their stuff, all that will happen is the contract
for their product/service installation will have
two additional clauses.
sudo ?
Of course, if you are running some crappy Micky Mouse OS like MS Windows then you won't have anything this useful, in which case you have to dump the OS first.
This problem simply would not exist if you ran a decent OS.
1) First off, it amazes me how many customers there are who don't have security. For example, If I just have a user account on your system on your network - I still have access to your whole network unless you specifically firewall me off.
2) I prefer be firewalled off, so I know their stuff won't mess with ours and our stuf won't mess with theirs - but the truth is most customers don't want to go thru that effort or cost. Also, unfortunately, most have no clue what their doing other than following a checklist. This becomes painfully obvious when somthing doesn't work and you need their help to diagnose and fix it.
3) The customers who have their own specialized VPN systems that require us to connect only thru that - may think they're more secure, but their not because it means that we half to use plain passwords for access from their provided box rather than out own encrypted 1024 bit key tunnel originating from an internal server.
4) At least for me, the boxes I work with have only our stuff on it. That means if things go to hell on that system it is me and my company that suffer the responsibility. So in terms of their network and other critical systems, having admin really changes very little - other than perhaps permiscous mode, and listening on used ports or ones lower than 1024.
5) It amazes me how many customers don't understand scp and ssh, or even their own VPN's.
6) One time we had a system that we didn't have access to, and it got a virus. I would have loved to take responsibility for it, but I couldn't because I can't do much to prevent that if I don't even have Admin access to the box. And also, it came from one of their internal servers - well what do you want me to do about that? Most of the time, I find that we need more protection from a customers internal network - then they need from us.
Here are some ideas for you. Though it will require a change in thinking.
1. Abandon the foolish idea of buying or leasing your software off other companies. The personnel of other companies, as has been pointed out, are not necessarily as technically competent and trustworthy as your own staff. The management of your company have control over who they employ but they do not have control over the recruitment policies of of third party vendors.
2. Use Free Software. If that is not up to scratch for your needs then divert the large sums of money your are paying your vendors into the development/enhancement of free software alternatives for your requirements. Configure and manage your software inhouse.
3. Employ talented IT people and software developers to achieve point 2. Don't outsource your software, instead insource it.
4. Contribute back to the community by GPLing your software.
Matthew P. O'Reilly, C.I.S.S.P.
806 Claremont Road 704-906-3422
Charlotte, NC 28214 matt@moreilly.com
Certified Information Systems Security Professional (C.I.S.S.P.)
Areas of Specialization: Access Control Systems & Methodology and Cryptography
of all the answers, here, i'm surprised that none of them mention these two projects:
http://sf.net/projects/selinux
http://sf.net/projects/xen
the first project allows you to grant root access to lusers, thereby convincing the program that it's got root access, but the SELinux security kicks in as well, which is far more flexible than the 20+-year-old unix security model, and most importantly SELinux doesn't give a rats arse about what a superuser is.
the second project, xen, is like vmware only faster and is a bit like where vmware installs the screen/keyboard/mouse driver and you _have_ to use those drivers. xen-linux compiles as a completely new architecture (named, duh, xen) and it comes with its own virtual device drivers.
by compartmentalising an SE/Linux XEN machine you can restrict the pants off of the vendor's software and can blow it away with ease if they start dicking about.
Solaris 10 has a very neat feature that lets you configure zones on a system. Each zone can run a specific service or applications and communicate via the network using its own designated IP address. In side the zone there is no way to break out into the rest of the system. Even with root access in that zone.
You would still want to take additional precautions if you let outsiders on your systems even in a zone. Monitor what is done and record all changes.
One approach is to give them what they want and fire them when they misbehave. I know it is tough to catch but it is just as time consuming to establish a complex set of rules that only inhibit the honest and never seem to stop the dishonest. Integrity can never be software based. Be careful who you hire.
Stay out of our way! I have no sympathy for IT personnel that neglect big picture business requirements. You people act as barriers to our success.
"This is standard operating procedure in the financial services industry."
Sorry, but *bzzz* you failed your test. The only thing requred is that the vendor submit financials, and that management approves the request.
I have a vendor that has access to 6 machines, 3 of which are servers. I also have a second vendor who has access to about 6-8 servers in one area and 2 others in a different area. Both vendors provide support to critical customer transactions. One is for check processing and the other is for other types of transactions.
In case you were wondering we are a 1.5 billion dollar bank. We also just passed (with flying colors) our OCC audit.
they're the SAME FUCKING COMPANY that REBRANDED WHEN THE GOING GOT TOUGH. like any fucking consultancy group. AND WE PAID FOR THE NEW LOGO through the fricking neck...
I believe our (Fortune 500 company) documented policy for dealing with vendors who demand superuser access is to laugh loudly for several seconds and then hang up.
Really, if they can't provide their software tailored to match our requirements -- including which hardware it runs on (IBM Opteron+SuSE for just about everything new), what ports it runs on, and what UIDs it uses -- there are plenty of other companies chomping at the bit to do so.
If they really need root for their own guys to do something they send someone out and we watch them like a hawk the entire time.
We pay for 5 minute outages with 5 months of executive reamings. Nobody touches our stuff unless it's us or we're very close by.
We have a DMZ between two Raptor firewalls. Applications that need access will either get a hole punched through the firewall for the particular port (IP limited) or if that's not sufficient / too complicated, we setup a VPN for the application. FWIW, we use Nortel Conntivity to do it but honestly I don't think there's much advantage to using a vendor vs an open source solution...
I am disrespectful to dirt! Can you see that I am serious?!
It seems that your biggest problem is that you don't have to mitigate the quirks of one vendor, you have to mitigate the quirks of many. Your workarounds may break your other workarounds and end up building a nightmare of cruft.
The solution I would have would be to create an internal 'vendor' that would slowly replace the services one at a time. Once you've successfully implemented and trained support staff for say 2 or 3 key applications, hire new business staff and junior support staff and make the unit a subsidiary, offering the same services to the market at a markup.
After initially offering those handful of services, keep rolling out apps until you are independent. You can use the revenue from the subsidiary to partially fund the effort. And since your primary goal is to replace shoddy untrustworthy services your products will be superior to those of your competition.
At this point the unit goes into maintainance mode and since other people are paying for support too you don't have to pay people to sit there and play solitaire when there isn't a problem.
Of course, it's never as easy as it sounds. My basic premise is that you'd accomplish a lot just getting all of these things under one roof so you only have to manage one set of headaches. And if you implement it yourself you'll have a lot of say in how these things function on the hosts and in the network. And since you're probably not the only company facing these problems, you could make a pretty penny or at least mitigate the costs by eventually scaling up your operation and casually offering your services to others.
anyway, just my 10 cents.
If you can afford to wait that long to have the vendor fix the problem, why not just use snail mail to send a copy of the file system(s) to the vendor and have them recreate the problem in-house?
And of course, leave all your boss's sensitive personal informatino intact in the image.
We are the 198 proof..
"We've painted ourselves into a corner and, since we can't think of a way out, we're asking Slashdot for free technical advice to further our commercial goals without actually paying any of you."
"Hey do you think they bought it?"
You have dial-in access to defence facilities and financial institutions? Why stop there?
To anyone reading the above post, if you value your job, do not suggest anything like this to your bosses "because that's how military does it".
Third-party vendors will eventually get to review your planning and I guarantee that hilarity will not ensue - not for you, anyway!
Jan
Be careful, or you will find your name on some project's "top issues" list, floating up to some VP to resolve. Even if you do a good job, your peers might be incompetent - or the vendor may have other legitimate reasons for needing this.
When your ploy starts affecting their timelines and payment schedules your reasoning won't matter much - and you most likely won't get to explain it to the irate VP who is seeing her pet project slipping.
I've been to this movie - and from the vendor's perspective there is nothing like an internal scapegoat to buy time!
Jan
I've had the same, but in large financial services companies and in govt.
:)
I love to find a moron on the inside though - especially if a project is starting to slip and you need a few extra weeks, there is nothing like an ignorant admin on the inside - especially if they want to put up a fight!
We can usually argue for whatever we asked for, and in the end, well, we get the cake and eat it too - the delay saves our skin and the admin in question gets an attitude re-adjustment from their senior mgmt. All goodness.
Warm regards, your vendor
I've worked this as a vendor and as a client. When I find vendors doing this to me, I escalate this very quickly.
:)
Put together a decent paper that details the potential risks that external root access carries. There is PLENTY of material on the internet to help you.
Make sure you QUANTIFY these. How many machines are put at risk? Which critical apps are running on these? Could the ERP or financials go down because of this? Great!
Put a price tag on whatever effort you need to expand internally to address these risks. Do you need more staff? More firewall equipment/software? Insurance??
In the end, as a techie, you CANNOT exert pressure on the vendors - so let other people who can do that for you.
Your excuse is quite simple - if next month the a key corporate app goes down for a day because of incompetent vendor YOU are going to be asked how YOU allowed this to happen. It is YOUR job to escalate risks to people who can accept them.
Oh, and if your mgmt makes the mistake of accepting those risks, well, your job just got a whole lot easier - didn't it?
Jan
A follow-up, which I forgot to mention. The system administrators must also obtain the a clearance level as high or higher then any/all data being stored or processed by their systems. As such, all the administrators are cleared to be able to view, used, and work with the data following the specified classification restrictions placed on that said data. It almost requires the system administrators to be completely knowledgable with the security measures and proceedures for any and all data being stored on their systems. Including knowing what can and can not be copied. Restrictions on where data can be seen, and proceedures on handling the transport of said data. We work very closely with our data control centers and security officers. You need to, otherwise you can and will very easily find yourself in federal prision (as well as be fired).
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
We've done this as well for a number of our clients. At the end of the day we get everyhing we need (i.e. root, dba, etc.) but under the (sometimes) watchful gaze of their admin.
This also sidesteps network security issues - we don't need secure connections to their network, etc. as long as the admin's PC is configured right.
On the whole, this is usually also the cheapest option - WebEx is cheap, and lets a number of our people join in as needed as well.
Jan
So my advice would be to go explain it to the lawyers. If it scares them (meaning if you communicate effectively why is scares you) they might have enough clout to kill the deal, causing others to try to convince the vendor to write the app in a way that is more secure for their customers.
Firewalls and Dual Homed Hosts...
Mo' money, mo' money, mo' money!
Yeah, right.
Some vendors CAN restrict their applications to run on a limited set of ports. Veritas-NetBackup, for instance has a single port firewall option.
[disclaimer - I work for Veritas].
--NerdMachine
Clamping down on IP/port only initially limits the vendor. Once they're on their box they can use that as a staging post.
;-).
You may not believe this (neither did I when I discovered it) but we actually had vendors using their test system to hop onto the production set where we had just put ACLs in. They got rather upset when we downed the box as soon as the IDS went off (we had calls from the vendor making statements like "unable to support, sadly unable to meet Service Level Agreements etc", you know the drill). I guess they got even more upset when legal started to highlight to their MD the relevant contract clauses and which criminal laws they had broken in the process.
Sometimes a clue can only be brought by brute force - they had something like 3 months warning we were going to tighten up..
Time I start playing with LaBrea tarpits - however much I like nmap I like the idea of nmap fly paper in the network better
Insert
sudo http://www.courtesan.com/sudo/ is your friend.
Ask them what commands they need to run and let them have sudo access to run thoes commands only. Make sure they can't edit those commands through. If they are shell/perl programs, write a simple C wrapper script and give them sudo access to that if you're really paranoid.
Even as a sysadmin for 9 years I didn't login as root. 99% of the stuff is done through sudo. Root is for console access to deal with hardware issues in single user mode and to fix NIS issues.
If you're using the sudo that came with your distro, just make sure that it is logging every time someone runs sudo. If not snag a new tarball from the URL above. It should compile right out of the, but if you're having issues, then check out the RUNSON file. It'll tell you what options to use.
good luck
We already have no problem with 'black box' vendors getting access to our environment. We have a firewalled LAN segment just for them.
We don't like it if their black box needs to be connected to a zillion other application servers - e.g. database clusters, shared backup servers, email servers, etc.
Also, we struggle with times when their application is on a box with other applications and demand root.
In particular, one of the applications giving me heartburn now lives on top of a middleware server. The vendor guys are used to developing on their 0wn3d boxes where they have the middleware admin password and the root password. When we ask them to work without those things, they get frustrated.
I think that a well-designed application would allow a vendor to troubleshoot and repair the app without root access.
BTW - your management needs to take a chill pill. I would be very reluctant to do business with people who had their point of view. So would my colleagues. (I work for a Fortune 100 company.)
FWIW - I was looking very seriously at an add-on to a COTS product that would have been a great value-add to our organization. We would have paid big dollars (US) for this add-on, but it was developed by a single guy who had written it for another customer and retained the rights to sell it.
Because I represent a BIG company that can't afford to risk having him get run over by a bus and totally losing support for that app, I told him that we wanted the source code. He was offended, and flatly refused.
He claimed that we might take the source code and give it away! Puhleeze. We sign contracts that protect the vendor and protect us. I asked him to put the source in escrow, and he would not do that either. We ended up taking a pass on that product. We both lost. As a company we are not out to shaft our vendors. We have enough trouble getting the apps in place and working in our environment to have time to mess around with clobbering our vendors.
I hope that your leadership can lighten up.
But Herr Heisenberg, how does the electron know when I'm looking?
Unfortunately the application works well. This means that we have to tolerate for the short term the lack of design present in the product.
The one application I'm dealing with presently is a J2EE app where the installation procedure requires publishing the EAR and then manually editing files in the expanded EAR. Ick.
The first several releases of this app were POJO, and their model was to manually edit files in the expanded JAR files, too.
We're working with them to make the product better by writing it in a way that fits the J2EE model, but in the interim, their functionality is substantially better than the competition so I have to live with this model.
FWIW, they are responsive - I think that these are growing pains related to learning how to develop J2EE apps and live in an enterprise environment, so the long-term outlook is good, but the short term outlook includes painful support.
But Herr Heisenberg, how does the electron know when I'm looking?
*smile*
Any posting in reference to an enterprise that starts "Dude" probably makes some assumptions that are not accurate.
But Herr Heisenberg, how does the electron know when I'm looking?
I mentioned your .sig to my wife. She wants to know how that attitude about women (and cofee) is working out for you.
But Herr Heisenberg, how does the electron know when I'm looking?
Clamping down on IP/port only initially limits the vendor. Once they're on their box they can use that as a staging post.
Which is why you limit it to protocols that are difficult to hack and lock down the enterprise security so the worst they can possibly do is root one box, and worst they can probably do is mess up their configuration and have to rebuild.
Of course the key is to overcome the barrier of incompetence. Make it so it takes more than incompetent fumbling to break security, and trust your partners, WHOM YOU GIVE MONEY TO REGULARLY, to not purposely break the security. Their primary interest is to protect that revenue stream, they'll do the right thing...
I am disrespectful to dirt! Can you see that I am serious?!
I work at some large banks as an external and frankly, I am shocked about the access given to offsite vendor support staff. The issue is that when the system goes down, you want them to be able to fix things, it is fastest and most 'convenient' that they can access your system, and this is what the bosses are told and thus the firewall is opened.
Layer everthing atop email messages, just like Outlook's calendaring function does.