Slashdot Mirror
Kinder, Gentler Security Scans?
klausner asks: "I'm working at a large company that is trying to be more thorough about things like network security scanning. When Security told Operations they were planning to do this, there were immediate screams of anguish, and insistence that scans could only be done in the maintenance window, only with prior notice, and with a bunch of other restrictions. Needless to say, this is less than ideal. Given the size of the network, it would take weeks to do a single scan set. However, it is reasonable to take steps to ensure that the scans do not interrupt business traffic, or cause undesirable side effects like crashing target systems. What sort of limits are the readers out there using to ensure safe scanning? Limiting the bandwidth to a fixed percentage? Limiting the number of simultaneous tests? What other kinds of things can I do to limit the scans effect on network performance?"
You goddamn IT monkeys better do the scans at 1am-3am as required and convenient for real workers if you don't want to be outsourced someplace that ends with "umbai" or "galore".
Identify nodes that are more likely to have security holes(ie phb's desktop), identify the nodes whose performance is most critical, etc.
That should give you a clue of who to scan and how often to scan them. Probably more intelligent than scanning your whole network all the time.
Security is a range, it isn't a switch. If maximum compute power is of upmost important to you - go ahead, turn off all your virus scanners, personal firewalls, etc. . However, if you need security - turn those services on, monitor their compliance, and take the overhead that it requires.
Scanning for security vulnerabilities at night won't do you any good if the PHB takes his laptop home w/ him, or joe user powers off his virus ridden PC every night before heading home. You must scan during the day (again, if that is important to your business).
I have mod points and I am not afraid to use them
99% of the stuff you'll be scanning for won't affect them. Sure, keep the DOS tests for after-hours and keep your probe timing to something reasonable (e.g. don't flood-ping across a dial-up) but the rest can be done any time with zero impact...
Every time there's a new worm the IT geeks for a certain department at a certain Big 10 school port scans the network and throws off computers that haven't installed the latest patch. Then they take weeks to come by and fix the machines. It wouldn't be bothersome if it weren't for the fact that the last time they did this they assured us that we were now set to automatically update patches. I guess unless you run update every hour instead of every couple days you can't win.
I perform this function on a network with around 2000 user workstations (don't scan those. I shudder at the thought, but it's not *my* job/area) and about 130+ windows servers. the aix and netware stuff I don't worry so much about, but the aix stuff still gets weekly scans. at the moment, we use ISS internet scanner (http://www.iss.net) which I think sucks big donkey schlong. my prefferred scanner is nessus, but I can't use linux on the production network. I weep, like a young child with a skinned knee... but that is beside the point =D I use scheduled batch files to run the scans (within the limitations of the reporting, i.e. 4 hosts at a time) late at night, so as to not impact the users. stuff always gets scanned before going on the network (servers that is, since I build most of them anyway) and sometimes, people even act on the reports of vulnerabilities that I give them! we are moving toward ca vulnerability manager, so it will keep track of all this crap for us, and so I don't have to run the scans. I think.... either way, I'd rather have nessus running on a host on the network, so I could use THAT to schedule scans, or even to do them myself, just to get something that reports accurately, and deeper than stupid ISS... "duh.. you have a vulnerability... I hear that that's bad... you should fix it..." stupid software!!!!!!!!!!!!!!!!!!
perl -e '`nmap 192.168.1.1/24 &` while 1' &
spoof your mac and run it in secret... then after you find all the massive holes (the REAL reason they dont want you to scan) you have some dirt on them. Then bargain for better pay and better work conditions, then whip out the scan results. Then sue for wrongful dismissal.
It's guerilla warfare!
Boredom's not a burden anyone should bear.
First run a slow portscan across your network, with clean connection tear-down (i.e. send QUIT to a SMTP server insted of just closing the connection) and look over your results. Operations shouldn't have too much of a problem with this if you do it right.
Second look at the least common ports. These will be the oddball services that an administrator tossed up to test, or an engineer was trying to sneak past security with, and are most likely to be overlooked when updates are released.
Third, look at the most common ports. If you have a lot of machines with port 80 open, you should invest some time into researching web vulnerabilities. Same for other protocols. Based on these results you can launch smaller scans within maitnence windows to check for say, open relays on all machines listening on port 25.
Building apon this process and fitting it to your situation would be a good course of action. This obviously isn't as indepth as a good auditing plan should be, but it will get you going in the right direction.
Also realize that yout operations team has a good point, regardless of how concerned about security you are. Don't do like I did and take a off the shelf application (Nessus or Cisco Security Scanner) and blast away at your network. I ended up taking down a dozen mission critical devices because the vendor of the hardware in question didn't account for portscans. The devices ended up hanging because they received a connection with no command in it.
symetrix. We are building a religion, a limited edition.
When Security told Operations they were planning to do this, there were immediate screams of anguish, and insistence that scans could only be done in the maintenance window, only with prior notice, and with a bunch of other restrictions.
Just make sure Operations let the crackers know about these restrictions as well, and you'll be fine.
To: Network Operations
In accordance with your policy on security related network traffic, please be advised that I will attempt to DDOS the web server located at IP XXX.XXX.XXX.XXX and compromise the database server located at IP XXX.XXX.XXX.XXX, starting shortly after the start of the maintainence window at 8:00 UTC. If all goes successfully, the database will be corrupted by 9:00 UTC and the DDOS will cease shortly thereafter. All due efforts will be taken to minimize effects on connectivity for other networks users, and network traffic for this sequrity breach will be limited to the two above mentioned IP addresses.
I appologize for any inconvieniece this may cause you, but it is nessasary to "ownerz" your system.
Thank You,
Jack Cracker
Vice Prezident of Black Hats P.S. I would appreciate it if you would facilitate my exploit by reverting to an unpatched version of IIS on the database server.
We've found that certain applications running on erm, VMS or something here at work - will allow only a certain number of connections to a service - and if they aren't closed down properly, will hang. This is perhaps the worst thing we've discovered after performing network scans.
;)
If your company want's you to do scheduled scans during maintenance windows, that is rather simple however. You can implement this with Nessus in command-line mode, called from crontab. Just be certain that when you are configuring your scan, that you do not perform any potential denial of service scans.
But to be honest, I've been blase' a few times and on a whim pointed my Nessus box at our internal exchange server and highly expensive monitoring cluster and scanned away - nothing horrible has come of it - apart from discovering about 10 remote root vulnerabilities on each. That is the main concern from these people I believe, that the security scans will highlight something they know they're slack in - regular patching.
If you run into any departments who point at a particular system and say "don't scan that - it's mission critical", get the highest manager responsible for that system and get him to personally sign off that he's unwilling to allow a scan. Then remind him of recent privacy laws that have come into force. If that mission critical server is holding customer data, and it gets cracked, he or the company may be liable for failing to perform due diligence with regards to securing their data. And you'll have their signoff on paper.
Man watching 6 MSCE's around a sun box, looks alot like the opening scene's of 2001:space odyssey...
The biggest threat is that many scanners have a habit of crashing services which the developers have never encountered. Sadly, for the open source fans out there, Nessus is particularly bad with their QA and crashes all sorts of stuff even when the DoS tests are turned off. Of the commercial applications Qualysguard (www.qualys.com) does a great job of playing softly softly with the network while still detecting more than anything else out there (at least according to the size of their database). Don't bother considering anything else, other commercial scanners are less capable than nessus or qualys. But..... if you're worried that a security scan is going to cause adverse effects then you've clearly got security issues with your network. If a system dies under the load of a scan, or if some scan script triggers a DoS on your code then it's a sign that your developers and admins aren't doing their job correctly. Look upon it as a challenge, you should be saying 'Bring it on!'. If you're not confident that an automated security scan won't cause trouble with your system then you should be having nightmares about what a real hacker could do to your network.
How much of the outcry wasn't due to the security testing, but to the fact that most software/hardware tends to be untested at the levels a proper network security tests exerts it?
(i.e. how much of it is due to the fact that the machine might be unavailable to regular users during the test, because a service might crash, and not the impact of testing itself?)
It's a pretty well known anecdote that there are some network vulnerability tests that will find "vulnerable" machines... Those vulnerable machines blue-screening, as a matter of fact.
On a related note, you have your vulnerability tests down path, but what's your strategy if you DO find vulnerable machines? Will you wait till the next maintenance window? Apply the fix immediately? Do you even know which vulnerabilities you're scanning for? Did you scan security focus for your os/application/hardware mix beforehand, to get some feel for what might go wrong?
If their system comes down from a simple scan, better you find out yourself than the hackers.
If the apps folks are so nervous about things, just remark that if their systems are well built, they will handle a scan with no problems.
This is America, damnit. Speak Spanish!
Don't do it. It's probably a bad idea.
For one, automated scans probably won't catch half of your problems. And two, anything that disruptive to your business probably costs money than it'll bring in. In any worthwhile business venture, the benefits must outweigh the cost. Look at your setup and try to identify your top risks. By prioritizing your risks based on your guess of the damage*frequency/costToPrevent of each, you should be able to achieve a greater level of security much faster and cheaper than with big, automated, disruptive network scans that check each system for tens of thousands of usually absurdly improbable known risks. You can still run such scans, but dealing with your priority risks first and keeping in mind "What are the chances this will make or save us money?"
Do it in reserved time for the first couple of runs. Have them involved in evaluating the impact of the scans, show them the results. After it is done a few times with no impact, theyll get bored^h^h^h^h^h comfortable with scans and you may be able to schedule them during the day.
They should be scheduled, so that if something does go wrong they can at least ask you to scan again and reproduce the problem, or eliminate your work from the list of suspects. If you do it without telling them and it causes problems that they spend hours or days figuring out, you can bet you will be confined to 3am forever thereafter.
I work as a contractor for big 5 letter chip company. I can tell you that security is only second to the fab, and that is because the fab makes money. Unless something crashing is going to cause you millions of dollars an hour, someone needs to decide what is more important, your network being slow because it is being scanned for unpatched systems, or having a nasty version of Sasser erase data, send out confidential information, and completely crash the whole network. And they are even pickier about fab security, because if something does get infected and go down, they are out big bucks
Is security in charge of making sure everything is patched also, or is operations in charge and they are trying to cover their ass by making you forewarn them of your scan?
Your production network should be segmented from the general network, and critical portions of the general network (say, helpdesk, hr, etc) should be on their own segments. This allows you to scan one entity at a time and if something does break, you have a defined area for your desktop support team to work in.
Regardless of if you must wait for a maintenance window for production equipment, who will get the blame if something breaks? Do the scan on the weekend, on test servers, whatever you can do the easiest first. You should have a standard build for servers, desktops, etc... and be able to test those systems and see the effects.
The release time between an exploit being found and being exploited is growing shorter all the time. What was the leadtime for sasser? Two, three weeks? The netops people here are shutting off the ports of systems that are not patched at the switch level already. The network comes to a crawl while they are doing the scans. And guess what? They do them during the day. Why? Because that is when people are at work! A maintenance window is useless if you cannot guarantee what percentage of your population you are going to hit. So if your window is 1am to 3am, you better be scanning a network full of Indian helpdesk agents.
--ngoy
Get your job description signed by your boss and verified by the upper-ups. You are responsible for:
(1) Insuring that your network is as resistant to attacks of any sort as possible.
(2) Identifying any attacks and investigating the cause thereof.
(3) Mitigating the effects of attacks while they are being done.
With this setup, you should have one more clause: That security specialists should be allowed to do whatever is necessary to fulfill the above three items, even using unconventional methods, provided that:
(1) They receive (written) permission from the person in charge of all security before implementing a new method.
(2) Their methods do not interefere with normal business unless required by (3) above (eg, shutting down mail access in the case of a mail storm.)
With those goals outlined, the security team should be able to use pretty much whatever methods they chose to do their job.
The radical sect of Islam would either see you dead or "reverted" to Islam.
Know your network!
Before starting, make sure your tools can be configured to avoid scans of sensitive equipment during work hours. You should know exactly where each server and router is on your network, and run scans against them during maintenance windows, when a crash will not impact the company and the admins are available to bring the systems back up.
For lesser important servers, scans should be run only once in a great while. For the vast majority of your IP space, where luser PCs lie, then security scans should be run during the time they will most likely be on, which is during normal work hours.
When you can categorize all of the IP space into levels of importance to corporate revenue, then you need to tune your tools to have as little impact as possible on important systems. This means turning off nasty parts of Nessus, and addressing those threats via other means (mandatory patch rollouts, system level reports). You should not be trying to make anything crash, because that is counter to good security practices. A DoS from the security group is just as effective as a DoS from some blackhats.
If the network is large enough, there should be a budget for multiple scanning machines. Since it can take 20 to 40 minutes to politely scan a single machine, you will need to have machines local to each segment of your network and scan in parallel. There are a number of commercial scanners which will consolodate the reports to a central server.
Automated scans against PCs should run during the day. Some automated scans need to run against infrastructure machines, but since those machines are on 24x7, the scans can be run at night. Manually scan important machines when the admins who can fix them are on hand to see and patch any problems found.
the AC
Hemos is like...sci-fi fans;he thinks technology is cool, but he hasn't bothered to understand the science it's based on
As far as I can see, it scans your network for the versions of software in use on each host to create a model, then performs simulated scans on that model.
I'm skeptical, but it might be worth looking into.
No, it's not Free (or even gratis).
--
I recall doing a simple scan with nmap and rather annoying the backup admin becuase it caused his amanda scripts to crap out. Unfortunatly the "cure" was to not run nmap scans.
I don't think he ever got the point.
Security is not and never will be a throw of a switch. It covers territories that include the physical, social, and electronic realms. It's constantly evolving and mutating, proactively in hope of avoiding attacks as well as in response to identified threats. Security is a constant battle against human nature, the better you understand people the better off you will be.
Ward
. Silence! Be thankful thy species is unpalatable! .
If the machines/infrastructure cannot handle even a security scan, how will it deal with an attack or worm?
I understand the desire to do the scan during "off" hours, but a good system should be ready for things at any time.
Wouldn't a good start be to ping sweep windows machines first with "Safe Scans" on, Then Take a look at the services running on the unix machines to give you a good idea of where your worst vulnerabilities are there. Then categorize in order of importance: example: financial, HRdata, ordering, ..... pc's
If you have segmented your pcs away from your production service env they can come last or be handled by the desktop group
This is what I do
Scan after hours/on the weekend
Consider using a passive vulnerability scanners (e.g. http://www.tenablesecurity.com/nevo.html )
Do a distributed scan
Use unaggressive settings
*******
One of the foremost security gurus of TCP/IP like Dan Kaminsky of Paketto Keiretsu/blackhat/defcon fame has some novel ways of performing network scans too. You might want to consider reading over his material at http://www.doxpara.com/
Step one: Make it abundantly clear to everybody involved, and get management/executive signoff, that you're looking to 'improve security.' You're not looking to expose incompetence, to find people to be fired, and so on. It's like bringing in outside auditors to go over the books every once in a while; it's not that your own beancounters are screw-ups, it's that you don't take chances with this kind of shit.
Step two: Compromise; arrange for a military style 'wargame.' No, not that kind of wargame. State that everything you do will be during a given window; say, one week, but actual things will be announced only after the fact.
That way, they know when to be watching for things to break, if something does go down, they know to call and find out if you are in process of doing something, but you still get to test the systems in a relatively 'in the wild' state.
Step three: Don't present 'flaws' or 'vulnerabilities.' Present 'improvements' or 'best practices.' Rather than say, for example, that half the network's PCs are unpatched, suggest a patch server to improve patch installation time, provide for centralized management, and so on.
Vintage computer games and RPG books available. Email me if you're interested.
It's a security scan, not a total attack. The scan is to look for holes and analyze them. That in itself doesn't take very much network traffic per machine. If that is going to clog the network, they need to re-organize it anyway.
Why are the ops guys so nervous? Could it be that a security scan would show how careless they've been with their systems?
What a disappointing bunch of replies! I thought /. readers would have a better understanding of how "real" computing gets done. There is no such thing as "after business hours." The business runs 24x7.
This isn't about workstations. I should have been more clear. This about money generating production machines. Operations doesn't care about workstations. That's an IT/Helpdesk issue. This is about thousands of servers and network devices. As I said, this is a LARGE company.
This isn't about what's in a job description, or about a security manager deciding to go ahead regardless. Trying to tell Ops "We are going to scan you, so get used to it" is so far from reality it belongs on a "Tripping The Rift" episode.
Doing limited scans to start is also not an option. NOTHING can be done outside a scheduled window for now. NOTHING can be done that isn't compliant with all the other existing restrictions, for now. Until we get the rules changed! Admittedly, most of the problem is FUD and Operations trying to do a CYA. But the rules is the rules. To get the rules changed, requires facts, and an alternate set of proposals. I was looking for suggestions for those alternate rules.