Slashdot Mirror
How Should One Respond to a Network Break In?
Jety asks: "I am the sole IT support for a medium sized residential real estate office. It has a network of one main server, 10 office workstations, and another 40 or so agent's personal computers. I discovered via logs that recently someone made about 50 remote login attempts to the server, guessing at passwords, but it would appear that they were not able to gain access. They did, however, leave an IP address in the logs. It turns out to be an Exchange server for another business in the same city.
What is an appropriate response to this sort of failed break-in attempt? How seriously should one react? How should it be presented to management, and should you encourage them to over or under react? Should the other business, whose server was used to launch the attack, be informed? Should you try to surveil them first to learn about who is doing their tech work? With what tone should they be approached and/or accused? What would a suitable response from that company entail?"
Document everything in writing, discuss the situation with your superiors, and seriously consider initiating some form of legal action. If you are the first to get litigious, you stand a better chance of having the situation resolved in your favor. Unfortunate, but true.
____
~ |rip/\/\aster /\/\onkey
Call 911 and let the Patriot Act take it from there... No one from that company will be trying to pwn you again.
You shouldn't start out accusatorily, because it's most likely that they're not the ones attempting the breakin. It's more likely that their box has been hijacked and is being used as a proxy to launch attacks against your computer for someone else.
After all, who uses an exchange server as their terminal to log in to other computers? If it was one of the desktops, then it would make sense that they were attacking.
Remember, there were no nuclear weapons before women were allowed to vote.
You try contacting abuse@ the other company.
If that fails, you call them up and ask for their tech-lead.
You already have your logfiles, and reasonably secured server.
What you can gain here is a partnership - or at least an exchange of favors every now and then - between your company and the remote one.
That said, if the other company isn't responsive, you firewall them to hell and get on with your daily work.
You'll want to give management a brief notice about what's happening before you do this, obviously.
After you've talked to abuse@, you tell management what happened.
Now is the time to see over your authentication schemes. Are your users logging in over SSH? With passwords instead of keys? (Hint: keys are nicer).
After this is said and done, you paypal me $90 for doing your job.
Cheers!
No damage was done to you, except the effort you put into investigating. They, on the other hand, will probably want to catch whoever's actively using their server to launch attacks.
Start off by blocking remote logins (ssh?) from anywhere except where you want to allow people to log in from. Second, I would send a polite, email to their tech contact, or if you can't find that, regular post mail to the company. Don't overreact. Their are a lot of ssh worms out there. I have one machine where I watch for these kinds of things. I see at least 3 or 4 worms hitting my box a day.
There is a good chance the whole business uses the one IP for everything, so it could be anyone at that business (or anyone accessing an unsecured wireless network they have setup, etc) that is attacking your network
I always celebrate. Oh wait, you mean as the victim? Hrm..
My guess is that it's that script trying to bruteforce random SSH servers, as mentioned on /. a couple weeks ago. My server here at work has been hit too, although the attacking machines were in Europe and Korea in my case. I emailed the owners of the IP blocks the attacks came from and have left it in their courts. My system is secure (I'm the only one who can login via SSH and I have a damn good password), so there was no harm done.
I think before you jump to any conclusions about it being malicious on the part of the other company, you should call the tech there and let him/her know what's happening. If it is indeed a script then there's no harm done and the other tech can take care of fixing their system. If it was actually a malicious attack, then you can try and figure out who is responsible.
With your boss' permission, of course. Set up an isolated area on the server that looks real, with real file names like TopSecret, and when the system next detects somebody from that other machine attempting to get in, let them. Then just make sure everything they are allowed to reach is chock-full of viruses.
Oh, and IWF means 'Idiot with firewall'.
What you should do is quite simple. You should call the other company, ask to speak with their techie/IT-department, and explain to them that their server has been used to launch an attack against you - and that their server is probably compromised.
You should also send them your logs. If they're clueless - well, try and help them out.
Yes, there _might_ be a bad guy at that company. The chances are pretty damn good that it's just another compromised host on the Internet.
This was an idiot question from the poster. The answer is pretty damn obvious to anyone that has run a firewall or server for more than a week or so.
If you send an 'abuse'-email you should formulate it along the lines of "Greetings. One of your customers seems to have a compromised host. Here are my logfiles. Please contact the customer and take appropriate actions".
I think you should call them and email them the relevant portions of your logs. It might be not even them due to spoofing, but most likely, it is unauthorized use of their machine. They need to know about a wayward employee.
1) Document Everything
2) Alert the owners/management of the company. Impress upon them how serious this is, and how it won't be tolerated. Most likely it's just one employee with a wild hair up his... and not a representation of their company's intent.
3) Give them a time frame to address it/correct their problem
4) If it happens again let them know you're considering legal action.
There's no excuse for this behavior. Would you tolerate someone skulking around your building looking for open doors and windows?
DON'T PANIC.
Frankly I'm a lot more afraid of a successful breakin that I don't discover than heaps of unsuccessful attempts that I do.
Essentially everyone who attempts to hit my ftp server with anonymous is trying to break in - the address is only known to a few people who have accounts and I can see from the logs that the other attempts are just scripted tries.
Similarly, I'm see several attempts every day to log into my machines via ssh (where an attempt may involve from a dozen to hundreds of tries to log in). Don't even get started on what I see in the http or smtp logs.
I work at a small company, too, and I could pull everyone off their jobs and still not have enough manpower to investigate each attempted breakin, locate and contact the appropriate parties, etc.
As mentioned elsewhere, most of these machines are compromised so you are really spending your time to provide unpaid antivirus support for the other party's machine. You have to pick your battles.
Depending on my workload and the probability of a positive result I'll contact someone as a courtesy. Generally my criteria is that I am able to make telephone contact with a person responsible for the machine relatively quickly.
~~~~~~~
"You are not remembered for doing what is expected of you." - Atul Chitnis
The advice that everyone else is giving is better than what I could give. But the important thing to note is that there wasn't actually a break in at all, so no crime has been committed. This is akin to someone coming around and giving your doorknob a good jiggling. Not exactly pleasant, but thankfully they didn't get in...
I'll turn into a supernova and burn up everything. Well I'll turn into a black little hole and you'll turn into string.
We recognized you as an idiot from the first comment. The second was simply superfluous.
Submit an interesting sounding post to slashdot with a link to their web site and watch their servers melt.
Of course, this assumes people try to RTFA.
I get this a bit on a server I run. I usually just forward a copy of the info from my logs to whatever technical contact I can find with a friendly note saying that someone from one of their addresses was trying something. That way, I have a nice record that I notified them.
If it keeps happening, I then usually block that address or range of addresses with my firewall. (I can do this since only a small number of users access the server, and I'll hear about it if they're having trouble accessing things.)
Keep in mind that you may never get a response... I'd say about only 10% of companies I notify even acknowledge my email.
If it was just some SSH login attempt, I'd send a friendly email and then ignore it unless it happens frequently. If it does, step up your notifications, e.g. email, then a phone call, then a formal letter on letterhead, then a letter from your lawyer, etc. That way, you've got a paper trail should things get nasty.
-- Fugacity: Confusing chemists since 1908
Doesn't everyone who leaves ssh open and unrestricted by IP for any length of time see people trying to brute-force it with password lists?
That said, there's no guarantee that it really is a malicious act on behalf of that other business - could be someone came through them to get to you "for a laugh", or the office junior or someone's 12-year-old messing about.
Oh - and document everything, and make sure that if asked how you knew exactly when something happened (such as when something happened) you have an answer (e.g. an ntp log or something).
He has trespassed and therefore must pay: track down the real source of the attack, sick law enforcement on him, shut off his power and water, destroy his hard drive with a hard head crash, and show him that hacking your system doesn't pay.
You could always just post the IP on Slashdot.
Some might consider that overkill though.
Invalid Checksum. Retrying.
Just because their NAT router has a port forwarded to an Exchange server doesn't mean that the Exchange server was necessarily the machine where the attack originated. It could have been that machine, or any other machine on the network.
"You must be new here" is what comes to mind.. I get hundreds of these per HOUR on most of my boxes. It could be anything: a curious worker, a hacker, a virus, a script gone bad.
First thing, check your important file checksums, run tripwire, or whatever. If you don't have a tripwire-like system set up, or a backup set you can compare against, you've got another problem, but let's assume somehow, you are sure your files were not compromised.
Once you're sure no damage was done, relax, the system did what it's supposed to and rejected the traffic. Do a quick audit to make sure everything is up to date, you're not running any insecure junk, no version numbers are revealed, IDS signatures are up do date, and so on.
It's likely just a virus or a hacked box.
My algorithm for dealing with this is:
if self.friend_of? other_business.admin
self.contact other_business.admin
else
document anomoly
possibly_firewall other_business.ips
get_back_to_work!
end
In other words, it's NOT YOUR PROBLEM if the other guy is hacked. In fact YOU could be blamed for it (yes, this shit happens, people are idiots). DO NOT portscan or telnet or attempt to learn anything about the other box (which you already did, oops).
Be sure to DOCUMENT everything. If you visit their web site, document it. If you call them, document who you talked to. Just document everything, even if you just file it away.
Whether or not you contact management is up to your business culture, position, etc. In my opinion, it's your job to deal with this stuff and if you "escalate" every little port scan, you're just making a lot of useless noise. However if the other business is a competitor of yours, or there's some business impact here, or they've been doing it for months, you should tell your management.
The attempt was to brute force VNC server running on a Windows 2000 server box. Because the attempt came from withing the same city, and because there aren't too many VNC worms out there (though there are some) I made the assumption that there was probably an actual person behind it. Also, I used a reverse DNS lookup to see where the IP address resolved, so I don't think it was NAT'd through a firewall.
--Scavenger-- http://www.playdecay.com Online gaming the old fashioned way.
a) They'd have to know the IP's of the allowed machines :-)
b) The ban would only last 3 minutes.
c) A 3 minute blockout is much better than an owned server
A few things which I could see wrong with the above:
a) If you login multiple times to your SSH server (say, 3 SSH sessions and one SCP) it will block you
b) As with above, it seems to apply to all SSH sessions, not just failed ones...
Fix it quickly - patch the hole. dont tell anyone, and hope to god your boss doesnt find out! ;)
Electronic Music Made Using Linux http://soundcloud.com/polyp
For the love of god and language there is no such word as surveil. Try SURVEY.
Thank you and good night.
SURELY NOT!!!!!
I created a script to dig through my logs and nmap every host that has been brute forcing my system. A few months ago, an attempt always meant a box with subseven, kuang, etc... and almost always had ms-rpc ports, proxy servers, or morpheus ports (as far as nmap knew). I feel like I get hit by every sub-sevened box on planet earth sometimes ... mostly Asian sites and what appears to be French public terminals.
I didn't worry when it just looked like some idiot that got bott-ed. But as of last night I started finding boxes that only have ssh and ssl ports running...those boxes seem like they might be due to intentional misuse by the owners.
Lock down your ssh server to only accept a public key from the systems you login from (laptop, etc) and disable passwords altogether if you are worried about users with bad passwords.
cheers
Having that IP isn't good enough. It doesn't prove it was valid, or if valid, originated at that other company.
You need the logs from the other company. those will prove if it came from through there, or from there.
Trying to handle that yourself with your counterpart in the other company could leave you open to several charges if you tried to go it along or with their admin's help. You'll need positive containment of evidence and chain of security.
If too much time has passed, alert the authorities and keep them on alerat in case it happens again.
A honeypot might be helpful.
"I may be synthetic, but I'm not stupid." -- Bishop 341-B
This should never have happened, it should not be possible. All servers should be protected by firewall from the internet, thus preventing nasty external attacks. At the very least, in case it's a web server/email server/ftp server as well as a file server, you should only allow those ports through the firewall.
Or, alternatively, place those functions onto a different server. Internet functions like WWW, FTP and so on are generally better served from a linux server, and those servers tend to have lower hardware requirements, so cost should not be an issue.
There is no excuse for shoddy security when it is such a basic thing to correctly install a firewall to protect your environment.
I am government man, come from the government. The government has sent me. -- G.I.R.
Block the the IP address attempting the hijack at your gateway, in fact block all IP ranges that don't need access. Limit your network in everyway possible to just what you want coming in.
I once had someone uninvitedly hanging out in the Gentoo server at work.
They were logged in with (password) SSH, as a legitimate user, merrily scanning random subnets for more victims.
I am kept busy with so many other things, mostly outside of the shop, that I only get a chance to peer at logs after I am informed of a problem.
And so I never got a chance, that week, to look at logs. And nobody bothered to email any of the usual addresses (root, abuse, whatever) during that time. Besides, I often only get to check my own email once or twice daily anyway.
Eventually, some clever and perturbed individual called on the phone to raise hell, because he was sick of our box poking at his.
And, finally aware of the problem, I was able to solve it. Turns out it was just an incredibly weak password, assigned by a cow-orker Way Back When, on a box Far, Far Away, that was kept around for legacy reasons. In discovering this, I found half a dozen other accounts with the SAME FUCKING PASSWORD, and changed all of those, too.
Learned: Let the responsible people know when particularly strange attempts occur, and the sooner the better. Had I been aware of the problem days earlier, I would have eliminated it days earlier, and the world would've been [slightly] better off. Email them, call them, get their attention. Like I said: If I'd known earlier that there was a problem, I'd have fixed it sooner.
Also learned: Crack the shit out of your password database on a regular basis, even if you know that there hasn't been a single change. If anything weak is found, fix it by using something more secure and notify the affected parties. (I wasn't doing this, but you bet I am now.) It doesn't matter how up-to-date you are on security notices and patches, if you've got even a single bad password somewhere.
if you have 0wn3d ANY of the routers between the target and the host then you can sniff/redirect any traffic you wish, including the return packets from the target to the host, sort of man in the middle (but technically thats different).
ERR 411[Max number of witty sigs reached]
It sounds completely irrelevant. They tried guessing some passwords a few times, they failed .. the problem is just not there.
I accept your argument with the reservation "It remains to be seen whether it too will eventually come to be regarded as useful and unexceptionable." :-)
SURELY NOT!!!!!
This is no serious threat. You had 50 attempts from one address big whoop. What is the timeline on the logfile? Look scriptish? I get these on my home network constantly, of course malicious laugh ensues, ah the noobishness. A locked down box is a happy place. If your network is deployed correctly, and the logs show no evidence of breakin, merely attempts, I would simply keep an eye on the logs. No action should be taken to notify anyone, if it is the rival company then let them try. Worse comes to worse you have logs of many many many failed login attempts. Management finds out, you simply say they were just that login attempts, no actual access gained, and that you didn't feel it was a severe enough threat to merit telling them. You are paid to handle it and handle it you did. With logs you can even show a timestamp of where it is an automated script if it is what is trying.
I am Bennett Haselton! I am Bennett Haselton!
My perspective is that of someone, in a past life, who hired network techs.
If this happened in my organization, I would expect three things from my network people:
1) Follow and stay within established policy; I would expect you to do what is needed to protect the security of the network short of attacking the presumed culprit. If it came to that, bring the network down. Attacking the apparent culprit puts my business at legal risk and you do not get to make that call.
2) Notify me (management) as soon as possible. Give me all the facts and answer my questions. Lay out my technical options objectively. Explain to me why our network was vulnerable and how we can remedy that. Don't try to spin me so I under or over react. It is my network and you work for me; I won't take kindly to attempts to manipulate me.
3) Then, follow my instructions.
-- Slashdot: When Public Access TV Says "No"