Dealing with Intruders?

drakyri asks: "I've been running a server for a small company for a few months. Recently, the number of attempted intrusions has jumped from about one every week to several per day - and these are only the really obvious attempts, like idiots who try to log in as root from the outside. The problem is that I'm not sure what to do about this. I've got their IP addresses and can usually tracert their ISP's - is there an accepted type of letter to send them without seeming like one of the corporate cease-and-desist gnomes?"

Comments

Abuse@

[craigske]

The accepted way is to send an email to abuse@ or to the abuse contact listed by ARIN for the netblock you are trying to lart.

http://www.arin.net
or lookup the RADB abuse contact
http://www.dnsstuff.org

Snort + Guardian

[UltiSkeeter]

These two will detect most automatic attempts and then add the IP's to a drop list on your Linux firewall. www.snort.org. Guardian is listed under 'other tools'

Well...

[MrWorf]

I always write a really "nice" letter to the ISP of the intruder, where I explain the problem, and that it is causing my customers trouble and that it eats up valuable bandwidth. I ask them to take action, and if not, that I'll have to proceed further (never been needed once). I send the email from the admin account, sign it with my name + admin at my system and then I attach the logs pertaining the intrusion attempt.

So far, all of these "cease and desist" letters has resulted in action on the ISPs part, and in 50% of the cases, their admins write me back and give me feedback on the problem.

Ofcourse, I don't do this for every attempt (all depending on my mood ... atleast nowadays), mostly for the more serious attempts (doing multiple attempts, different attempts, etc).

The worst (or craziest?) attempt yet was by some nut who portscanned the system, port by port from start to finish. I actaully managed to get hold of the owner of the computer system that was scanning me and phoned him. Quite a hilarious experience. Needless to say, the portscanning stopped :)

Re:Well...

[zoom]

I've had similar experiences. I've noticed several SSH attempts on my server recently - just a personal server at home. I've written to the abuse addresses found by running WHOIS and politely informed the ISP that there was an intrusion attempt and could they please inform the user that we are not a public service.
Many times the ISP has responded and usually their customer has a zombie box.
Always include a log if possible so they know the time and the IP-address. Remember to tell them what timezone the timestamps are from.
WHOIS links
http://ws.arin.net/cgi-bin/whois.pl
http:/ /www.ripe.net/db/whois/whois.html
http://www.apni c.net/apnic-bin/whois.pl

Yes, there are several good ways.

[arcade]

Personally I tend to ignore the scans for ssh and so forth, as they're just SYN-packets and doesn't consume too much of my resources. Call me a lazy/non-caring bastard. However, it would surely be nice to send off a message to the ISP, as the machines the scans are originating from are probably cracked too.

I tend to report viruses. I grep my logs daily for viruses from various norwegian ISPs, to the mailserver I admin for my company. During the last five months I've sent daily virus reports to the largest ISP in norway, and they tend to reply within one business day - having notified their customer about the infection. If the customer gets several 'heads up' messages from the ISP without removing the virus, they get their port 25 access filtered until they've confirmed that they've removed the virus.

I tend to send emails such as this.

"
Hi there.

I've got several viruses from your customers today, and would appreciate it if you could notify your customers about the virus infections they probably have.

Here are the relevant snippets from my logs:

Virus: Netsky.B
Received: from at

Virus: Bagle.C
Received: from at

All timestamps on the server are NTP-sync'ed against .

Thanks for your time
"

Recently I've also included a more personalized

"Oh, and I have to commend your ISPs efficiency, as since march - you've managed to reduce the number of virus sending users to us from about per day, to this .. it's days since the last virus from you! Keep up the good work!"

You could probably just adapt what I'm writing to something saying that a customer of theirs probably has been cracked, and that they are currently scanning for .. and so forth.

If it's the actual cracker that's stupid enough to use his own computer, he'll get scared enough if they contact him telling him that his computers has been abused by others to scan people -- and will probably quit doing it. :)

at some level you have to ignore it....

[cbdavis]

or you'll spend half your time at work writing abuse letters. My logs at work show a constant barrage of windows attacks ( yes, code red is still there), 137 scans, numerous login hacks for any number of OS's, port scans that increment by 1 each time, etc. Sometimes it slows down. I am beginning to just consider it background noise. Just the cost of doing business on the web. As long as the probes arent massive or working, I just note and ignore. I only have so much time for this - it keeps me from downloading all that porn!

Ignore it?

[Inominate]

This kind of stuff is all over the place. Odds are most of these are automated worms and similar crap. Unless it's really a concerted attack on your machines, as opposed to random scanning, it's not worth the effort to do anything about it except maybe firewall the IP.

Re:Somewhat offtopic, but how do people deal with

[Vo0k]

Look up HTB on the net (Heuristic Token Bucket) - a firewall rule that limits network abuse while not obstructing normal network usage - every IP gets a pool of "tokens". One token is removed from the pool when a packet is sent, packets won't be sent as long as the pool is empty, but it gets refilled at constant, slow rate, until it's "full" again. So a user can download, say, 500K in one rapid burst at maximum network capacity, then his connection bandwidth goes down to some 5K. If he waits 100s he will be able to get 500K in similar burst again. This way, one page loads really fast. User reads the page, goes back, loads another one (minute later) very fast again. A loser who keeps reloading, exceeds his 500K bucket content in 2-3 reloads and then gets a constant drip of 5K upstream, hardly disturbing the others.

Re:Very Easy

[BlackHawk-666]

Don't bother, the real crackers are probably usings some lusers box to launch the attack from. You're just warning the person who didn't secure their box, and they're not likely to understand why you are telling them they are attacking your box.

Remote logins? Are you insane?

[smoon]

Don't you use a firewall? You can't attempt to log in remotely if you're blocking the typical remote access ports -- SSH, telnet, etc.

So you've got a machine sitting on the internet, home to a million and one active worms, and are surprised that it gets scanned constantly?

Don't bother with the abuse reports -- more than likely it's just worm activity from computers whose clueless owners don't realize have been infected. A more recent one attempts SSH logins, which may be what you're seeing.

It it was a _real_ crack attempt then you:
1: Wouldn't know about it.
2: Would be unable to pin it down. It would be bounced through several victim networks, so your ability to see where it's "coming from" is really just the last victim machine in the chain.

Third possibility is script kiddies, in which case you would know about it and where they were coming from, but they would have no chance of success unless you are unwilling to keep up on patches and follow basic security practices like decent passwords.

Best would be to close off remote-login ports altogether. If you need remote login then block for all but the address range you'd be coming from. If you need remote access from random locations, then at least consider using a heavily locked down system (e.g.: OpenBSD) or work _really hard_ to get your systems firewall/logging/etc. set up well.

One OpenBSD/pf feature you might be interested in (also available from other systems) is the ability to tie Snort into the pf ruleset so that remote scanners, once detected, are ignored.

This is more fun!

[Ch_Omega]

In my opinon, Tom Hudson's way of dealing with these critters, is far more entertaining, than just ignoring them.

Re:This is more fun!

[nahdude812]

A lot of these exploits are typically ancient worms that someone has managed to not clean off their computer. If it's not an ancient worm, it's probably a zomibe in someone's hoarde.

The problem with these two (most common) scenarios that the person who owns the computer isn't the real perpetrator, and the ability to track the perp down requires much more work than a simple whois lookup of the offending IP.

Most attacks you see are going to be automated and launched on a wide scale. There are thousands and thousands of compromised Windows machines out on the net that are being used by people such as spammers and crackers for their dirty work.

Lock your box down.
Don't allow root to log in on SSH.
Lock SSH and other sensitive services down to specific IP address blocks if you can. If you can't, investigate port knocking if you can do that. If you can't even go that far, investigate implementing a lockout policy for failed login attempts.

Unless you see a single host being the source of a large pile of offensive behavior, chances are these are machines in a zombie hoarde. If it is limited to a single IP or a few IP's in a single C class, contact the ISP's abuse department *politely* (remember these are folks like you in jobs like yours, if you go in with guns blazing, they're less likely to help) and provide as much information as you can regarding the nature of the attack. Then firewall off the offending IPs.

I used to aggressively track intrusion attempts and spam. I had a little PHP/MySQL tool I wrote where I could log these things, dumping in offending logs (or spam source), and it'd extract the culprit IP address, and once a day go through, looking up abuse addresses on whois and mailing a digest of the day's activities for that ISP to them.

Ultimately I probably got about a 1% response rate from the ISP's (excluding auto-responses). After ~6 months of this, and about 40,000 records in my database, I started some statistical analysis. It turns out that there were no significant outliers for abusive activity from any given ISP (considering the size of that ISP's net blocks). Basically every intrusion attempt was some kind of zombie. There were probably a few by-hand attempts, but these are typically so low profile that there's no easy way to distinguish them from the hoardes.

Some time later I was the recipient of a DDoS attack. Someone's zombie hoarde decided to repeatedly visit a page on my website that turns out to be a bit resource intensive to generate (my code is open source, so whoever devised this probably knew that). Every day, ~25,000 IP's each requested the same page every 4 minutes (+/- a few seconds I suppose for network latency). 375,000 hits an hour = 9,000,000 bogus hits a day. Day to day this number fluctuated, and the ISP's involved in the attack kept changing. It was obvious to me that whoever was driving the attack wasn't exposing the entire zombie hoarde to me at any given point because of how the ISP's involved kept shifting around. I figured he probably had a script set up to launch X number of zombies every day, and they probably had commands to execute for ~24 hours. The number was always pretty close to 25,000, never over, but usually more than 24,500.

Ultimately the attack lasted about a month. I figured out a simple way to distinguish the zombie computers from legitimate users based on an error in the request headers, and I could just exit() at the top of my site for those who exhibited this error. I also logged the attempts I blocked, and was left with over 900,000 distinct IP addresses once the attack finally stopped.

My point in all of that is that there *are* zombie hoardes out there, and it's the zombie hoardes that are most likely to compromise you. There's little you can do about it because getting a single IP from a hoarde firewalled off or cleaned up won't slow down your real attacker who was going to use a different zombie the next day anyhow.

Two things

[Xner]

1) Tripwire is a file integrity checker. I suppose you mean portsentry or similar. 2) Automatic firewalling a VERY bad idea. Remember that most modern scanning techniques do not require a full TCP connection, and are therefore eminently spoofable. Not imagine someone spoofing a syn scan from the IPs of google.com. BOOM! No more google for you, you just firewalled it off yourself. BOOM! No more slashdot. BOOM! No more quake server. You get the idea.

Re:Firewall?

[vandan]

Yeah I know the gentleman's approach.

I don't subscribe to it. I look at it like this:

To drive a car, you need a licence. You have to follow rules. You drive on the correct side of the road. You don't drink and drive. You obey the speed limit. And why do we have to follow the fules? It's because there are other people who also want to use the road, and therefore all drivers have a responsibility to ensure that the safety of others is protected.

Sounds like common sense, right? Well the same should apply to placing computers on the internet. If you want to have viruses and backdoors and worms etc running on your home PC, then fine. Whatever. But if you put your home PC on the internet and take absolutely no fucking responsibility for what you are doing then you are waiving all rights you have over the the safety of your computer. If your computer now pisses me off, I'll 'smbdie' it off the internet. If you're fine with all the rest of the shit that's infecting your PC, then you don't really have any right to complain about me rebooting it once every 5 minutes. And yes I'm doing everyone a service. Firstly, the computer is on the internet for less time than it otherwise would have been, so there's less chance of others being infected. Also, the idiot who owns the computer will be far more likely to do a complete re-install, or at least get a god-damned virus checker and get Windows up-to-date.

Do you know how many people come bitching and complaining to me about their PC being rooted, and when I boot it up find that they're running Windows 2000 SP1 and NO virus protection at all? It's not good enough. And the only ways to get them to take responsibility for their computer are:

a) Legislate. No-one wants legislation covering their computer. It will screw things up for the responsible among us and have no effect on the rest.

b) Make it so unconfortable to run an unprotected computer that they get the hint and protect it.

Having said all this, I know most people will still disagree with me. That's fine. Be angels. Just keep your damned computer secure and you've got nothing to worry about.

Document Everything

[catdevnull]

Data integrity is more important than catching them. Rememeber that first.

1) Make notes about what you've found
2) Report the the abuse as per the WHOIS info for the offenders
3) Block their IPs at your border

If you're using a firewall, great. If not--get one.
If you haven't read Frisch's "Essential System Admnistration" read it:
http://www.oreilly.com/catalog/esa3/index.html

If you haven't read Stephen Northcutt's "Network Intrusion Detection" you should probably give it a good read as well:
http://www.amazon.com/exec/obidos/tg/detail/-/0735 708681/104-7409931-6853536?v=glance

There are some good articles all over the web regarding Linux security. A few google searches will help uncover them.

Patch. It's not just for Windows.

Limit services with ACLs and host restriction.

Harden your system by partitioning read/write slices away from static mountpoints where your binaries are by mounting the read only ones as read only.

chattr +i on your binaries--makes it tougher for skript kiddies.

Talk to other admins--every day is a school day.

AND

Face the fact that you're not as smart as the crackers so you just have to create layers of security that keep you from being an easy target.

Companies don't care.

[dougmc]

I've had problems with this a lot myself. Not intrusion attempts, but DDoS attacks. Apparantly people want my nickname on IRC, and think that hitting me with a DDoS attack until I drop off is an acceptable way of freeing it up for their own use. It's not so bad when they just go after my cable modem, but they've also gone after the place that I work at, even when I'm not IRCing from there at the time, and that's much much worse. Also, they often don't attack for the needed ten minutes -- I've had attacks going on for 15 hours, and perhaps even longer but at that point I had the ISP filter out the traffic for me.

So, being a good guy, I never respond in kind (I could, but 1) it's wrong, 2) it affects more than just the target and 3) I don't feel like going to pound-me-in-the-ass prison), I just log every single packet I can, and when the attack is over find the worst offenders (typically the packets are not spoofed) and use Spamcop and whois to find the responsible parties for each one, and send them all an email.

Many (most?) emails elicit an automatic response.

Perhaps 10% get a personalized response, but usually this response says that I should contact the ISP of the offender (when in fact that's exactly what I'm doing.) Perhaps half of the responses I do get say they'll do something about it, which is good -- usually these are compromised drone/zombie machines, and need cleaning anyways.

Quite often, the attacker is stupid enough to ping my machine from his home machine (so he can see how it's going), not thinking I'll notice that. When this happens, I can also email his home ISP, the people who really know who he is, and the people who can really hit him where it hurts. Except that they ignore my email too, and if they do email me back, they just tell me that the attack did not come from their ISP so they can't do anything, or there's no proof that the pinging is related to the attack.

Phone calls are much more effective than emails, but you really need to make them during the attack for them to take them seriously. And often the attacks happen outside of business hours, so there's nobody to call. And they're very time consuming.

Though I did succeed in nailing at least one guy. He was in Romania, and he messaged me a few weeks after the attack basically pleading with me that it wasn't him, but his brother using his computer. Apparantly the police (in Romania) were questioning him, and one of the things they showed him was my email. The police had never contacted me -- I'm guessing that my email was just one of many pieces of evidence they had against the guy. I felt a bit bad for him, but not that bad. Not that I had any control over what was happening to him at that point -- it was out of my hands the moment I sent my email.

So, if it happens again, I'll do the same thing. I know it's not likely that anything substantial will come from my emails, but there's still a chance. Every time it happens, I know I nail at least some of his compromised machines, and have a chance at getting him. I'll win eventually -- either that, or he'll hit puberty, in which case we both win.