How To Secure A Cracked Box

Noel sent us a collection of stories from rootprompt on how to secure your box. The articles include Denial and truth, Watching and Waiting, Hunting the hunter, The Sniffer, and Rebuilding the system to recover from the crack. It's an interesting discussion on what it's like (and enough to churn the stomachs of anyone who's ever been there).

Comments

Re:Dot Matrix Printers and security?

[InfoMonk]

Even though Clifford Stol used that method in the infamous Cuckoo's Egg, operating systems seemed to have progressed well beyond that type of jerry-rigging. With the potential for autoarchiving log files, automating their conversion into different formats, and not too mention the cheap availability of older PCs that can serve as independent and secure log servers, a dot matrix seems to be a resolution only for the most paranoid sysadmin. (Of course, dot matrix printouts do still retain their age old hacker appeal.)

Re:Dot Matrix Printers and security?

[vyesue]

uhhh...

auth.* /dev/lp0

...might be a way to do this without tail -f sucking half your processor 24/7.

man syslog.conf, dude.

Re:Dot Matrix Printers and security?

[Azog]

If you have a hub between the ADSL modem and the firewall box, instead of cutting wires in a perfectly good network cable, just attach a cheap old pentium machine to the hub. Set its NIC to promiscuous mode and sniff everything that goes by, but set up its own packet filtering to drop EVERYTHING.

Then it's like that box just doesn't exist to the rest of the network, but it sees everything, and can log it any way you want... It's like a shadow of the firewall - it can run any kind of security software, to set off alarms or whatever.

Disclaimer: I am not a security expert. If there are problems with my idea I would like to know about it (because I am using this idea on my own firewall setup).

Another idea I had but have not implemented is to modify the login software on my machines: If anyone logs in, they would have to run a specific "secret" program in 15 seconds or less. If not, a timer expires and shuts off the UPS powering the box.

Heh heh heh. Not suitable for systems that need to keep running, but nice for home machines that you want to keep secure.

A less extreme approach would just use ifconfig to turn off the network card, instead of having the UPS kill the power.


Torrey Hoffman (Azog)

Re:a good reason not to use *nix

[dragonfly_blue]

You know, I agree with that. You really don't need to be running things as powerful as Bind.

Unfortunately, the default installations of many Linux distros seem to be getting more and more top-heavy. Even things like Bind and Sendmail are getting installed by default; I'm not sure if this is a good thing.

One thing I like about OpenBSD is the very sparse, almost Bauhaus-style install. You have to go through manually and set things up if you want to use them.

It seems like a lot of work, and it perhaps is very cumbersome if you've never done it before, but I just feel much more comfortable running an OS that doesn't have a whole bunch of crufty packages installed that I may or may not ever want or need.

The security audit for OpenBSD helps, too, though. ;-)

Re:a good reason not to use *nix

[stab]

The FreeBSD ports tree is a one of the best package systems Ive ever seen ... it has ports for every server under the sun, including qmail and postfix, as well as a lot of DJB's other tools like dnscache and so on. And if you're so inclined, exim/postfix as well :)

The OpenBSD dudes made a wise choice and picked the FreeBSD system as their base, and they have a rapidly growing collection as well. Although I'm not familiar with it, NetBSD seems to have something similar as well.

If only we could see this under Linux now, without all the RPM crap :( The number of times I've run into stupid cross-dependencies, and corrupt RPM dbs goes on and on ...

Can't say it enough: use Kerberos. NOW!

[coyote-san]

It seems that I can't say it enough. Install and use Kerberos. NOW!

SSH is great for connecting to a shell account, but you may still leak passwords once you've established a secure connection to your "trusted" network. Kerberos, properly installed, ensures that your passwords *NEVER* appear *ANYWHERE* in plaintext, and rarely appear in ciphertext. After all, you never know when someone has compromised one of your local tools, e.g., psql.

Re:Why can't we all just get along

[ajs]

You say this jokingly, I presume, but what if such a beast did exist? I would run a client of such a system *if* it contained a Slashdot-like moderation system that allowed people to propose code (e.g. post code they wanted to run to a public forum). When a piece of code gets enough votes, it "goes live" and people start executing it. Would this result in problems? Sure. Do I care? No.

Seems odd, no? Well, I say the Internet was put in place by people who had bigger dreams than a really fat pipe for advertizing. I think the Internet is actually a cool thing, and should be used to its fullest. This would give it that chance, but would also come with risk. Ok, I can do risk....

Anyone up for writing it?

Re:how to secure a cracked box:

[Admiral Burrito]

well, if your binaries are all on read-only media, maybe there are subtle backdoors hidden in your rc.files; maybe configurations files for daemons have been subtly altered to provide a way back into your machine even after you think it's resecured.

Indeed. I've spent many hours thinking about this...

Suppose you lock down your system really tight. You use Linux capabilities or BSD securelevel to set your binaries and config files (and directories! don't forget the directories and their parents or "mv" followed by "cp" is all it takes to trojan your stuff) read-only and your log files append-only in such a way that not even root can mess with them.

Being a security-concious person you insist upon changing your passwords regularly. This requires /etc/passwd to be writable by root.

Your login shell is specified in /etc/passwd.

Some intruder gains root, discovers he can't trojan the system binaries or wipe his footprints from the logfiles because of all the lock-down you've done. No problem! He changes your login shell in /etc/passwd to point to a little program that chroots you into a special jail directory heirarchy where all of your usual tools and logfiles can be found, in trojan form. Since the intruder hasn't altered the protected stuff in /bin, /var/log, etc. he hasn't done anything your capabilities system can prevent.

Bingo, you are now the clueless luser in the honeypot.

Re:Interesting... is there such a thing for Win NT

[LiNT_]

http://www.ntsecurity.net

They have a great NT Security book online as well as a bunch of great articles, tools and links.

LiNT

Why not CDR? (Re:Dot Matrix Printers and security?

[yellowstone]

I have been playing with the idea of hooking up an old dot-matrix printer to print out certain log files, or lines from log files with keywords in them.

Why not stream them to a CDR? IANASE (I am not a security expert), but it seems to be CDRs are also write-once, but have the additional advantage of being greppable (not to mention cheaper and more environmentally friendly -- you would have to kill a lot of trees to print out the text that fits in 650 megs...

How to secure a cracked box

[Hard_Code]

Items you will need for this procedure:

1. Superglue
2. Strip of cloth or large bandage
3. Tape, twist tie, or rubber band

First, apply superglue to both sides of crack, and press pieces together. If superglue comes into contact with hands, follow instructions on back of package to remove. Do not attempt to lick off superglue.

Wait. Until you're tired of waiting.

Take strip of cloth or bandage and tie it around box, perpendicular to the axis of the crack. Secure cloth tightly by either tying it in a knot, or by using tape, a twist tie, or a rubber band.

Refrain from dropping or throwing your box out a window to avoid the risk of future cracks.

(sorry, something makes me do this)

Re:a good reason not to use *nix

[stab]

Well, to be honest, its your fault for using BIND!

BIND is notoriously insecure, so you should always run it in a chrooted environment if you are going to use it.

Also, investigate alternative, and far superior servers for services you want to run.

Instead of BIND, look at Dan Bernstein's DNSCache package, which is lightweight, stable and uncrackable. In fact, he offers a monetary reward to the first person who can find a security hole in it.

Similarly, replace sendmail with either qmail, exim, or postfix and get a superior, more intuitive feature set, and better peace of mind security wise.

Also, look at a more secure OS than Linux, for example OpenBSD which has not had a remote security hole in its default installation for over two years now.

Re:a good reason not to use *nix

[jeffstar]

cmon, distros should not come with packages enabled by default that are notoriously exploitable ...

people who are new to *nix need time to learn the ropes, and if they lose all their data and have to reinstall it can be a major turn off

Re:a good reason not to use *nix

[Hardwyred]

If the box was being used for the same purpose that a windows box can serve, why run bind anyway? The problem is not the OS, you'll be hard pressed to argue that comparing linux running no services and a windows box running no services, that linux is less secure. Or any un*x for that matter. The key is to know the purpose of your box from the start. Are you building this box just as a gateway? Then you shouldnt need any services running. If you are going to use a linux box as a router, then think of it as a router. If you are going to use it as a firewall, then think of it as a firewall. How many firewalls have you seen, PIX and what not, that have DNS or mail servers running on them? None. The problem is not the OS, the problem is education.

If you want an all-in-wonder box that will do your masq'ing and firewalling and mail hosting and web hosting and DNS and wash the dog, then you need to at least research the services you are going to use and be prepared for the attacks. BTW, a do-all box is just a bad idea IMHO. Whats the point of having a secure firewall and then running non-secure public services on it? A little forethought would have saved you a lot of time.

... and the geek shall inherit the earth...

Rebuilding is what they suggest in the end

[sleeping wolf]

Well, if you read the series of articles, you would know that they finally decided they had to rebuild the system.

That in fact seems to be one of the two morals of this bunch of articles (yes, the series isn't over yet): If you're cracked, start from scratch; If you're not, make sure your network is planned from the beginning. It's far too easy to patch it together and have it work "well enough" and discover some bitrot (or worse, someone crawling in your walls like they did).

Of course, the fact that they had it done by volunteer sysadmins didn't help -- they didn't have the time to watch things as well as paid ones might.

Re:Reacting to port scans

[Shadowlion]

Truth decays into beauty, while beauty soon becomes merely charm. Charm ends up as strangeness, and even that doesn't last, but up and down are forever. Quarks, right?

Re:Dot Matrix Printers and security?

[vyesue]

on a machine:

*.* @loghost.my.net

on loghost:

auth.* /dev/lp0

make sure you give loghost's syslogd a -u on the command line.

Err...

[Wakko Warner]

...why not just buy a cheap Mac or something that's relatively impossible to hack and run syslogd remotely on it? Send a duplicate of all log messages to the Mac.

Can syslogd be forced to send messages to a serial port? Connect a non-networked machine of some sort to the networked machine(s) and have it listen on the serial port for data.

Either way, you save reams of paper.

- A.P.
--

"One World, one Web, one Program" - Microsoft promotional ad

Dot Matrix Printers and security?

[billh]

I like the idea of certain log files that cannot be erased, so...

Upon installation of SDSL, I will be moving my webhosting services to my home. I have been playing with the idea of hooking up an old dot-matrix printer to print out certain log files, or lines from log files with keywords in them.

Am I hopelessly out of date with this idea? I have seen some mention of systems like this, and I think it will be a good complement to other security. The idea is that if I get a penetration, I will at least have an idea about *when* the initial intrusion was, and be able to work with that.

Anyone else with a similar system care to comment?

Re: Dot Matrix Printers and security?

[InitZero]

You should never have security logs the machine those logs secure. While I find the printer idea pretty darn cool, syslogd directed to another box would be fine, too.

Assuming, of course, that machine was secure.

I worked at a company where the most secured system in the entire building wasn't the firewall, mainframe or the accouting system. It was the syslog box.

The only service the box was running was syslogd. It allowed no telnet access and all ports except 514 were closed into and out of that system. In order to physically touch the machine, you had to break a seal on the box (literally, a locked plywood box with a fan in the back) to get in. (Remember, seals are not locks and locks are not seals.)

Our position was that you couldn't keep people from doing bad things but that if someone did a bad thing, we wanted to be able to hunt said person down and render vengeance not heard of in thousands of years.

Of course, not even this system was fool-proof given UDP and that the network had to be up. But nothing is perfect.

Dot matrix printers rock.

InitZero

Re:Dot Matrix Printers and security?

[Anonymous Coward]

Linux ext2 supports append-only files:

chattr +a /var/log/syslog
echo 1 > /proc/sys/kernel/securelevel

chattr +i makes it immutable (read-only)
e.g., chattr +i /bin/login

you want to have the system change to securelevel 1 prior to going multi-user. That way, the system is only at securelevel 0 in single-user (non-network, console only) mode. At securelevel 1 and above, chattr doesn't work (so the h4x0r can't chattr -a /var/log/syslog).

A very cool security feature that doesn't get much "press". This is 2.0.x, btw, dunno how it works in 2.2.x. Anyone? Anyone? Ferris?

Re:Dot Matrix Printers and security?

[blakestah]

There are a variety of ways to do this.

For a large enough center you can set up a box to receive log files and only allow console logins. For example, set it up with the only inet service being qmail, and redirect all logs to mail to various inboxes. I am sure there are more elegant ways to do this. The concept is that the box saves all the log files, and you require physical access to check the box.

Dot matrix printers are not a bad idea. But a one way box doesn't run out of paper.

Chances are, though, you will know when you've been hacked (if you check logs regularly). Often someone else will tell you. Your only real recourse is reinstalling the OS - especially all the boot scripts and boot binaries. Keep backups, and don't pull your hair out when you get hacked - it happens all the time. Just be responsible, reinstall, and set up more securely the next time.

Re:Dot Matrix Printers and security?

[synaptic-impulse]

here is the way I do it:

we have many systems in house and collocated that get scanned and attacked regularly. we use syslog to pipe ALL logs back to a central server. this server then runs LOGCHECK against the logs, and emails and prints all signatures found.

Logs are reviewed as the come in via email - and daily the printed logs are reviewed by several ppl to ensure that "many eyes" look for anomolies.
This way - we never miss anything that looks strange.

We ran this system with no filters for about a month and a half to determine what items would be ok to ignore (standard system events, cipe key syncs etc)

In addition we run port sentry, and lids. port sentry will permanently block any IP that scans us (we get scanned at least 3 times a week) and lids is setup to make all log files (and others) to be APPEND ONLY - even by root.

Unless our systems get compromised AND the hacker can unlock lids - he really doesnt even have root access.

Last - any scan that comes in gets investigated.
1. permently block that IP
2. trace the IP (ping,nslookup,whois etc)
3. contact that site/isp/etc. via email with the log excerpts that show the attack.
4. archive for eventual turn in to FBI

here is something that you will really find interesting: this is the response from one scan that came through:

We sincerely regret any inconvenience/consternation the probing from 216.181.81.11 may have caused you and/or your organization. The machines that have had the name excaliber.barnhard.net have been the subject of a number of attacks which have been investigated by the FBI and in some case may still be under investigation. Based on the prior investigations we agreed to make a reasonable effort to collect data concerning any subsequent attacks, and in particular any attacks which may have some relationship to prior events. Whereas it is possible that three different random hackers have figured out independently that the machines bearing that name are used for testing/training on our network and have used an exploit suitable for whatever variant of Linux happened to be installed at the time. I think as the number of times it gets hit increases it is increasingly unlikely. Regardless, the boys/girls responsible for this are likely unaware that once an attack is confirmed we activate an upstream monitoring process that records all of the incoming packets, which we will forward to the FBI. Our poor abused testing machine then gets backed up to tape, wiped, and then reincarnated when needed again. It is interesting, but it is also getting old fast. As such, we have made the decision that our future test machines will be locked down boxes like our production equipment. If anyone is interested in the construction of suitable blackhole boxes that could assist the FBI in tracing these folks, instead of having to leave hacked machines live I think it would be a good thing. I am sure they would be interested. If we could lay a cracker trap that would only cost a modest amount of bandwidth and CPU cycles that could gather the necessary evidence on the cracker without enabling them to carry out real attacks, I know I would be interested.

Once again, thanks for letting us know you were scanned, We sorry to have darkened your doorstep in these regards. The machine has been taken down and subsequently replaced.

If you have any questions related to this matter I can be contacted at the address/email/phone shown below. Our contact with the FBI is Special Agent Kevin M. Walsh who can be reached at kwalsh@leo.gov.

Re:Dot Matrix Printers and security?

[ion]

if you want more security... and is really paranoid like me ;-0
then why not this senario

ADSL -> hub -> server
also in the hub is a network cable, that has the SEND lines cut over, so the machine only can receive.
On this machine you constantly "record" anything on the network, much like the tivo.
Then you run real time checks on the netpackets, and the most strange you log to your hd. If it is big enough, and the site is small enough you could have a day, a week, perhaps a month's data on the disk of suspisious connections.

As for the syslog... why not send them over the serial line into the previously described computer, and on this computer dump everything into a text file so no command could ever be executed, simply anything from com1 is saved as /var/log/log_from_server

and then you run your logcheck program on the log.

ion++
ps: i remember someone video recording the console which was writting everything to the console.

Re:Dot Matrix Printers and security?

[Nater]

I tried this for a while, but my printer was a little weird dealing with log files, so I eventually got rid of it. When I did tho, I deleted /dev/lp1 and then remade it with the device numbers for tty9, and left the syslog configuration files alone.

Some time later, someone did try to get in, but they saw a bunch of stuff logging to "/dev/lp1" and left. So even if you don't have a printer, if you can make it look like you do, you'll scare off a few that way.

Re:Dot Matrix Printers and security?

[Colitis]

How about logging via serial port?

A machine that's only saving to disk anything coming in via serial and has no network connection will be *very* hard to crack, and you have the advantage that your logs are still in electronically searchable form.

Why can't we all just get along

[Shoeboy]

A modest proposal for making life easier for DDoS crackers

I have an idea. I think it's brilliant. When you want to DDoS a big site into the stone age, most of your time is spent infecting hosts to use in the attack. This is annoying and it causes us to behave in antisocial ways. If I wan't to bring down Yahoo, I want to do it NOW!!, not after I finish setting up a subseven network. All the work I have to do makes me pissy. When I get pissy I wipe your hard drive to cover my tracks. Now you're pissy too. Misery loves company.

What I have in mind is a massive voluntary distributed computing effort along the lines of Seti @ Home. I call it kiddie @ home. Basically, those of you with cpu and bandwith to spare should sign up. When you aren't using your computer, I'll use it to launch SYN attacks and settle grudges. Now I don't have to crack anything, and you don't have to bother reading cert advisories. We're both happy.

What do you guys think? Can I sign you up?

--Shoeboy
(former microserf)

Don't run demons or a distro that defaults them on

[divec]

For someone moving from windows to linux (say like my dad) [...] it's a good reason to stay with windows

It's better for a non-server machine to be running as few services as possible - at most, only ssh should be neccessary. Get your Dad to pick a Linux distro that doesn't install lots of cruft by default. (I've heard that Red Hat is bad at this but I wouldn't know).

BTW is it possible to run BIND ok in a BSD jail? (jail is chroot's big brother)

Re:The cost of growing to fast

[kuro5hin]

You know what's really funny? We've run links to, I think, all of the installments of the Cracked! series individually, and every time another one comes out, somebody bitches about how "if we wanted to read this, we'd be reading rootprompt already". Just goes to show, you can't please everyone. :-)

--

tripwire note

[gbnewby]

IFF your tripwire is statically linked AND launched from a read-only medium (CD, locked floppy...): you might have more of a chance.

Anyone have further illumination to offer? tripwire still needs to call system functions (e.g., to open files), even if it's statically linked. So, a call to open the changed/hacked files might result in forged data being sent.

But this would be a much messier hack...if, for example, the legit sysadmin makes a change to / (the directory), the hacked kernel would need to know to send the current info back via tripwire, instead of the info from when the kernel was hacked. It seems to me like hacking around tripwire would be its own project! (Anyone done it yet? Anyone?)

Wouldn't a WORM drive work a lot better?

[El]

The benefit of keeping logs in electronic form is you can search through them a heckofa lot quicker... ever try to find an event in a 2000-page stack of printouts?

A gram of prevention is worth a Kg of cure....

[fluffhead]

Try securing your systems BEFORE they get cracked. A good few places to start:

Insecure.org, especially this top 50 security tools page.
SecurityFocus the disseminators of the BUGTRAQ list among others.
Attrition.org, especially their security page.
And of course 2600, the l0pht, and Phrack for the latest tasty street info....

#include "disclaim.h"
"All the best people in life seem to like LINUX." - Steve Wozniak

Re:rootprompt.org Is Down!

[ptomblin]

It is back up now. :)

Wanna bet?
--
A "freaking free-loading Canadian" stealing jobs from good honest hard working Americans since 1997.

That's insufficient

[coyote-san]

Apt-get will install an updated package, but (unless you --force it) won't reinstall a currently installed version. This means cracked applications stay cracked.

(And don't make me laugh by suggesting that a cron job running apt-get install --force ... will be enough to stop a knowledgeable intruder who already has root access on your system.)

Running apt-get religiously is a good start, but you also need a well-configured tripwire, log host, etc.

Next How-To...

[rjamestaylor]

the Next How-To should be "How To Restore a Slashed Box".

There's just too many of us!! ;)

Re:The cost of growing to fast

[noeld]

FYI The latest article in the series was posted today. Not weeks ago. I am glad you have enjoyed them :) Thanks

Noel

RootPrompt.org -- Nothing but Unix

Re:Interesting... is there such a thing for Win NT

[AugstWest]

No NT Admin should ever be without NT Bugtraq.

Subscribe to the mailing list and sit back and watch your inbox. Dig through the archives if you're a new user. You'll be amazed at the sheer volume of security issues that floats through on a daily basis.

Security LRP?

[whoop]

Over the last few months I have taken to running tcpdump on my connection just to see how many folks try and get in. Understand I am in a cable modem/DSL deprived area, so I dial up with my mighty 56k modem. My ISP uses two C class blocks for the dynamic IP dialup sessions. So I guess crackers are just making attempts at any/all of these class C networks.

I'd say I get about 4-6 attempts per day to do something on my box. Mostly it is folks looking for something good on Windows SMB ports. I'm sure there are millions of 2 PC households that share their C drive wide open so they can copy to and fro. I've gone through the logs keeping a list, and banning the entire class C network of offending IPs. You can see some of that on my site under Security.

All those attempts got me to thinking. I should set up a much simpler firewall/masquerade box that doesn't run too much. Holes could be poked in the wall for necessary services (web, mail, etc) and forwarded to an internal machine. Perhaps something like the Linux Router Project would work. But what I'm looking into is that, with good crack monitors, syslogging things to another box, checking for portscans, running snort or tcpdump. Are there any? If not I may have to start one.

Even if someone finds a hole in the mail server (or whatever), it is on a second machine beind the wall and they cannot (easily) get to it to run that suid shell they just created. If the system is kept down to a floppy or small bit of a CDROM, you can easily mount the entire ramdisk readonly, or just reboot and have the original setup restored. Just having a full Linux desktop setup directly on the 'net worries me when/if I move to a DSL area.

Where to begin?

[Inoshiro]

As others have said, you should always reinstall after noticing your boxes have been cracked (you'll also want to check on things to see if you can determine the point of entry and person(s) responsible).

The better solution is to just not be cracked in the first place. The way to do this is to be known-secure. How do you do that? Audited code, such as OpenBSD provides peace of mind. Secure logging (i.e.: logging to another internal machine whose job it is to accept log reports) -- this gives you a nice write-only log target, making it easier to trace intitial probes and attacks.

Next, you'll want to check existing services, and review any services you want to add. I discussed this in Securing the Border, parts 1, 2, 3, and 4.

You might also want to read "Auditing Kuro5hin" where I found a root compromise on Kuro5hin.org when reviewing the system with Rusty, the site owner and creator. It has tips on how to recover cleanly.
---

freakin DDOS attacks.

[jon_c]

looks like rootPrompt has succumbed to the infamous ./ DDOS attack!

-Jon

Be sure to catch Noel's next article

[ch-chuck]

titled: "How I delt with over 48,762 simultaneous http connections refered from /."

Part 1. The onslaught
Part 2. I've never seen a disk so busy
Part 3. Out of swap space
Part 4. Internal Server Error
Part 5. The crowd finally goes away

Re:Oxymoron

[Accipiter]

And 'Microsoft Works'

:)

-- Give him Head? Be a Beacon?

rootprompt.org Is Down!

[sumana]

Rootprompt.org is down. I got a server error:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, noeld@pair.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.

couldn't spawn child process:
/usr/www/cgi-bin/php-cgiwrap

Noel, please fix! I was about to read the last installment! The suspense is unbearable.

Re:rootprompt.org Is Down!

[noeld]

Help I am under SDDDOS... :)

It is back up now. :)

Noel

RootPrompt.org -- Nothing but Unix

Re:securified

[AugstWest]

Great, this will really help the first-time users of Linux.

Nothing helps a newbie secure a box quite like someone telling him he should already know how.

Re:Who was it that said...

[AugstWest]

Who was it that said that the most secure computer is one that is not connected in any way to anything (including power), that has no periferals, and that is burried 8 feet down - and even this level is arguably insufficient....

I once had a technologically-unaware boss (owner of an ISP [sigh]) who suddenly freaked out and decided that I was hacking every system he owned.

At one point he was telling people that I had edited his autoexec.bat file on a machine in his home that wasn't turned on and had no modem.

I was also supposedly running DNS servers that circumvented Internic, Quake servers for all my friends, and also stomped.com on office machines, all on a 33.6 modem.

Ah, paranoia and ignorance, what a blend of ambrosia you meld.

System Design From The Beginning

[I0ta]

If you've been hacked, reload.. It's that simple. If you designed your system 'right' from the beginning, this isn't a big deal. Here are some basic steps I've used that anyone can use.

1) Get your *nix (or any os) setup the way you want, with patches, drivers, etc..Don't load application software yet. Create an Image of that machine.(ghost, drive image, etc..)
2.Load your applications.
3.Set your syslog to mirror your logs on another server.
4. If possible, try to move your 'data' directories (from your applications) to another directory for just 'data'. (You'll have to create symbolic links from their original locations.)
5. Backup your DATA Directory/Drive ONLY!

Too many times do I see people backup their entire system whether it be Winblows or *nix. If you get a virus, or comprimised binary, that file/binary will be backed up! If you don't catch the attack, all of your backups could be infected.
A good rule of thumb is too only backup your DATA, not your binaries. After all, you own the software, right ? *grin*
Then, the obvious solution after a hack is to:
1) Reapply your OS image (ghost, drive image, etc)
2) Apply new patches/fixes/close security holes.
3) Reload your Applications
4) Reload your data
5) Point the applications to your data on the other drive.

Yes, it can be a long, drawn out affair initially, but whether it be a hacker or just plain system crash, the restoration process goes rather smoothly.

-Iota

how to secure a cracked box:

[vyesue]

reinstall.

seriously, if your machine has been compromised by anyone other than a completely retarded skriptkiddie, chances are there's going to be lots of "new functionality" in some of the bins on your machine. reinstall from read-only media.

Re:how to secure a cracked box:

[heliocentric]

But what if you were paranoid and kept your binaries on read-only media (think link /usr to /mnt/cdrom/usr or whatever, maybe a samba mount of another machine's CD drive) such that no one could touch it?

Or in a less paranoid sense, you kept copies of said files on-read only material so that a trip-wire-ish program could compare the information about your current binaries and those in the "backedup" state. If there were inconsistencies you could just restore from the backup.

If you are worried the script kiddie got to your trip wire program then maybe you should have been running your trip wire remotely (just mount your current hard drive system to that of your "security" server) so the cracker would now have to get to another machine to cover their tracks and do inflict their damage (if any).

Re:how to secure a cracked box:

[vyesue]

well, if your binaries are all on read-only media, maybe there are subtle backdoors hidden in your rc.files; maybe configurations files for daemons have been subtly altered to provide a way back into your machine even after you think it's resecured.

if your machine is owned, tripwire can be subverted. it's not trivial to use tripwire correctly, and even if it is used correctly, it can still be tricked.

as to your last point - once one machine falls, other machines on the netowrk become progressively more prone to falling too. think communists in SE asia, you know? :D

What I'd probably do

[antizeus]

I'd probably just copy anything that couldn't be infected by viruses/trojans/etc off to another system, then wipe the machine and start over, perhaps paying more attention to security next time. There are probably a lot of people for whom that wouldn't suffice.

a good reason not to use *nix

[jeffstar]

I have been compromised twice before. Both times through bind. The first time some hax0r group found my box by scanning for computers running bind. they installed all kinds of root kits, and i didnt realized it was jooked for a few months. Its a headless ipmasq, so as long as it works i dont care... Anyway i unplugged it from the net when i found out. I found out when i tried to install something and ps aux was not showing a bunch of things that were supposed to be running. they had messed with it so it didn't show their eggdrops.

the next time was bind again, but the guy rebooted the box for some reason and then i found him on irc (was using the same nick as the account that he added, and IPs matched), and i asked him how he did it and he said bind.

i dont run bind anymore ...

i reinstalled after the first time, but not the second.

That kind of thing can't really happen with windows, yeah you can get back orifice but norton antivirus takes care of that. For someone moving from windows to linux (say like my dad) if he hears that he has to check some web page and subscribe to mailing lists to keep on top of latest exploits that will root his box, it's a good reason to stay with windows.

Re:a good reason not to use *nix

[Phroggy]

If you run any services at all, you should know exactly what daemons are responsible for them, and keep those daemons updated periodically. If you don't need the daemons, you should turn them off. This applies to any OS - UNIX, Windows, Mac OS.

Sure, there are a lot of UNIX exploits published frequently. Just because the Windows exploits aren't published so frequently or the details made readily available, don't delude yourself into thinking that Windows has fewer exploits.

--

Slashdotted!

[Megane]

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, noeld@pair.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.

couldn't spawn child process: /usr/www/cgi-bin/php-cgiwrap

Even on their main page. Damn. Just as I was getting to part 5.

This begs for a follow-up series on rootprompt.org: How To Secure A Slashdotted Box

An overlooked possibility?

[YASD]

Well, rootprompt got /.ed before I could read any of it, but in the meantime, here's a scenario I thought of a while back, that doesn't seem to have been dealt with much.

OK, some kiddie has cracked your box, played around with files, executables, logs, etc. So you start from scratch: boot off a CD, fdisk the partitions to hell, reinstall. Great. Everything's clean.

Now: what if you have flash BIOS?

At the very least he could zero out your BIOS and make your machine unbootable. If your version of Un*x uses the BIOS for anything but booting, it might be possible to leave a back door, too.

Thoughts?

------

OpenBSD

[ArchieBunker]

How about using OpenBSD? You won't have to check bugtraq every few hours. Two years without a root exploit is a pretty good track record.

Two Letters: PM

[Amphigory]

Namely, Preventative Maintenance. If you're running Linux, with it's fast turnaround of bug fixes, you can prevent most cracks by just installing the OS upgrades on a regular basis.

C'mon guys... I know you love your uptime. But if you download the Redhat (or Debian, or whatever) updates once a week, install them and reboot, you'll save yourself a world of trouble. Depending on the updates, you don't even need to reboot -- but it's usually the easiest way to make sure all the daemons have been restarted. Plus it cleans up your memory pool.

I have seen many boxes compromised. But there are two configurations I've never seen hacked:

1. Redhat w/ latest updates.
2. OpenBSD.

Note that closed source OS's seriously suffer in this area. Running Solaris (second only to Linux in the number of exploits), your best bet is to replace /every/ server program you can with the latest Open Source alternative. I've seen sun take 6 months to turnaround security bugs (granted years ago, but still).

--

no such thing as recovery...

[blaine]

Only rebuild. The only possible ways of fully recovering a cracked system are:

1. reload the entire system from a known-good backup

2. reinstall the entire system

However, #1 isn't always possible. First of all, if you don't keep backups of your system, you are SOL. Even if you do, if you don't keep backups around for long periods of time, it is possible you don't have a backup from before the initial intrusion.

If anything, you CANNOT trust ANY data/programs/etc from the cracked system. ANYTHING and EVERYTHING could have been modified by the cracker. Trying to plug the hole after its already been used is pointless, as you have no way of knowing what they've changed. If you just update whatever program was the problem and move along your merry way, you're just asking for a repeat of the initial breakin.

Re:Interesting... is there such a thing for Win NT

[stokessd]

Anyone know of a place for Windows NT Security?

Betty Ford Clinic.

Sheldon

Why is security so hard?

[scruffy]

With all advances in computers, why is a secure computer so difficult to do? To me, it appears to be a fundamental design error in the way that operating systems are put together. But where is the error and how can it be fixed?

Yes, I know there is OpenBSD and other more-or-less secure OSes. But it is still very easy to create security holes, and it is a lot of work to keep a system secure. The millions of ordinary users soon to come on cable modem and ADSL won't appreciate doing this sort of work.

So what is really the problem?

Re:securified

[bluGill]

Although no dates are given, the way the artical reads I suspect the attack took place several years ago. In 1995 there were remotly exploitable root cracks in openBSD. (Which if I remember right was just coming into being and still was mostly netBSD+ and not really worthy of its own name yet - maybe it didn't even exist at that time)

Work with the best tools avaiable. But sometimes the best tools are not very good.

PS, I could be wrong on the date, but this is my impression. It seems the author has learned a lot since then.

securified

[xp0rnstar]

If administrators kept on point checking out advisories as well as following forums such as securityfocus, etc. This wouldn't be a problem.

When someone has to go as far as detailing a document on recovering a cracked box you have to stop and wonder about the level of security this person knows about since their machine was "rooted" in the first place.

Sure you could moan and bitch about script kiddiots/crackers/e-vandals but a secure box isn't as far fetched as a clean install of OpenBSD or even running Titan on your clean install of Solaris.

Sorry to say but slackness is to blame when dealing with situations like this. Never... Wait no... NEVER have I had to worry about recovering a "cracked" box since it'd been secure from the get.

Someone root me so I can have fun creating my own docs...

sil@deficiency.org www.deficiency.org
sil@antioffline.com www.antioffline.com