Slashdot Mirror
Ongoing Linux/Solaris Compromise Epidemic
An anonymous reader writes to point out that Stanford's Information Technology Systems and Services "has written a summary of a series of compromises that have been happening at universities, research institutions, and high performance computing centers, for the last month or more. The attackers are using known vulnerabilities in Linux and Solaris, along with compromised user accounts, to gain access and control of systems, from standalone servers to HPC clusters ... (the attacks are still ongoing)."
A good substitute for Linux and Sun boxes. My school migrated two years ago, weren't happier ever since.
Here - those guys make a kernel, kickass GUI environment (faster than GNOME and easier to use than KDE) plus some office word editors and educational stuff like encyclopedias and maps.
I'm running Windows XP!
aQazaQa
It is important that when we wave our flags and cheer when Microsoft is laid low by the latest security flaw that we not close our eyes to the very real vulnerabilities in the Unix/Linux system. No OS can be fully secured, and it is absolutely mandatory that we remain vigilant to the possibility of a heretofore unknown security hole in our systems, regardless of the system OS.
Assuming that Unix/Linux is invulnerable to security holes is deadly. Though the OS may have more security features and "more eyes" on the code than closed source operating systems, we must not rest on our laurels watching Windows implode while our own house is burning.
I have been pwned because my
a variety of local exploits, including the do_brk() and mremap() exploits on Linux
In other words, Stanford doesn't keep its Linux boxes up to date. These exploits have been fixed. Linux too requires maintenance and patching, not just Windows.
"A door is what a dog is perpetually on the wrong side of" - Ogden Nash
It says that good passwords are a good defense.
We know this.
No more default last 4 digits of SSN as a password.
Make them use something more secure! And disable telnet, for goodness sakes.
Inconvieience (sp?) your students in order to secure your system. It's all fun and games until someone uses a rootkit to play with GPAs.
Jay | http://oldos.org
going back to the back-door insertion attempt on the Kernel, the rooting of gnu.org's ftp server, the compromise of Debian's servers... it's the same people doing this.
Just a feeling.
on just how widespread this attack really is. The story IS HERE
...because you never know who you're dealing with.
Don't send passwords in plain text on the network, and enforce proper password policies (8 char minimum, numbers, letters and symbols etc).
Change Linux root password from 1234 to something harder to guess
Well, there's spam egg sausage and spam, that's not got much spam in it.
What gets me is that you can tell the white hats and black hats are both lazy.
If the sysadmins had actually patched their servers with the appropriate security patches the "hackers" would have never gotten in, in the first place. If you read the counter measure section this isn't anything new that they shouldn't be doing every day and enforcing.
If you look at the section entitled Evidence of compromise you can see that the people breaking into the systems are leaving a pretty big trail to follow. In my job, when customers start complaining that their servers are working quite right, when you take a look at whats going on you can see a root kits been installed. The whole idea of a root kit is to cover your tracks. If these guys did a better job you'd never know you were hacked. Its quite sad really. Laziness is the biggest security problem if you ask me.
Servers were down much of last week. The ITS website has a few brief details.
I'm running a live cd distro based on Damn Small Linux. Is this the coming thing to prevent attacks and viruses from getting anywhere?
Nothing is written to a hard drive with this OS.
If so, how would this apply to the story on these attacks? How would anyone "gain control" of my computer under these circumstances.
BTW, Damn Small has a limit of 50 Mb, mine runs a little over 60 MB, and I put Mozilla Firefox and Wvdial in the remaster, as well as some office applications from the Debian list of over 8000 items.
The entire (up to date) Windows lab here gets compromised & backdoored to hell and everyone just says "Have it working by tommorrow". A Linux cluster gets compromised and they issue a press-conference.
my sig's at the bottom of the page.
Isn't that an oxymoron? Cray Canada's CTO says so. Then again, Borland's CTO said "OS X is my favorite Linux distribution.", so maybe CTOs aren't so smart about Technology after all ;)
Honey, I shrunk the Cygwin
I dont think we will ever have a fully secure box, these vulnerabilies will continue to pop up occassionally and there's nothing we (the developers) can do about that. It is just a testimony of the fact that we are imperfect beings and sooner or later we will have our errors exposed. It is not a bad thing, in the evolutionary way of dealing things, this (finding and sorting out bugs) could probably be a good thing. Having said that, I think developers do have control over how they respond to these problems, like coming up a problem that doesn't just band-aid the wound hoping to find a cure for in the future. Also developers have control over how fast they respond. On both criterias, open source peer reviewing is winner over closed sourced development. One tends to promote security through openness and and in the other security through obscurity like think MSFT( Read comments from a MSFT bigwig who said the only reason MSFT servers are compromised because the vulnerabilities are announced).
Activists United
If you believe your Unix computer has been affected by these intrusions, please contact the Information Security Services office (650-723-2911 or security@stanford.edu). Please include the name or IP address of the affected machine, as well as any compromised userIDs.
Never mind the compromised machines. Let's try social engineering instead. I know! We'll make a security alert, get it on Slashdot, and the poor trusting souls will beat a path to our POP3 account!
Seriously, you might as well just hand them your hard drive and credit card number.
You know he's at it again!
could someone more familiar with HPC systems please explain to me why any cluster is attached to the internet? I'm assuming these are externally routable addresses. I just dont understand why you would do this.
If more sysadmins installed this, perhaps we wouldn't have problems with so many Linux compromises? Of course it's no substitute for patching, but seems like a good additional security measure.
This is from the gnu.org software directory
The exploitation of buffer overflow and format string vulnerabilities in process stacks are a significant portion of security attacks. 'libsafe' is based on a middleware software layer that intercepts all function calls made to library functions known to be vulnerable. A substitute version of the corresponding function implements the original function in a way that ensures that any buffer overflows are contained within the current stack frame, which prevents attackers from overwriting the return address and hijacking the control flow of a running program.
The true benefit of using libsafe is protection against future attacks on programs not yet known to be vulnerable. The performance overhead of libsafe is negligible, it does not require changes to the OS, it works with existing binary programs, and it does not need access to the source code of defective programs, or recompilation or off-line processing of binaries.
From the Stanford article:
And further down...
To paraphrase a cliche without any attempt at humor:
Imagine a Beowulf cluster running John the Ripper.
A marriage is always made up of two people who are prepared to swear that only the other one snores.
Now, my opinion of MS is not that great, but this just seems wrong.
Regards,
John
Falling You - beautiful
As long as we are being consistent. If unpatched Windows boxes count when complaining about or keeping statistics on compromised systems then unpatched Linux boxes should count as well. Personally I believe Windows' perceived insecurity has more to do with poor administration than technical shortcomings, well at least with the NT family. Linux's intimidation of traditional PC users may work to Linux's benefit here, fewer PHB think they can have an "amateur" administer the Linux box as they believe they can do with the Windows box. If Linux becomes less intimidating we may find more "amateurs" administering them and find them about as vulnerable as the average Windows box. On the other hand, Mac OS X is an excellent example of what Linux could do if it ever gets over its "by geeks for geeks" attitude.
Funny, the same argument is also heard when a new worm attacks an age-old-there's-a-patch-for-it Windows exploit.
Of course, most Windows users are clueless, so the Linux/Unix admins are pretty much guilty in this situation.
To confess (anonymously), where I work we are pretty slack about security as well.. we use ssh and pam, wasn't there a known security risk with these 2 a few months ago?
Yes, I use this.
I also combine it with grsecurity, which adds even more protection.
You should always remember though, these are just added layers of security. If someone can sniff your root password you're still cactus.
I can see why they would want to target academic boxen if they wanted high-powered computers to do some serious slaved number crunching. If they are just going to launch a DDoS attack or send a bunch of spam though, academic computers are not the best. Most academic sysadmins have fairly limited budgets, and spend a fair amount on bandwidth. As such, they rule their bandwidth with an iron fist in many cases. The Admins at my particular college have bandwidth flags on certain ports and a global flag of somewhere around 1gb/day over 3 days. Break that, and the admin gets very interested in what you are doing with your boxen.
I'm sure other colleges have similar schemes, and I've heard of many colleges which are even more strict with their bandwith (200mb/day limit, etc). These academic boxes may make good targets because of their relatively user intervention and user experience, but they don't have that great of a pipe on them, relatively speaking. If it was me, I would have gone after servers that also run wireless access points. Hard to tell where the bandwidth goes in some cases with those.
Even those who arrange and design shrubberies are under considerable economic stress at this period in history.
...Is that you cannot make sure your users are careful.
You pretty much have to assume that black-hats are going to be able to runs escalation exploits and work accordingly. That or severely limit how users are allowed to interact with the machine (if they only need to access email or upload files, WTF should they be able to run anything else?).
But yeah, good passwords limit the opportunities.
Xix.
"Everything is adjustable, provided you have the right tools"
I still use libsafe. It is the greatest thing since sliced bread. Ok, that and distcc. Distcc and rsync... and ssh... DOH!
"Learning is not compulsory... neither is survival."
--Dr.W.Edwards Deming
Every single god damn worm would not work if users would patch their god damn systems. That's not news. Tell me something new to support that "Linux is secure" myth.
PC's get compromised if security patches are not applied!
and in other news...
cheerio's get soggy in milk
Troll, Troll, go away and flame again some other day
I was looking at one of the Solaris vulnerabilities, and I saw "sadmind".
I thought it was some kind of nasty name for a hacking daemon - until I found out that sadmind was the "Solaris ADMIN Daemon"
READY.
PRINT ""+-0
Straw, meet man.
It is important that when we wave our flags and cheer when Microsoft is laid low by the latest security flaw that we not close our eyes to the very real vulnerabilities in the Unix/Linux system.
No one is. Work is always being done to find and fix vulnerabilities in *nix variants.
No OS can be fully secured
No one with a brain ever claimed that was the case.
Assuming that Unix/Linux is invulnerable to security holes is deadly.
See last comment.
Though the OS may have more security features and "more eyes" on the code than closed source operating systems
Which is true...
we must not rest on our laurels watching Windows implode while our own house is burning.
Last time, NO ONE IS.
Geez. I know your nick is "Obvious Guy", and that's pretty much all you're saying. Well, except for the entire argument about "watching Windows implode while we rest on our laurels", which no one is doing, talking about doing, nor thinking about doing.
Straw, meet man. I'm still befuddled as to the upwards moderation you consistently get, however.
Endless arguments over trivial contradictions in books written by ignorant savages to explain thunder in the dark.
I see a day coming when, in one day, half the computers in the US have their disks erased.
Washington Post has more coverage in this article, Hackers Strike Advanced Computing Networks.
If the Government becomes a lawbreaker, it breeds contempt for law;
Heh, I'm running Windows 95. I figure by now the hackers are just bored of hacking me.
Security through boredom, my new secret weapon take th^454&*%2^$^^^B
Your CPU is not doing anything else, at least do something.
... at any of these places where the attacks are occuring have any other information to add? I am interested if there is information that might have been gleaned from any captured code that might indicate the exact identity of whom the attackers were going to DDoS once they had their zombied supercomputers. Or was it going to be a DDoS? Another exploit? I think that info might be a clue (well obviously) to who is behind this. One would think that attempting to zombiefy a super computer run by some advanced admins would be more difficult (and thus more unlikley to be used for such a mundane cause) that just gathering-say- dsl connected joe user boxes. Wouldn't you think they might be up to something else? Such as using these supercomputers in an attempt to crack even larger and perhaps more .. sensitive... supercomputers or facilities elsewhere? A two steps removed compromise in other words, a "force multiplier" effort, perhaps "masked" to the ultimate target by seemingly being a benign connection from a respected place, if you follow? Or better, is there a critical tactical penetration advantage in using a zombied super computer on a big pipe that goes beyond the obvious that is already stated/speculated on in the disclosure?
Or do you (anyone who might have some more AC insider info) have any other pertinent data not covered in the articles?
Not a security guru here, but last time I remember anything like this was like around 2 years ago or so when banks were targeted, something like that anyway.
Every day we see the constant stream of Microsoft security failures.
And those aren't minor, obscure failures. They affect millions of Windows users. They fill up our our reject logs. And they don't require special conditions -- Windows exploits can hit you simply because you browsed a webpage, played an MP3, received an e-mail, or just by having your PC connected to the Internet.
In fact, not only was there a story about three new Windows vulnerabilities, just two stories before this one, but Windows vulnerabilities set an all time record in February for the number of new exploits in a month. According to The Washington Times, "Internet attacks in February caused an estimated $68 billion to $83 billion in damages worldwide."
And to counter the impression that Windows has bad security, we are presented with... wait for it... a single Linux site, whose faulty administration procedures have left their machines vulnerable to local exploits, requiring the cracker(s) to first sniff a password.
And then the parent poster suggests that the two are somehow equivalent???
How lame!!!
No, it doesn't... many of the same types of reports about windows attacks are ALSO due to UNPATCHED machines.
It's the one-eyed, severely slanted nature of the Slashdot readership that:
* Microsoft is evil, stupid, moronic, evil, nasty, unsafe, did I mention evil?
* Linux is the shining non-denominational grail.
For god sake, there are security vulnerabilities in both people... and they aren't taken advantage of within the *nix world, because... hey, guess what? The majority of users are computer savvy, and know about passwords and firewalls and not leaving ports open etc.
Windows users on the whole have issues programming their VCRs.
As you start to get what you want, which is widespread Linux adoption, you'll start getting more of the VCR no-hopers using Linux, not patching it, not having secure passwords... and GUESS WHAT? Linux will start having major security issues in the same way as Windows does now... not as severe most likely due to better design, but they'll be widespread... there'll be a doozy, and it'll cause all sorts of problems and then people will be "Hey, I thought when we all moved to Linux the world would be a safer place for me and my little children, but now that a vulnerability has allowed my Linux box to be used as a Spam mail distribution point, I feel dirty and scared. I might install XP again."
Stop being so damn one sided.
Don't use passwords *at all*.
Wrong! Use tokens *and passwords* !
Using just tokens opens your users to a wide range of physical attacks... especially if they're college students with roomates who can "borrow" things for a few minutes of infringement.
I wonder if Debian supports any of those systems yet?
Yes. RSA SecureIDs can be used with almost any computer system. (They are a combined physical-token + password solution, and have better hardware compatiblity than a usb-key, as the user reads an LCD screen on the card to see a passkey that expires every 60 seconds)
Here is a list of some things that I feel are worth considering:
:). Next configure swatch to alert you upon recieving such messages! Of course you can always use perl or even grep -v to parse logs, but for repeated use I think a specialised tool would save you some trouble in the long run.
1. Patch your system! As soon as a patch comes out, get it applied and reboot if you have to! Also, stay up to date on security issues by subscribing to mailing lists that are related to the software your using. One good general purpose site is cert.org. Keep in mind that while mailing lists are great ways of being notified, they arent fool proof. If your subscription expires and you dont know about it, you wont be exactly up to date in the community now will you?
2. Use grsecurity. This is a kernel patch that is briefly lagged behind official Linux kernel versions. It has many great features for protecting against stack attacks/buffer overflows. ie: Those latest greatest scripts your local script kiddie just downloaded wont likely do anything against you since special addresses are randomised. It can also hide files on your computer such as intergrity checkers so nobody except you know they exist. Plus it can stop insert code into a running kernel by making kernel memory readonly (which btw, would have prevented at least one of the attacks they mentioned).
3. Install a filesystem intergrity checker. Aide, integrit and tripwire all come to mind and essentially all do the same thing but with different config file syntax. Besides, how can you tell if a file is changed if you dont actually check? Also, dont forget to hide the existence of this program using something like grsec's gradm filesystem ACL util and be careful of automating checks in the crontab!
4. Read a good linux securing article. One such article I have read is called Securing & Optimizing Linux: The Ultimate Solution. It will teach you how to lock a system down a fair bit and how to remove unused/unneeded services from your computer.
5. Watch those logs! Log files provide a wealth of information, but administrators rarely check them (well, not all). If you dont know what a log entry means, research it, or else you may be looking at an attack and not even realise it. Now I know some of you are thinking I am nuts considering just how many logs even a small system generates, but there are tools to help you. One way is to use a program called swatch (a perl script). It can parse existing and old archived log files using a perl regex syntax and trigger actions based on found text. Start by configuring the system to ignore any log entries that are known to be friendly and show you everything. Then slowly eliminate each friendly entry one at a time. What will be left is a list of purely evil enteries
Now I know I could go on forever with suggestions, but I think that these few things should give anyone a kick in the right direction. I hope this has been helpful.
Well theres 2 sides to that coin. Some say its really bad to rely on libsafe because the underlying source never gets fixed, therefore libsafe becomes and indispensible middlelayer you rely on more and more to protect legacy code which is inelegant. So in the long run much better to sort out the original source and do the job properly from the top. Just another 0.2c from a different school of thought.
The p;roblem, among others, is that we don't have enough real punishment going on for hacking activities.
The problem is that the concentration of clue among sysadmins is just too low. If you are still running a do_brk vulnerable kernel 5 months after the vulnerability was discovered and patched and widely publicised (remember the Debian and Gentoo server compromises that were all over the news?), you deserve whatever you get. I mean, sure, if you were hacked on December 5, my sympathy goes out to you, but if you are running unpatched 2.4.22 right now, there is no excuse.
As for jail time for hackers: to justify that, you would need to show that a moderately skilled sysadmin, one that reads a security-related news source at least on a quarterly basis, physically cannot protect his/her system from a moderately skilled attacker. For example, suppose someone proved P=NP and made a polynomial-time ssh decryptor. Only then we would need laws against password sniffing, because once you let a government have a taste of regulating the Internet, it will not stop until it has, so to speak, filled its belly with electronic freedoms.
I've heard a lot of people say something like, "It's their own fault for not installing the latest patches." Doesn't that suck anyway though? It's a major pain to need to keep a human around to twiddle some bits periodically.
I'm not sure it really has to be this way. It seems to me, that it is a major design flaw that if there is a small error in one of the *many* programs from *many* different parties being run as root, that it can be exploited so that an arbitrary attacker can end up getting root access or executing arbitrary code or whatever. For that matter, it seems silly that (for desktop systems) disastrous effects can come from code run by Joe user. After all, desktop users store all their important files in some place they *don't* have to authenticate as root to get to.
Rather than just assuming that the ever watchful eyes of open source uber hackers are the only remedy for this as well as all of life's problems, maybe it is possible to come up with some easy solutions, or at least partial solutions, to this problem?
1. Use software that watches the beginning and end of every stack frame for an overflow. If an app overflows *kill it dead*. Similarly, the beginning and end of every block allocated on the heap can be watched. Software like this exists, and it is about time it is built directly into the standard distributions and *turned on by default*.
2. Develop a new security model. The current system sucks out loud. Really, access lists (a la microsoft) are a step in the right direction. Finer grained and more flexible controls are good, but a totally new security model would be better. I've seen some things like this developed as academic projects, but it would be nice to see a patch available for a main stream OS like linux.
3. It might also be useful to have virtualization (think VMWare) built into standard distros and used by default for services like apache that need to run some stuff as root. My understanding is that you can do something like this with chroot currently, but that it is a clumsy and dangerous tool.
I'm not a big security buff, but even I can see that there are some things we can actually *do* about this problem.
The thing to remember with cd-based distros is that, even though the media cannot be changed, many things that are stored in writable memory can be, up to and including the system BIOS. It's a good idea to reboot them periodically to verify that you're working with a "clean" OS and that any intrusions or modifications have been reverted.
"I assumed blithely that there were no elves out there in the darkness"
So if I put -fstack-protector in my global CFLAGS, I can ignore all the critical buffer overflow exploit warnings? Why isn't it on by default?
I mod down anyone who says "I will be modded down for this", regardless of the rest of their comment
we have a sun system at our institution that runs a webserver for a very specific application. an unnamed vendor (we'll keep it that way) installed this machine and pretty much told us to keep hands-off of it except to change the backup tape. if we made any modifications to the machine or its software, then our service agreement was void and they would not support this particular app. so, we firewall the crap out of this thing, only allowing access to httpd (apache), making sure to explictly block any high port in use. well, this machine gets compromised about a week ago because this vendor has an ancient version of apache (1.3.3 or something) running suid/sgid root. idiots.......this is a problem we could have prevented if our vendor wasnt as dumb as they were. being a small .edu, we cant just pack up and change without spending 6 figures, so we are pretty much stuck with it until their contract comes up in a couple of years (this is an inherited problem). want their take on the problem - apache only will work suid/sgid. wont run unless permissions are that way. so i ask them to change it, and after about 10 minutes of arguing with their lead UNIX guy he does so. he was amazed that it would run......