Slashdot Mirror

← Back to Stories (view on slashdot.org)

Hackers Compromised Yahoo Servers Using Shellshock Bug

Posted by samzenpus on from the protect-ya-neck dept.

wiredmikey writes Hackers were able to break into some of Yahoo's servers by exploiting the recently disclosed Shellshock bug over the past few weeks. This may be the first confirmed case of a major company being hit with attacks exploiting the vulnerability in bash. Contacted by SecurityWeek, a Yahoo spokesperson provided the following statement Monday afternoon: "A security flaw, called Shellshock, that could expose vulnerabilities in many web servers was identified on September 24. As soon as we became aware of the issue, we began patching our systems and have been closely monitoring our network. Last night, we isolated a handful of our impacted servers and at this time we have no evidence of a compromise to user data. We're focused on providing the most secure experience possible for our users worldwide and are continuously working to protect our users' data."

50 of 69 comments (clear)

  1. Can someone explain... by TWX · · Score: 1

    ...the process from poking unusual commands at Apache or another web daemon to how that allows control of the box?

    When I ran web servers I ran the daemons as unprivileged accounts that had no shell, and in a couple of instances there was chroot sandboxing to further help to mitigate penetration even if someone managed to exploit a vulnerability in the web daemon.

    How is this working? Are people not folliowing good practices?

    --
    Do not look into laser with remaining eye.
    1. Re:Can someone explain... by Anonymous Coward · · Score: 3, Informative

      Chroot is not a security tool.

      Running as a no-shell account doesn't help when your process invokes a shell directly.

      Local privilege escalation vulnerabilities are rampant.

    2. Re:Can someone explain... by Anonymous Coward · · Score: 3, Informative

      No, people are not following best practices. Bash is a DISASTER for tight security, and shouldn't even be on a box facing the Internet.

      Look at the code used by John Hall (http://www.futuresouth.us/yahoo_hacked.html). It is Bash itself that is pinging back to Mr. Hall's box. No other executables were required for this. I wouldn't put it past someone to make an evil client-server system with nothing but Bash scripts sent over in headers.

    3. Re:Can someone explain... by Megane · · Score: 4, Informative

      The primary method is to send a browser user agent string that starts with "() { : ; } ; " and try to run a stupid (as in stupid people never remove them out of default installs) CGI script. Then when the shell gets invoked (either for a shell script CGI, or a dumbass system() call from another language CGI), the bug causes bash to execute whatever is on the end of the user agent string, before doing anything else. This is because the cgi-bin module takes all the various parameters of the HTTP request and sticks them into environment variables, and the bug executes environment variables before doing what it's been called up to do.

      The easiest thing to do whether or not you can get a patched bash yet is to disable Apache's cgi-bin module.

      --
      #naabhaprzrag, #sverubfr-000, #agi-fcbafberq, negvpyr[pynff*=' negvpyr-ary-'] { qvfcynl: abar !vzcbegnag; }
    4. Re:Can someone explain... by tlhIngan · · Score: 4, Informative

      ...the process from poking unusual commands at Apache or another web daemon to how that allows control of the box?

        When I ran web servers I ran the daemons as unprivileged accounts that had no shell, and in a couple of instances there was chroot sandboxing to further help to mitigate penetration even if someone managed to exploit a vulnerability in the web daemon.

        How is this working? Are people not folliowing good practices?

      First off, let's clear the air - the apache or whatever HTTPd is being used is still running unpriviledged.

      Second, what hackers are doing is exploiting CGI - because CGI data is often passed as environment variables, people are setting their User-Agent and Referer headers to exploit this: "() { :; }; command" being common (with command being ping a certain address).

      What happens is the CGI gateway takes those, then sets environment variables like "REFERER=() { :; }; ping ..." and "USER_AGENT=...", then calls system() to run the appropriate CGI script.

      system() calls fork() and starts /bin/sh -c "command" and therein is the problem. Because /bin/sh almost always is /bin/bash on Linux, that means bash starts up, and it starts looking at the environment variables to set up the environment. It runs across REFERER and USER_AGENT, and sees that it's how bash passes exported shell functions to subshells, so it creates those functions. Unfortunately, the bug means that when those functions are defined, bash continues parsing - it sees the semicolon and the parses it as a regular command while it's still parsing environment variables.

      This lets bash execute anything as the afflicted user.

      So what can you do? Everything httpd can do - which may include accessing databases, or loading in other scripts and then getting those to run to get at databases, or dumping the server files.

      You may not be able to write /etc/passwd, but you can certainly try to dump the user database for the web application.

      It's a pretty deep bug because it's a design bug - POSIX doesn't specify how exported functions are passed from shell to subshell, so bash uses environment variables in a special format. One patch to fix it makes it so all functions must be declared as _BASH_FUNC_name= which helps limit exposure. There are going to be many other patches because fixing this is hard.

    5. Re:Can someone explain... by Cramer · · Score: 1

      It isn't a matter of "subshell" -- which gets a copy of the parent process's memory from fork() -- but how functions are exported for use by future shells. For example, your shell (bash) runs less or vi (or python or php...), and from there a shell is started. That's not a subshell, it's a new process; it wasn't created by fork() so it doesn't have a copy of the original bash memory. The logical question is why anyone would need that functionality? In general, no one does, but it does offer some optimization for complex shell environments (read: the idiotic bash_completion BS. That's why bash takes an eon and a half to load.) Yes, my first response was "turn that crap off; nobody uses it anyway."

      The "namespace" patch does absolutely NOTHING to address this bug, or security in general. So I now have to name my variable BASH_FUNC_foo() (or some variant.) If inputs aren't validated, the system is still vulnerable.

    6. Re:Can someone explain... by tlhIngan · · Score: 2

      It isn't a matter of "subshell" -- which gets a copy of the parent process's memory from fork() -- but how functions are exported for use by future shells. For example, your shell (bash) runs less or vi (or python or php...), and from there a shell is started. That's not a subshell, it's a new process; it wasn't created by fork() so it doesn't have a copy of the original bash memory. The logical question is why anyone would need that functionality? In general, no one does, but it does offer some optimization for complex shell environments (read: the idiotic bash_completion BS. That's why bash takes an eon and a half to load.) Yes, my first response was "turn that crap off; nobody uses it anyway."

      The "namespace" patch does absolutely NOTHING to address this bug, or security in general. So I now have to name my variable BASH_FUNC_foo() (or some variant.) If inputs aren't validated, the system is still vulnerable.

      No, the bug IS related to subshells. Environment variables are "exported" to make them available to subshells to use. POSIX defines the mechanism (and the environment is passed in as parameteres to the exec() family of functions)

      Now, bash has a design flaw in how it exports FUNCTIONS to subshells, namely if it starts with () { then bash treats it as a function declaration.

      Of course, the problem is not related to subshells - but it stems from there. Because if you invoke bash, and pass it a function declaration like that, bash will think it's inheriting a function (which could've been started many, many processes ago because exported functions in bash look like exported environment variables).

      So you can do the bash-->httpd-->bash thing and if you export a function in the partent bash, even though you're going through httpd, the sub-bash (probably started as CGI) will see the parent exported function.

      And CGI typically calls system() to perform the script execution, which by definition launches the system shell to run the command you passed to system().

      The bug originates from a design flaw in POSIX that fails to specify how to securely pass in exported functions, so the bash devs made up some method. Shellshock basically relies on this feature.

      The best way, of course, is to disable exporting of functions, but there may be many reasons why this may not be possible - including entrenched software that relies on this behavior and has for many years (it's a design flaw dating back over 20 years). And one should also note that 20-30 years ago, Unix security was practically non-existent.

      Things like this were a product of a different era.

    7. Re:Can someone explain... by ThePhilips · · Score: 2

      Because /bin/sh almost always is /bin/bash on Linux [...]

      On most systems - number-wise - which are Debian-based - it is not.

      But on the RHEL, the golden boy of corporate uni-culture, it is. The UNIX reborn.

      Let that be a moment of the appreciation for all the work Debian and Ubuntu people do behind the scenes. (Yes, Ubuntu too: the transition to dash was actively supported by Canonical since they wanted Ubuntu to boot faster and Debian folks were in agreement with the goal.)

      P.S. Why RHEL hasn't picked up all the work Debian/Ubuntu has done for them? The usual - NIH - I suspect.

      --
      All hope abandon ye who enter here.
    8. Re:Can someone explain... by defaria · · Score: 1

      This lets bash execute anything as the afflicted user.

      Yeah and who exactly is this afflicted user? Right, normally apache or some other unprivileged user who has relatively little power though granted you don't even want unprivileged users logging in from the Internet

      So what can you do? Everything httpd can do - which may include accessing databases, or loading in other scripts and then getting those to run to get at databases, or dumping the server files.

      You may not be able to write /etc/passwd, but you can certainly try to dump the user database for the web application.

      Like suing in America, you could try. The question is will you succeed. IN the case of most DBMS's you need to log in to access anything. The unprivileged apache user without the knowledge of the username and password to access the database that contains the list of users is still a rather large huddle

      It's a pretty deep bug because it's a design bug - POSIX doesn't specify how exported functions are passed from shell to subshell, so bash uses environment variables in a special format. One patch to fix it makes it so all functions must be declared as _BASH_FUNC_name= which helps limit exposure. There are going to be many other patches because fixing this is hard.

      Note really. Just tell bash "hey, after the function definition is completed stop parsing (unless perhaps it's yet another function - which I don't think is allowed anyway).

    9. Re:Can someone explain... by benjymouse · · Score: 1

      This lets bash execute anything as the afflicted user.

      Yeah and who exactly is this afflicted user? Right, normally apache or some other unprivileged user who has relatively little power though granted you don't even want unprivileged users logging in from the Internet

      You are one setuid/SUID utility away from total system compromise. Even one such utility that invokes bash (or the default shell which is bash on Fedora and RH systems) and your box isn't yours any more.

      Shellshock is *also* a privilege escalation vulnerability when exploited locally. Granted, you need to find such a setuid utility. But the utility does not need to be vulnerable by itself. It just needs to invoke the shell through system() or similar.

      --
      Reading slashdot one-liner: (irm http://rss.slashdot.org/Slashdot/slashdot).rdf.item | fl title,desc*
    10. Re:Can someone explain... by ray-auch · · Score: 1

      The easiest thing to do whether or not you can get a patched bash yet is to disable Apache's cgi-bin module.

      and qmail procmail exim postfix sshd openVPN some inetds... and all the vectors that haven't been found yet.

      The easiest thing to do is to fix bash - using rm if necessary

    11. Re:Can someone explain... by ray-auch · · Score: 1

      Yeah and who exactly is this afflicted user? Right, normally apache or some other unprivileged user who has relatively little power though granted you don't even want unprivileged users logging in from the Internet

      For some, the ability to run their own code on a server with high bandwidth outward connection to the internet is all the power they need/want.
      If the server is an authorised mail source for a domain (e.g. spf) then so much the better.
      If the user has access to some writable disk space that can be used to host some interesting files, then there are uses for that.
      If the user can read the web site source or config files that may have value in itself or may lead to further penetration - but I'm sure you've never seen DB user/pw in the source for a production website, ever ?

  2. I can believe it... by DarthVain · · Score: 2

    I have gotten spam mail from myself several times the last few weeks (From Yahoo to Gmail), and have gone into the stupid yahoo site to change passwords several times. They were obviously compromised as hours after changing passwords, I would get more spam. Little point to changing passwords if they have total access to them. Might be time to finally drop them.

    They were all a fictional job offer, that I guess I was going to give to myself for big bucks... Sounds like something I would do! :)

    1. Re:I can believe it... by dunkindave · · Score: 1

      Did you check the email headers? On multiple occasions I have received emails showing my email address as the From, but the email headers showed the email originated from machines in foreign countries. Spoofing the From part of an email is trivial. This is a common technique by spammers to avoid spam filters since the account's own address is normally considered trusted. Now if the header says the email really did originate from Yahoo or Gmail, then that is a different matter, but again read the headers closely since many of those fields/lines can still be forged.

    2. Re:I can believe it... by ADRA · · Score: 1

      Maybe if you bothered to look at the relay logs in the header, you'd probably know they were sent from a fictional server pretending to be you (or totally legit from an exploit), but since said note was absent in your comment, I'm assuming you didn't know about spoofing email, which any script kiddie could do trivially.

      Secondly, I can't say about Yahoo, but google has 2 factor auth to avoid said problems. Third, Google has account access history so that if 'bad people' were logged into the account, you'd at least be able to view a record of who/when. I'd be very surprised if Yahoo didn't offer a similar example.

      --
      Bye!
    3. Re:I can believe it... by DarthVain · · Score: 2

      No didn't bother, very well could be that. I use that as my throw away account. However that said, spoofing would be random. One would have to access my Yahoo address list to get my Gmail account. So unless spoofers used my Yahoo account and sent it around the world, it would be pretty coincidental that using my Yahoo address would hit my Gmail address several times in the last couple of weeks, and as I said, within hours of me changing passwords. Otherwise how would they know to use one against the other. Unless they were specifically targeting me, and I don't think I am that special! :)

      I think it is more likely that Yahoo is compromised (or was) and in that time they were scripting spam using personal address books (sending from my account to everyone in my address book) over and over again. If they have access to the passwords, then logging in and repeating would be trivial, even when changes are made... until of course Yahoo patches their damn servers so that they cannot anymore.

    4. Re:I can believe it... by wcrowe · · Score: 1

      I've seen the same thing, but as a couple of responders said, the emails are not actually coming from my account, but are coming from somewhere else. You're probably changing your passwords unnecessarily.

      --
      Proverbs 21:19
    5. Re:I can believe it... by dunkindave · · Score: 1

      No, not random. Today malware will commonly harvest a person's address book (among many things to exploit what it can get off a person's machine), and once the address book has been harvested, sold to spammers. The spammers send emails to people in the address book with the email pretending to be from another person in the address book. The theory is that if both addresses are in a person's address book then there is a good chance they know each other, or they will have received legitimate email from that address before, both with the intention of getting around the spam filter and getting the victim to open the email. Note that the spoofed source email address isn't normally the email of the person whose machine has the malware, but rather others found in the contacts list. All these spam emails mean is that there are people out there whose address book, including "collected" addresses, contain both of your email addresses, and one of those people got infected with malware.

    6. Re:I can believe it... by CaptainDork · · Score: 1

      Hell, I have been trying to get me to buy wAtche$ for years.

      Last time I looked at the headers, it was from Romania.

      --
      It little behooves the best of us to comment on the rest of us.
  3. Re:Color me shellshocked! by TWX · · Score: 3, Funny

    You're not a very good sysadmin, you'd know that you'll never see "fixed!" becuase the reboot will terminate the rest of the command before it can run.

    --
    Do not look into laser with remaining eye.
  4. Re:How Hard? by Anonymous Coward · · Score: 1

    If somebody 30 years ago would have created a sentence like that one, they would have received nothing but puzzled stares.

  5. Re:How Hard? by Anrego · · Score: 1

    In their defense, it's more like "update bash over and over again".

  6. Re:Color me shellshocked! by Anonymous Coward · · Score: 1

    Most likely the reboot is actually unnecessary as well.

  7. Yahoo Sports by Slider451 · · Score: 3, Funny

    I'm going to blame this for my fantasy football loss this week... and all previous weeks.

    --
    Nostalgia isn't what it used to be.
    1. Re:Yahoo Sports by rockabilly · · Score: 1

      I'm going to blame Obama since I cannot pass blame on the CNN Discus boards anymore.

  8. Yahoo still has servers? by Kenja · · Score: 1, Informative

    Huh... I would'a guessed they were long gone by now.

    --

    "Have you ever thought about just turning off the TV, sitting down with your kids, and hitting them?"
  9. FTFY by Archangel+Michael · · Score: 1

    Last night, we isolated a handful of our impacted servers and at this time we have no evidence of a compromise to user data ... that we want to admit to

    FTFY

    --
    Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
    1. Re:FTFY by SuiteSisterMary · · Score: 1

      Also, if they isolated a handful of their impacted servers, they left at least the same amount of impacted servers untouched.....

      --
      Vintage computer games and RPG books available. Email me if you're interested.
  10. OMG! by __aaclcg7560 · · Score: 1

    I better change my 15-year-old account password. :/

  11. Comment removed by account_deleted · · Score: 1

    Comment removed based on user account deletion

  12. Baaaa! by Delicious+Pun · · Score: 5, Insightful

    We're focused on providing the most secure experience possible for our users worldwide and are continuously working to protect our users' data."

    Who else is sick of reading this sentence and its variants from faceless corporate entities? In my mind it translates to "Be calm, sheep. Be calm."

    1. Re:Baaaa! by dunkindave · · Score: 4, Interesting

      No, the real problem is this is the same response you would get from a company no matter what happened so it is meaningless. You screwed up but don't want to admit it? Say you are committed to security and it was a fluke. It really was a one time fluke by someone exploiting a near-zero-day? Say you are committed to security and it was a fluke. You deliberately sold out your customers and someone noticed their info was in the wild? Say you are committed to security and it was a fluke. Since it is always the same no matter what happened, what real use is the statement? Yes, I know it is to persuade those who don't know better.

  13. Re:Color me shellshocked! by Vlad_the_Inhaler · · Score: 2

    It is.
    I ran a sanitised version of the initial exploit in a virtual Konsole, updated and ran it again in a new Konsole. The second time the attempted exploit was rejected, no reboot required.

    This was early last week, the day the update became available. What made these muppets wait until they were attacked? Do they have some cretinous system in place where even security-relevant updates have to be scheduled a week in advance?

    --
    Mielipiteet omiani - Opinions personal, facts suspect.
  14. Re:How Hard? by Vlad_the_Inhaler · · Score: 1

    If somebody 30 years ago would have created a sentence like that one, they would have received nothing but puzzled stares.

    Sounds like Yahoo's managers have thet problem as well.

    --
    Mielipiteet omiani - Opinions personal, facts suspect.
  15. Marketspeak by wcrowe · · Score: 4, Insightful

    "...We're focused on providing the most secure experience possible for our users worldwide and are continuously working to protect our users' data."

    Marketspeak. I guess communications majors are taught to always do this. The problem is, we've heard crap like this so much, we've become inured to it. Nowadays, the minute I see a sentence like this, I assume everything else the spokesperson has said is a complete fabrication.

    --
    Proverbs 21:19
  16. Consequences for treating employees like garbage by Anonymous Coward · · Score: 2, Interesting

    Yahoo treats their employees like garbage. To wit, Yahoo uses "stacked ratings" which results in backstabbing and good employees getting fired.

    When Yahoo's employees are spending all their time making sure someone else gets fired when the next employee reviews are made, instead of actually being able to do their job, it comes as no surprise that basic crap like updating security on servers falls by the wayside.

  17. Inside View by Anonymous Coward · · Score: 1

    Yahoo does, in fact, make extensive use of chroots jails, both on BSD and RHEL. Mainly, this is to isolate the platform software from the OS software from what I've seen -- but I'm not a security guru.

    There's also the human problem of churn that goes on in the valley. Many things Just Work (TM), guy who write it leaves, still works, and then a bug like this comes out with no one to ask "Doing this vulnerability affect X?" because the person working on X hasn't worked here for years. There may be some admin somewhere getting alerts when X breaks, but he/she may not have an extensive security background and may not know the server has been compromised.

    (Posted as AC from Yahoo!)

  18. Re:air travel is the big elephant by peter303 · · Score: 1

    I posted this under the following Recreation thread. Looks like another in a growing list of Slashdot bugs.

  19. Script Kiddies Compromised Yahoo Servers Using ... by Dishwasha · · Score: 1

    Script Kiddies Compromised Yahoo Servers Using Shellshock Bug

    There, fixed that for you.

  20. Re:Color me shellshocked! by mysidia · · Score: 1

    Dude..... you just hosed production. You're getting written up, and also, nobody is going home until everything is back up, and the Myisamchk --safe-recover on the 5.6 TB mission-critical can NEVER go down or we lose $50000 an hour database you just broke is finished, oh, and by the way, you'll be getting the bill.

  21. Millions by joh · · Score: 1

    There are millions of servers out there that have not been patched yet.

    To be more precise: There are uncounted servers out there that have a teeming population of parasites anyway.

    But Yahoo has always been and still is the most incompetent of the big players, every time they screw up I'm surprised they still are around, since I never hear from them in between. There's not even a Yahoo phone... Not even that!

    1. Re:Millions by jedidiah · · Score: 1

      Anytime anyone says that you should not bother running your own stuff and that you should just "outsource" it all to "the cloud" my response is: Yahoo.

      We've been hearing a lot about this bug and people trying to allegedly take advantage of it but who is the first entity to be confirmed to actually be bit by this but: Yahoo.

      --
      A Pirate and a Puritan look the same on a balance sheet.
  22. The Simpsons (spoilers from last night's episode) by antdude · · Score: 1

    At least Homer J(ay) Simpson can use this excuse after last night's episode. ;)

    --
    Ant(Dude) @ Quality Foraged Links (AQFL.net) & The Ant Farm (antfarm.ma.cx / antfarm.home.dhs.org).
  23. Yeah ... by CaptainDork · · Score: 1

    If the FBI has any sense they'll reply to his emails with a search warrant.
    Reply to This Parent Share

    ... that will fix the problem real good and stuff.

    --
    It little behooves the best of us to comment on the rest of us.
  24. Re:Color me shellshocked! by Nikker · · Score: 1

    A new console would have loaded the patched routines. The question of needing the reboot would depend on the first terminal.

    --
    A loop, by its nature, continues. If that didn't make sense, start reading this sentence again.
  25. Kinda curious... by koan · · Score: 1

    Why does Yahoo still exist?

    --
    "If any question why we died, Tell them because our fathers lied."
  26. Re:Script Kiddies Compromised Yahoo Servers Using by koan · · Score: 1

    Script kiddies? They prefer pre-coded youth.

    --
    "If any question why we died, Tell them because our fathers lied."
  27. Nothing compared to what's coming. by koan · · Score: 1

    http://krebsonsecurity.com/201...

    --
    "If any question why we died, Tell them because our fathers lied."
  28. Re:Color me shellshocked! by stiggle · · Score: 1

    So sys-admin only use Debian based systems?
    What about RHEL, AIX, HP-UX, Solaris, Tru64, VMS, Windows/Cygwin

    How about all that networking kit which use Bash for their scripting & WebUI

  29. CISO of Yahoo says "not Shellshock" by newfurniturey · · Score: 1

    Alex Stamos, the CISO of Yahoo, posted an in-response bulletin on Hacker News to clear up the rumor that this breach was caused by Shellshock.

    Straight to the point, he states that it was not Shellshock that the system was vulnerable to but a separate command-injection vulnerability in their log parsing scripts. Though... Shellshock itself is a command-injection / parsing vulnerability so I'm sure many will skip over the technicalities and consider them one-in-the-same.

    At first I was surprised that he came forward and gave explicit details that, well, can now be targeted against. On the other hand, I think it's pretty cool of them to be so open (either that, or they really didn't want to be the "large company" that was effected by Shellshock =P).