Slashdot Mirror

← Back to Stories (view on slashdot.org)

FireEye Tries to Bury Keynote Reporting That It Ran Apache As Root On Security Servers

Posted by timothy on from the doctor-streisand-on-line-one dept.

An anonymous reader writes: Leading network security company FireEye, which has customers in government and the Fortune 500 list, has caused a controversy at a London security conference today after its legal attempts to stop a keynote speech detailing the repair of major security loopholes in its customer-facing systems this year. Reported among these now-fixed vulnerabilities were the running of a significant number of FireEye's Apache-based security servers as 'root' — meaning that any attacker able to compromise the servers would have had absolute power over all its operations and commercial connections.

68 of 108 comments (clear)

  1. What? by Etherwalk · · Score: 5, Funny

    Why is 'root' in quotes? Why is it defined (poorly) as if it were this mysterious thing giving absolute power over "commercial" connections?

    We're not the general public. We're nerds. Don't submit articles written for people who don't know what "root" is.

    1. Re:What? by JustAnotherOldGuy · · Score: 2, Informative

      Why is 'root' in quotes? Why is it defined (poorly) as if it were this mysterious thing giving absolute power over "commercial" connections

      Well, as "regular users" and "technically oriented" people we may not require "definitions" but "no-technical people" (aka "ordinary end users") may require "things" be more "spelled out" so they "understand" that the word is a "technical term". heh

      --
      Just cruising through this digital world at 33 1/3 rpm...
    2. Re:What? by Anonymous Coward · · Score: 4, Funny

      I run all my security-sensitive services as the "streisand" user

    3. Re:What? by Anonymous Coward · · Score: 1

      I run all my security-sensitive services as the "streisand" user

      # id babbs
      uid=0(root) gid=0(root)
      #

    4. Re:What? by satch89450 · · Score: 3, Informative

      It's proper writing style to enclose text like user names and passwords in some sort of quotation mark in formal writing. I do it all the time in magazine articles, white papers, and technical documentation.

    5. Re:What? by Zontar+The+Mindless · · Score: 1

      It's actually more proper to use <literal> elements for these cases.

      Oh, you're not using DocBook?

      Too bad.

      Nevermind.

      --
      Il n'y a pas de Planet B.
    6. Re:What? by ArsonSmith · · Score: 1

      and really what's the point any more, so now you have root on some limited VM that only has access to the same connections you would have as any other user that apache may be running as. Ohh, but you could install drivers or some crap? Who cares, delete the vm fix the security leak in the config management and redeploy.

      --
      Paying taxes to buy civilization is like paying a hooker to buy love.
    7. Re:What? by davester666 · · Score: 2

      Awhile ago, slashdot let a bunch of people making web sites create logins here. Sure, they believe they are "developers", but you have to explain stuff real slow to them.

      --
      Sleep your way to a whiter smile...date a dentist!
    8. Re:What? by Etherwalk · · Score: 1

      Yes, it's certainly an acceptable style on its own, but combined with the fact that they were trying to *define* it, it became obvious that it was written badly for a non-technical audience.

    9. Re:What? by rtb61 · · Score: 1

      The point is "U.S. security company FireEyeâ(TM)s attempts to stifle any public disclosure of a major series of vulnerabilities in its suite", so the legal attempts to silence exsopire,obviously they thought it was really, really bad, otherwise why spend money for lawyers and "Felix Wilhelm, a security researcher for ERNW GmBH, made FireEye aware of the vulnerabilities five months ago, and reportedly worked with the company to help them resolve the issues successfully.".

      So basically be smart piss off https://www.fireeye.com/ and go with https://www.ernw.de/, after all it was who FireEye went to, so you might as well cut out the middle man (if you speak German of course and currently the secure choice is to avoid all things NSA 'er' American because if you are a filthy foreigner they do not have you security at heart, they are crazy feaks who want to probe you)

      --
      Chaos - everything, everywhere, everywhen
    10. Re:What? by satch89450 · · Score: 1

      I used to use italics for such things in magazines, until one of my editors set me straight on the proper style. (The copy desk would do the conversions silently.) I suppose it's a matter of where the material is to appear, and the style the publication wants to use.

    11. Re:What? by metrix007 · · Score: 1

      There is no such things as proper style. Just subjective standards pushed on people.

      --
      If you ignore ACs because they are anonymous - you're an idiot.
  2. Amusing coincidence... by bob_super · · Score: 3, Interesting

    I was just staring at Process Explorer, wondering why my company decided that the FireEye policy would allow it to max out one of my cores in the middle of the afternoon.

    1. Re:Amusing coincidence... by l0n3s0m3phr34k · · Score: 3, Funny

      FireEye has replaced nice with "angry". Every thread immediately grabs all the resources it can as soon as it's launched and refuses to give up anything until you reboot every device on the network.

  3. dumb fux by Anonymous Coward · · Score: 1

    Why, it is their intellectual property, it has to be protected. I suppose you could protect it in many different ways, they decided to rely on their lawyers to do it. Couldn't rely on their sysadmins to do it, quite obviously they haven't got any.

  4. What is really worrying ... by Alain+Williams · · Score: 5, Insightful

    is not that they were running Apache as root - although that it a stupid thing to do, it could have been an oversight (just about). What is of major concern is how they try to hide their mistake by abuse of legal system - this abuse is not an oversight and only makes me wonder what else FireEye is hiding -- I would think 3 times before hiring them.

    I am also disgusted at the German judge who gave an ex-parte order without having a return date so that the defendant (security researcher) could present his side of the argument. It does happen often in spite of heads of courts saying that it must not happen (in some UK court divisions anyway).

    1. Re:What is really worrying ... by Anonymous Coward · · Score: 3, Insightful

      No, the really worrying part is that a modern tech company actually believed a court order would stop the spread of information.

    2. Re:What is really worrying ... by tnk1 · · Score: 3, Insightful

      When does a "security company" not understand that you don't run a webserver as root? Just about every distro's webserver package will make a webserver run as a non-root user by default. These guys not only overlooked the fact that their webserver was running as root, they probably rolled their own web server install to begin with to even make that possible.

      As someone else pointed out, they must have used lawyers to protect their data, because they clearly didn't employ any system administrators.

    3. Re: What is really worrying ... by Anonymous Coward · · Score: 2, Insightful

      Shove the damn app into a docker container (kernel namespace) with read only storage. In this day and age, every application (even apps on your mobile phone) should be jailed in isolation. If someone manages to get "root" inside the jail, big deal, they can be king of the jail cell but not the entire prison.

    4. Re: What is really worrying ... by mlts · · Score: 3, Insightful

      SELinux is quite similar. Root might let them out of the cell, but they are not getting out of the cellblock. However, the ideal is definitely a docker container, just because it can run anywhere.

    5. Re:What is really worrying ... by spauldo · · Score: 2

      It's worse that that.

      I used to compile Apache myself (now I just use FreeBSD's port) and do all the setup manually.

      You have to intentionally set it to run as root. Every piece of documentation, including the sample config file, has the configuration set up to run as a user.

      The only way you could "accidentally" run it as root would be if you started with a blank config and only read part of the documentation. I have a hard time believing that anyone would actually do that.

      No, if they're running as root, they have a reason. I have no idea what that reason could possibly be, but there has to be one. Even VPs' nephews aren't stupid enough to make that mistake.

      --
      Those who can't do, teach. Those who can't teach either, do tech support.
    6. Re:What is really worrying ... by RabidReindeer · · Score: 2

      Precisely. Amenities like selinux and docker containers are all very well, but most distros these days install an apache or http userid and run Apache under that ID and ONLY if you deliberately switch it off will you EVER run apache as root.

      Something's rotten in the State of Denmark.

    7. Re:What is really worrying ... by RabidReindeer · · Score: 2

      Meh. Too much meth. Seriously hallucinating.

      Port 80 has been around a long time. 8080 got nominated for things like Tomcat which cannot chroot themselves.

    8. Re:What is really worrying ... by Enigma2175 · · Score: 1

      > connects to a port 1023,

      That is not what I wrote. I know it is less than port 1024. I wrote less than or equal to. It is sad this site is such shit that the people running it feel the need to corrupt what we post. I did not post that. /. is trying to make us look like idiots by misquoting us.

      /. comments allow HTML tags so it tries to interpret < and > as HTML. Your point is stupid in the first place, I wish ./ would have just deleted it. Pretty much any service that needs to open a privileged port opens the port and then executes the server under a service user rather than root. Apache comes configured like this by default on any Linux platform I have ever seen, they would have had to manually change the config to make it run as root.

      --

      Enigma

    9. Re:What is really worrying ... by lucm · · Score: 1

      "Preview" is for sissies. Cool people use "Cancel".

      --
      lucm, indeed.
    10. Re:What is really worrying ... by Zontar+The+Mindless · · Score: 1

      What's tragic is that, in this second decade of the 21st Century, there are still ignorami who don't know what entity references are for.

      --
      Il n'y a pas de Planet B.
    11. Re:What is really worrying ... by lucm · · Score: 1

      Your point is stupid in the first place, I wish ./ would have just deleted it.

      No, for that the syntax is ./rm, but only if your current working directory is /bin. Also depending on your distro you may have to add -f otherwise it won't just delete things, it will ask for a confirmation first.

      --
      lucm, indeed.
    12. Re: What is really worrying ... by Dutch+Gun · · Score: 1

      every application (even apps on your mobile phone) should be jailed in isolation.

      Modern phone OSes already work this way. Additionally, applications downloaded from the Apple OS X store or Microsoft's Universal Apps also use a stronger permissions system and sandboxed model, as far as I understand.

      I agree that docker containers are a good starting point, but keep in mind they're not the end-all, be-all of security. Remember, exploits have been found that allow applications to escape virtual machines, and we've seen plenty of other sandboxes breached, so it seems foolish to believe that no exploits will ever be found that allow malicious actors to break free of containers. Plus, don't forget that even if they haven't compromised the system, gaining all access to a container still may compromise private data.

      --
      Irony: Agile development has too much intertia to be abandoned now.
    13. Re:What is really worrying ... by JohnVanVliet · · Score: 1

      then you have not read the "linuxQuestions" forum
      the bleeped bleeps that do not even BOTHER to read and study the documentation and think a few mouse cklicks will install and CONFIGURE it

      i am in the group that ENCOURAGES that new to Apache people build the stack from source and manually install the parts about 12 times
      then use the package manager to save 30 min to 1 hour on install time

      --
      "I don't pitch OpenSUSE Linux to my friends, i let Microsoft do it for me
    14. Re: What is really worrying ... by ArsonSmith · · Score: 1

      Nobody has replied about how easy it is to get out of a docker container so they are insecure crappy software that can't run in enterprise.

      Of course it means that someone has to break your code AND break docker, no matter how easy docker is to break it's still harder than not using docker.

      --
      Paying taxes to buy civilization is like paying a hooker to buy love.
    15. Re: What is really worrying ... by cbhacking · · Score: 1

      There have been plenty of security holes with Docker. Many of them were (and are) just simple misconfigurations, such as you could make with any security model (but Docker definitely doesn't inherently safeguard you from them, though its defaults have gotten better). Some were bugs in Docker itself, though they've gone pretty well there. Some were Linux bugs nobody had looked for / cared about until people started trying to do things like restrict root to not *actually* be root.

      Don't get me wrong, the whole container idea is awesome, both in general and specifically as a security "sandbox on demand" deal. But Docker is not mature yet, and people who act as though it's a security panacea sufficient to render things like webservers running as root a minor concern... those people are part of the problem.

      --
      There's no place I could be, since I've found Serenity...
    16. Re:What is really worrying ... by gweihir · · Score: 1

      Well, the take-away is clearly to never ever buy Fire-Eye, as they will shamelessly lie about their incompetence. Of course, the same applies to most other vendors. Capitalism screws most people up that way.

      --
      Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
    17. Re:What is really worrying ... by gweihir · · Score: 1

      Was possibly outsourced somewhere where they have even less skill (because the skilled ones all left...) and then not really tested or looked at because that costs money. This is a sign of clear and present danger from all Fire-Eye products though, as they apparently do not even understand the basics.

      --
      Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
    18. Re:What is really worrying ... by TheRaven64 · · Score: 1
      There are several ways around this. The simplest is for the process to run setuid root, open the listening socket as the first thing, then setuid() to the correct user. That's not ideal, because it means config file parsing as root. The second option is to have a simple setuid root binary that opens the listening socket on behalf of the untrusted program and passes it back over a UNIX domain socket. This needs some careful design to avoid confused deputy attacks. As an administrator, you can set your firewall to forward connections on port 80 to another port. This is a bit dangerous on multi-user systems because, if a local user can crash apache, they can race it to listen on the forwarded port and then intercept all inbound HTTP traffic. inetd can also be configured to forward connections from port 80 to another process or pass the accepted socket over a UNIX-domain socket, which allows it to pass the connection only to a process that can open a UNIX domain socket in a specific location (e.g. a directory owned by the Apache user).

      Finally, the simplest and least error-prone solution: MAC policies (e.g. SELinux, TrustedBSD, and so on) can be configured to permit a specific application, running as a specific user, to open listening sockets on specified privileged ports. The httpd binary running as the apache user may listen on port 80, but no other program running as the appache user and no other using running the httpd program may do so.

      In short, while this is a problem with classic UNIX, it's one that was solved at least a decade ago.

      --
      I am TheRaven on Soylent News
    19. Re:What is really worrying ... by Chris+Mattern · · Score: 1

      No, if they're running as root, they have a reason. I have no idea what that reason could possibly be, but there has to be one.

      Five will get you ten that they had a permissions problem and instead of fixing it right, they "solved" it by running the webserver as root.

    20. Re: What is really worrying ... by mlts · · Score: 1

      Isn't Docker OS agnostic, where a container can be sitting on a Linux box, or a Windows Server 2016 machine (when the OS goes RTM) that either runs the container in the current VM, or spawns a VM using Hyper-V for it.

      To me, I read/hear about how great it is for applications to be in the neat little vacuum beds that Docker provides... but the fact that UID root in the container is UID root in the underlying machine or VM is concerning. At least MS solves this by giving the option to put containers in their own VMs with a mini version of Windows.

    21. Re: What is really worrying ... by devman · · Score: 1

      What you are describing is basically what Docker Machine does. It creates/controls VMs with a docker host on it and then allows you to run docker containers on that host.

    22. Re:What is really worrying ... by ebvwfbw · · Score: 1

      Running a web server as root is a 1990s thing. We used to laugh at it, fix it. 10 years ago it was considered professional incompetence. Today, for a security company, it's unforgivable. If you install apache on any of these distros, it's not root by default. Hasn't been for well over a decade. Meaning they had to set it that way. Probably because they weren't smart enough to get something to work using the regular security access controls. I bet - turn selinux off, set stuff to 777... hell run it as root! There, it works now.

      Probably a windows guy that thinks he knows about security. Run into them all the time.

    23. Re: What is really worrying ... by ArsonSmith · · Score: 1

      docker is a different approach to the same problem that VMs solve. UID 0 in a container may be nearly the same as UID 0 outside a container, but the simple fix to that is don't run anything inside the container as UID 0 and don't install anything that grants permission upgrade, they are not needed.

      --
      Paying taxes to buy civilization is like paying a hooker to buy love.
    24. Re: What is really worrying ... by ArsonSmith · · Score: 1

      If you can't trust your developers you shouldn't be blindly deploying their code either.

      --
      Paying taxes to buy civilization is like paying a hooker to buy love.
  5. Should really eat your own dog food. by unimacs · · Score: 2

    Sometimes the companies most in need of the services they provide are themselves.

    I frequently walk by this handyman's house where he has a sign advertising his various services including painting. I shake my head every time I see it because his house needs a good paint job more than any other house on the block.

    1. Re:Should really eat your own dog food. by 93+Escort+Wagon · · Score: 2

      I used to regularly pass by a auto repair shop whose sign read "Percision Automotive".

      --
      #DeleteChrome
    2. Re:Should really eat your own dog food. by spauldo · · Score: 1

      I do woodworking as a hobby.

      I recently fixed a cabinet for a family member. A glue joint had come loose, not a big deal.

      My cabinets are missing half the doors and two of the drawers are busted. I just never seem to get around to fixing it...

      --
      Those who can't do, teach. Those who can't teach either, do tech support.
    3. Re:Should really eat your own dog food. by RabidReindeer · · Score: 1

      How would you feel about a painter who has so little work that he spends all of his time detailing his house?

      I'd say he's got at least enough free time to keep his skills up.

      If he doesn't have time to do his own house, then he's not using his resources efficiently, as he should either be profitable enough to hire help and free up some time or he's just slapping paint as fast as he can to save money. NEVER trust anyone who's working 100%. They don't have the reserve resources to handle life's obstacles.

      Or he's bone lazy and only works when someone's paying (if then).

    4. Re:Should really eat your own dog food. by David_Hart · · Score: 1

      I do woodworking as a hobby.

      I recently fixed a cabinet for a family member. A glue joint had come loose, not a big deal.

      My cabinets are missing half the doors and two of the drawers are busted. I just never seem to get around to fixing it...

      No different than people who work in IT, programming, etc and don't backup their systems....

  6. Root for the convience of it all. by Camel+Pilot · · Score: 1

    Running httpd as root really solves a lot of those file permissions problems when you writing files with cgi :)

  7. I didn't think you could run apache as root by nedlohs · · Score: 2

    Well not without compiling from source with -DBIG_SECURITY_HOLE set, which surely provides a "maybe we are doing this wrong" double check...

  8. Of course they ran it as root. by Anonymous Coward · · Score: 1

    I mean, how else are you going to be able to listen on port 80?

  9. Unbelievable, yet believable by JustAnotherOldGuy · · Score: 1

    A "security" company running their servers as root...honestly, you can't make this stuff up.

    --
    Just cruising through this digital world at 33 1/3 rpm...
  10. And they hate open source by Anonymous Coward · · Score: 1

    Per FireEye's official response to the The Stack article: "No company in the world would want their IP revealed. "

    Wait, they *were* using open source software. Now I'm really confused...

    1. Re:And they hate open source by dunkindave · · Score: 2

      From the Forbes article, there were many problems, with running the webservers as 'root' just one of them. Another was a pair of zip email attachments could trigger the FireEye software to "open the files for analysis and in doing so open a backdoor on its appliance". It sounds like the researcher heavily redacted his presentation, then presented, which is why we know what we do. It also means a lot of other juicy bits were probably removed and not presented, so the bad we know about (which is bad) is just part of their problems. My guess is they considered what the guy discovered in the process to be revealing of their software's architecture and therefore would be revealing IP.

    2. Re:And they hate open source by turbidostato · · Score: 1

      "No company in the world would want their IP revealed. "

      Of course not.

      Specially if you are running a daemon with root privileges on a port on that IP.

  11. he did this work under contract to FireEye by YesIAmAScript · · Score: 5, Insightful

    If you do work for hire, you do not control whether you can publish information you discovering doing that work.

    And what kind of security consultant airs his customers' dirty laundry? Not one that wants future customers.

    If he had found this on his own, it'd be his call. But if he did it for FireEye, it's FireEye's call.

    --
    http://lkml.org/lkml/2005/8/20/95
  12. Yeah but by JustAnotherOldGuy · · Score: 1

    Yeah but running everything as root is super-convenient, guys.

    --
    Just cruising through this digital world at 33 1/3 rpm...
  13. Who paid off the German District Court? by bobthesungeek76036 · · Score: 1

    Three (3) weeks to serve the injunction? Someone has a new pair of shoes...

    --
    Karma: Bad
  14. Almost as bad... by h33t+l4x0r · · Score: 2

    It turned out that the root password was "password"

    1. Re:Almost as bad... by turbidostato · · Score: 1

      "It turned out that the root password was "password""

      DAMN! Now I know how they managed to resist my cracking attempts: I didn't think about the double quotes on "password"!

  15. Re:LMAO & in small letters below that? by 93+Escort+Wagon · · Score: 1

    P.S.=> Gotta ask: Was the place any good @ fixing rides or what? apk

    I don't know first-hand, since they weren't located all that close to where I live; but they do still exist, and fixed their sign perhaps a decade ago - so I'm guessing they must at least be good enough to keep clients.

    --
    #DeleteChrome
  16. FireEye wanted to conceal IP .. by nickweller · · Score: 2

    "We tried to conceal from the researchers to publish our IP. No company in the world would want their IP revealed. We did that to protect our customers. We openly worked with them to fix the vulnerabilities, and patches have been available for months now. Our Customers are protected." ref

    1. Re:FireEye wanted to conceal IP .. by Anonymous Coward · · Score: 2, Funny

      199.83.131.186 - no big secret.

    2. Re:FireEye wanted to conceal IP .. by phorm · · Score: 1

      What they don't mention is that IP in this case stands for Idiotic Problems.

    3. Re:FireEye wanted to conceal IP .. by jbmartin6 · · Score: 2

      You should never give out your IP address on the Internet!

      --
      This posting is provided 'AS IS' without warranty of any kind, implied or otherwise.
  17. Wikipedia says you're wrong by lucm · · Score: 1

    I wish you kids would stop running your mouths while the adults are talking. Port 8080 has been used since the beginning for the web. It was used long before Java even existed.

    According to Wikipedia, the "web" was created in November of 1990, and Java in June 1991.

    Also according to Wikipedia, port 8080 is associated with Tomcat.

    You may now apologize to RabidReindeer for being wrong and disrespectful, and also apologize to adults in general for making stupid statements in their name.

    --
    lucm, indeed.
    1. Re:Wikipedia says you're wrong by lucm · · Score: 1

      the grumpy said "long before java was even created", not tomcat. I just wanted to let him see how annoying it is when someone takes stuff verbatim and uses it to "prove" someone wrong, like he did to the other guy. And he has the audacity of telling other people to stop posting, as if he was the owner of this forum. Disrespectful.

      Anyways if we were to consider this thing in the context of this story, which is security experts running their web server as root, the relevance of whatever the fuck port 8080 was used for 20 years ago is at best weak.
       

      --
      lucm, indeed.
  18. Clickbait Headlines by Anonymous Coward · · Score: 2, Interesting

    So looking at this in depth, it looks like FireEye has already publicly disclosed said vulnerabilities after fixing them months ago. They then try to stop the presentation because it allegedly reveals too much of their IP (which is itself worth discussing but totally separate) and we get a bunch of headlines saying "ZOMG! FireEye is trying to silence people for revealing vulnerabilities!". This is trigger happy, bullsh*t journalism at its finest. Not quite accurate or informative but just close enough to get people prematurely worked up in a tizzy for page views.

  19. "server"? by Mats+Svensson · · Score: 1

    Hmm, what is a "server", and what does it "do"?

  20. Re: They want to lay the blame on a guy named roo by loufoque · · Score: 1

    I wonder what happens when Robert Oot is assigned a unix login.

  21. Re: They want to lay the blame on a guy named roo by Swave+An+deBwoner · · Score: 1

    useradd: user 'root' already exists