Slashdot Mirror


Security Company Says NASDAQ Waited Two Weeks To Fix XSS Flaw

alphadogg writes "A Swiss security company said the NASDAQ website had a serious cross-site scripting vulnerability for two weeks before being fixed on Monday, despite earlier warnings. Ilia Kolochenko, CEO of the Geneva-based penetration testing company High-Tech Bridge, said he repeatedly emailed NASDAQ and warned of the XSS flaw. 'I can basically say I have spammed them,' Kolochenko said in an interview. A NASDAQ spokesman did not have immediate comment. NASDAQ.com lets users create accounts and build a profile to monitor stocks and news."

61 comments

  1. Very difficult. by d33tah · · Score: 4, Funny

    What are you laughing at, it's clearly very difficult to fix one XSS vulnerability.

    1. Re:Very difficult. by Anonymous Coward · · Score: 1

      -- "Your car is broken"

      -- "Yeah we know, but it's really difficult to fix it"...

      WTF???

    2. Re:Very difficult. by cbhacking · · Score: 5, Insightful

      For the unaware: this is serious sarcasm. Fixing XSS is usually pretty trivial; just apply output encoding (usually HTML entity encoding, but there are other valid approaches) to the user-supplied data before reflecting it into the page. Even in weird edge cases, like where the user is explicitly allowed to insert their own HTML (Slashdot, for example) you can get around the problem by whitelisting certain elements and parameters, and rejecting (or removing, though this must be done carefully) anything which doesn't conform. It's A long-ago solved problem that some people still have incredible difficulty with.

      Doing security work myself, I've seen XSS fix times ranging from "within the hour" to "three weeks or so", and the median is probably about two days. I always wonder what the hell is up with the companies on the long end of that scale.

      --
      There's no place I could be, since I've found Serenity...
    3. Re:Very difficult. by sjames · · Score: 2

      Perhaps it is a serious problem. Without actually seeing the vulnerability, it's hard to say.

      WAY too many security scanners panic the instant they get any fragment of text to display on the page, even if it is sufficiently munged that no script could ever be made to run. They'll still cry wolf when you entity code it.

      They're like elementary school kids the first time they get the calculator to say hell or boobs.

    4. Re:Very difficult. by Anonymous Coward · · Score: 1

      -- "Your car is broken"

      -- "Yeah we know, but it's really difficult to fix it"...

      WTF???

      You'd be shocked at how many people regularly drive around with check-engine lights on, bald tires, burned-out taillamps, broken window motors/tracks...

    5. Re:Very difficult. by Anonymous Coward · · Score: 0

      They had to wait for a long enough break between trades to slip in the new code. Patience, my friends.

    6. Re:Very difficult. by Anonymous Coward · · Score: 0

      I can't speak for others. But, it's not like the code is *always* in a state where I can just drop a new version out there quickly. In addition, it depends on the possible impact of the vulnerability. If it has low risk, it would be tagged onto the next release. If it's high risk, it will get it's own release.

    7. Re:Very difficult. by Anonymous Coward · · Score: 0

      Thjey had to wait the NSA hadnt finished laundering their blackops slush funds

    8. Re:Very difficult. by Anonymous Coward · · Score: 0

      People always believe bad things happen to other people. I used to work at a software company when I found a gaping hole in their application's security. It would allow any employee in a client's firm to log directly into the timekeeping database and manipulate anybody's in and out times without a trace. The company's response was, "no one's likely to find that." It didn't even require an exploit or code. All you had to do was open the config file on any client PC to read the SQL username and password in plaintext. The rest is easy to figure out.

    9. Re:Very difficult. by Zero__Kelvin · · Score: 1

      Yes, but you'd be surprised how few Wall Street Investors drive around with check-engine lights on, bald tires, burned-out taillamps, broken window motors/tracks.

      --
      Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
    10. Re:Very difficult. by Anonymous Coward · · Score: 0

      Not really. You're describing my car, you insensitive clod!

    11. Re:Very difficult. by TechyImmigrant · · Score: 1

      I've never owned a car that doesn't have a TPMS light on, regardless of the state of the tires.

      --
      I should use this sig to advertise my book ISBN-13 : 978-1501515132.
    12. Re:Very difficult. by slick7 · · Score: 1

      ...in the meantime...how was the vulnerability exploited? Furthermore, when will we find out? Subsequentially, will this loophole be closed to prevent further exploits?

      --
      The mind conceives, the body achieves, the spirit manifests.
    13. Re:Very difficult. by Anonymous Coward · · Score: 1

      Perhaps it is a serious problem. Without actually seeing the vulnerability, it's hard to say.

      WAY too many security scanners panic the instant they get any fragment of text to display on the page, even if it is sufficiently munged that any script could ever be easily made to run in contexts not conceived of by the site builders. They'll still cry wolf when you completely misunderstand the problem and fail to fix it, repeatedly

      They're like elementary school kids the first time they get the shit kicked out of them, and try to warn the other kids before they get the shit kicked out of them.

      Fixed that for you so you don't look like an imbecile. You're welcome.

    14. Re:Very difficult. by sjames · · Score: 1

      With a post like that, I can see why you went the AC route. Good call!

    15. Re:Very difficult. by Anonymous Coward · · Score: 0

      The answer to the problem you suffer from: git flow

      i.e. If you can't throw some code into the production codebase for a release (or even hotfix on production and merge back down), you are Doing It Wrong.

    16. Re:Very difficult. by cbiltcliffe · · Score: 1

      I was following a Mercedes SUV the other day, I'd guess at least a $100k vehicle. One of its rear tires was so worn that the steel belts were showing through the rubber of the tread. Keep in mind, this wasn't an up close inspection. This was what I could see from the driver's seat of my vehicle while I was behind them. I'd guess the belts were showing for between a quarter and a third of the circumference of the tire. The other side wasn't showing belts, but it was obviously pretty much bald, too.

      --
      "City hall" in German is "Rathaus" Kinda explains a few things......
  2. Sounds like a fast response... by CajunArson · · Score: 4, Interesting

    Despite the twitch mindset that many people on this website have about security vulnerabilities, fixing a bug like that and deploying the fix in only 2-weeks is excellent for any project (open/closed/otherwise) and is especially good for a large commercial service like Nasdaq.

    --
    AntiFA: An abbreviation for Anti First Amendment.
    1. Re:Sounds like a fast response... by cbhacking · · Score: 5, Interesting

      Um... no. Fixing XSS is trivial. I work in this field myself; only a small percentage of our clients take more than a week to fix a reported issue, and many manage it same-day. This includes quite large and well-known software companies and websites, including in the financial sector (although I'll admit that the financial sector tends to be on the slower end of things).

      --
      There's no place I could be, since I've found Serenity...
    2. Re:Sounds like a fast response... by h4rr4r · · Score: 1

      Uh no, getting 10 minutes worth of work done in two weeks is a bad sign for an organization.

    3. Re:Sounds like a fast response... by CodeReign · · Score: 2

      as a software developer I will refer you to the dictionary for the word `overhead` and dilbert for a business case example.

    4. Re:Sounds like a fast response... by QilessQi · · Score: 4, Insightful

      No, taking the time to
      (1) evaluate the problem,
      (2) determine the best approach for a fix,
      (3) weight the time commitment against any other critical activities going on,
      (4) assign the best person to code it,
      (5) review the code,
      (6) rebuild and deploy it up through various testing environments,
      (7) test the hell out of it in each environment, and
      (8) deploy it into production
      in a mere 10 work days is excellent, given the importance of the system. That's what Enterprise System timelines are often like.

      If a major financial system like NASDAQ had managers run into the developer area and shouting "ZOMG someone fix this and slam it into Production now now now!", then I'd be more concerned.

    5. Re:Sounds like a fast response... by h4rr4r · · Score: 3, Insightful

      There is such a thing as too much process, this would be an example.

      Besides all those steps should take a couple days tops.

    6. Re:Sounds like a fast response... by Anonymous Coward · · Score: 0

      10 minutes to code... then you have code review, and QA verification, and deployment. It's not how long it takes to code, but how long to get those schedules lined up. It's not like those 4 people are all sitting around just waiting for the next vulnerability to come though. BTW deployment is easy when you have one server that the developer can write to. It's not so easy when you have multiple farms over multiple data centers.

      Also, all those checks and balances that I just mentioned may seem like a lot of overhead, but they significantly reduce the number of bugs and vulnerabilities.

    7. Re:Sounds like a fast response... by Em+Adespoton · · Score: 2

      Um... no. Fixing XSS is trivial. I work in this field myself; only a small percentage of our clients take more than a week to fix a reported issue, and many manage it same-day. This includes quite large and well-known software companies and websites, including in the financial sector (although I'll admit that the financial sector tends to be on the slower end of things).

      Actually fixing the XSS is trivial; it's QAing and publishing the fix that takes the time.

      Just think of Microsoft and their recent security patches; you can really make a mess of things while applying a trivial fix. The testing is needed to verify that applying the fix doesn't do something bad (doesn't matter what quality the fix is or how long it takes to write).

      That said, it really shouldn't take longer than a week, assuming it's been properly flagged as critical. My guess is that it was incorrectly triaged in this case, so the entire change schedule started late.

    8. Re:Sounds like a fast response... by TechyImmigrant · · Score: 1

      Yup. A simple fast track for simple security problems.

      1) work out solution
      2) Implement & deploy solution as a hotfix same day
      3) Spend 2 week massaging and validating the solution
      4) Update the codebase with the validated solution (which may be identical).

      --
      I should use this sig to advertise my book ISBN-13 : 978-1501515132.
    9. Re:Sounds like a fast response... by Anonymous Coward · · Score: 0

      Microsoft has a known history of writing terrible code though. if you do things right, pushing out fixes really is trivial. Hell with a good QA team even the most junior of developers won't be able to get a vulnerability into the production codebase (not for lack of trying though).

      Just a note, if you compare your organization to Microsoft, you're basically telling every professional developer and engineer in the world that your work is hideously subpar. You might as well tell us who you work for, so we can cancel any contracts we might have with you (or avoid them in the future).

    10. Re:Sounds like a fast response... by Anonymous Coward · · Score: 0

      Sorry, but that "enterprise" you're talking about either has an atrocious security process or a horrific codebase (or both). 90% of your process is a problem with your codebase, another 5% is your process, and probably another 2% is organizational (if you have to "assign" the best person to code it, then obviously no one has any idea who should be in charge of things). Five of your steps are completely due to bad practices to begin with. Clean up your technical debt and you can fix things a LOT quicker. That's how we do it in the "enterprise" :)

  3. How about the real story today? by the+eric+conspiracy · · Score: 4, Interesting

    The NASDAQ today had it's 3rd significant pricing problem in the past few weeks.

    http://www.nasdaq.com/article/options-exchanges-halt-trading-20130916-00868

    These guys seriously need to improve their reliability.

    1. Re:How about the real story today? by CajunArson · · Score: 1, Funny

      Just remember, NASDAQ runs on Windows on days when things go wrong (it runs Linux during the rest of the week).

      --
      AntiFA: An abbreviation for Anti First Amendment.
    2. Re:How about the real story today? by Anonymous Coward · · Score: 0

      Your comment would be more useful if you understood what happened today. OPRA, which is an independent company had an issue which effected every options trading exchange in the US. It was not NASDAQ's fault.

    3. Re:How about the real story today? by the+eric+conspiracy · · Score: 1

      OPRA is not really an independent company. Yes, it is a company separate from the NASDAQ, however is is controlled by the exchanges that are members of it's plan. NASDAQ is one of these.

      http://www.opradata.com/overview/opra_over.jsp

    4. Re:How about the real story today? by Anonymous Coward · · Score: 0

      The NASDAQ today had it's 3rd significant pricing problem in the past few weeks.

      http://www.nasdaq.com/article/options-exchanges-halt-trading-20130916-00868

      These guys seriously need to improve their reliability.

      correction - OPRA is an Options Industry Utility run by New York Stock Exchange (SIAC) not NASDAQ.

    5. Re:How about the real story today? by cbiltcliffe · · Score: 1

      So basically, it's a separate company in the same way that Chevrolet is a separate company from General Motors?

      --
      "City hall" in German is "Rathaus" Kinda explains a few things......
  4. OH NO. Two whole weeks?!?!!11ONE!! by Lieutenant_Dan · · Score: 2

    That's not too bad all things considering. Maybe they have a proper structured development shop (not too structured, since it obviously doesn't include code reviews or vuln scanning)? Maybe they had maintenance windows which they are contractually bound to (and more expensive to make an exception then to do deal with a flaw)? Maybe once they were made aware of the problem they were scanning the database system for odd entries or suspicious activity? Maybe they needed to get an independent audtor to review so they can appease their various stakeholders?

    Hopefully they learned from this, and will at least run an automated vulnerability tool against the app for future releases.

    --
    Wearing pants should always be optional.
  5. Penetration testing company by Anonymous Coward · · Score: 0, Funny

    Butt-Head: Huh huh, you said penetration.
    Beavis: I'd love to work at that place!

  6. Who cares? by Gizzmonic · · Score: 5, Insightful

    So, it's the NASDAQ website. Who goes the NASDAQ website? You can't trade stocks there. Financial information was not leaked, so BFD. This is fairly common on any website. Sounds to me like a single security research got butthurt because they didn't acknowledge his finding quickly enough.

    --
    (-1, Raw and Uncut is the only way to read)
    1. Re:Who cares? by Luthair · · Score: 1

      I agree, sounds like someone trying to make a name for themselves. Seems unlikely that the large scale investors would use the NASDAQ to monitor their portfolio so its unlikely someone would be spear phishing for stock tips.

    2. Re:Who cares? by Princeofcups · · Score: 2

      So, it's the NASDAQ website. Who goes the NASDAQ website? You can't trade stocks there. Financial information was not leaked, so BFD. This is fairly common on any website. Sounds to me like a single security research got butthurt because they didn't acknowledge his finding quickly enough.

      Agreed. So they emailed NASDAQ. Who at NASDAQ? Who picks up and reads that email? Was it just the web site tech support number? That might get read only once a week if you are lucky. Unless they actually spoke to a manager/director at NASDAQ who is responsible for security, then they were not really talking to "NASDAQ."

      --
      The only thing worse than a Democrat is a Republican.
  7. Spam by tanujt · · Score: 1

    I can basically say I've spammed them

    Well, there's your answer.

  8. NASDAQ web site != NASDAQ trading system by JoeyRox · · Score: 5, Informative

    nasdaq.com is a simple front-end fluff site for viewing quotes and doing basic company research. No critical systems or customer data.

    1. Re:NASDAQ web site != NASDAQ trading system by trmj · · Score: 1

      Obligatory xkcd

      --
      Work sucked, until it became unemployment, when it became slightly more tolerable. -Tet
    2. Re:NASDAQ web site != NASDAQ trading system by MXPS · · Score: 3, Insightful

      Seriously. With all the actual issues they have been having with trading system I'm sure the vulnerability on the website was the least of their worries. It is like saying FBI.gov or NSA.gov has a vulnerability so all our countries secrets will be exposed.

    3. Re:NASDAQ web site != NASDAQ trading system by Anonymous Coward · · Score: 0

      ... I'm sure the vulnerability on the website was the least of their worries. ...

      Why should we care about their worries? Exploiting an injection vulnerability is something that causes problems to us, not directly to them.

  9. Re:OH NO. Two whole weeks?!?!!11ONE!! by h4rr4r · · Score: 1

    Two weeks is a long time for a few minutes worth of work. It is symptomatic of the kind of disease you find in all large organizations.

    They could have implemented the fix before checking the DB or doing other auditing. Those things are not dependent on each other.

  10. So What? by Anonymous Coward · · Score: 0

    Don't you get it yet? Most people don't understand security issues. Even fewer are interested in theoretical security issues. Here we have a perfect example of apathy without cost.

    So what if there was a security hole? It's been fixed. So what if it existed for one week/month/year? No one exploited it so it can't have been all that "real" a risk. So what if NASDAQ could have lost millions? It's not my money and I didn't lose a penny, ergo, I don't care!

    So what? No one cares!

  11. good process is not trivial by Anonymous Coward · · Score: 2, Insightful

    Its called process.

    Product Owner is sent notice of vulnerability.

    Operation or QA tries to reproduce the issue.

    Upon confirming the vulnerability, Product Owner tells business analyst and dev. manager about issue: change request is created.

    Dev team picks up ticket, and does more analysis.

    Geek reproduces issue locally.

    Geek writes failing, automated test that reproduces the error.

    Geek fixes error, automated test is passing.

    Geek has code reviewed by team members, and probably infosec.

    Geek hands code off to QA.

    QA checks first does a check to see the vulnerability, then tests that code is really fixed, also validating that Geek's script isn't broken. Then does regression test to make sure that code isn't broken.

    QA schedules time with operations to meet and discuss deployment plan.

    Code is deployed to stage environment and tested some more.

    Operations then deploys the fixed code.

    1. Re:good process is not trivial by NatasRevol · · Score: 4, Funny

      In reality,

      Dev gets email, updates code, posts to live website.

      He's just 3 weeks behind on email.

      --
      There are two types of people in the world: Those who crave closure
    2. Re:good process is not trivial by Anonymous Coward · · Score: 0

      When millions are flowing through your system, you have checks and balances... Dev does not have permission to write to the production website. OPs cannot install to servers without Product owner's signoff, etc.

      The idea is that no one person can fuck with all that money.

      (I work on systems that have large amounts of money running through them)

    3. Re:good process is not trivial by Anonymous Coward · · Score: 0

      You left out a step. Dev calls NSA to find out if FTM program (Follow The Money) makes use of the XSS 'feature' when monitoring known money laundering networks and terrorist organizations that use NASDAQ accounts... As a Dev, you wouldn't want to throw a monkey wrench into the surveillance network that keeps investments safe from evil-doers... now would you.

      Perhaps the delay wasn't due to incompetence but the secret cooperation of Patriots who watch over your eggs while you sleep.

    4. Re:good process is not trivial by Anonymous Coward · · Score: 0

      More like:

      - Dev gets email
      - Dev fixes issue
      - Snapshot build released to test environment (all of this happens in half a day or so).
      - Testers test fix; run regression suite to verify nothing else has been broken.
      - Testers don't really understand what they're looking for; some back and forth with dev team.
      - Environment issues prevent full testing from occuring until off-shore support team comes online. (maybe a bit of to-and-fro taking us to Thursday of the first week).
      - Change request raised
      - Change goes through 10 different change review boards.
      - Change gets packaged up, signed-off, documented, handed off to implementation team for installation.
      - Change categorised as "non-urgent" as it's not preventing users from accessing the system. Normal lead time 3 days. Cannot be implemented during the work-week as system up-time/availability is critical, and code hot-swap isn't well supported.
      - Change conflicts with another high importance change on the immediate weekend, and the risk of making two such changes is deemed too high, therefore this one gets pushed to the week after.

    5. Re:good process is not trivial by philip.paradis · · Score: 1

      You must work for an outfit with a rather fast and loose approach to both infosec and quality assurance.

      --
      Write failed: Broken pipe
    6. Re:good process is not trivial by Anonymous Coward · · Score: 0

      The systems I oversee have tens of millions flowing through per week. I and the other senior dev have write access to production. Our decisions overrule any PO's word on pretty much anything when it comes to security (for obvious reasons that your organization has failed at) and have probably saved the company billions in losses by now had we not been so anal about security.

      We also don't have XSS or XSRF vulnerabilities* because of thorough code review, extremely well dictated best practices, and diligent unit testing; as well as skilled QA that will attempt to usurp every possible interaction between the client and the server to insert vulnerabilities.

      *Not saying we DON'T absolutely, but we have never suffered from such nor had a site compromised or defaced through a vulnerability, despite some VERY clever efforts to do so.

  12. the IT probably needs some slack by goffster · · Score: 0

    If, at NASDAQ, you create even a moment of downtime,
    your ass is chewed, spit out, and handed back to you.

  13. Fast enough. by RightSaidFred99 · · Score: 1

    I love how apparently everyone complaining about this must have mastered all unintended consequences.

    Sometimes fixing a bug just isn't that important, even a security bug. Sometimes the stuff you break can be worse than what you fixed. This is why it may take 30 minutes or 2 days to fix the issue, but much longer to make sure you actually want to push it to production.

    If they had private medical data or insider information that could be hacked by exploiting the issue I imagine they would have fixed it within an hour. As it stands, they probably didn't _need_ to fix it any faster than a few weeks.

  14. Re:OH NO. Two whole weeks?!?!!11ONE!! by Lieutenant_Dan · · Score: 1

    Agreed. Chances are there are a bunch of PMPs and ITIL processes in place. Could be internal politics.

    Coding a few minutes is a one thing. Testing it, getting someone to approve to move something to prod, and herding people to actually do work is a bunch of other things. Legal and PR may get involved too.

    In some corps I worked, the finger-pointing usually takes days and involves a bunch of crappy meetings. It can be days before someone engages InfoSec or the developers to confirm a problem.

    Two weeks is not terrible; better than most large corporations.

    --
    Wearing pants should always be optional.
  15. So, bad Website design but what was the exposure by Virtucon · · Score: 1

    I realize it takes a few minutes to fix an individual XSS problem, but what was the vulnerability here?

    NASDAQ says this from the article:

    Nasdaq.com lets users create accounts and build a profile to monitor stocks and news. Nasdaq said it did not believe the flaw was used by an attacker, and no personal data was compromised.

    "We responded to his concerns immediately," Nasdaq said in an email statement. "We take all information security matters seriously. We work with leading security vendors and have a trained and professional team that evaluates all credible threats across our digital assets."

    So they took the information seriously, evaluated it and said that it didn't create a big gaping hole.

    while they guy who notified them of the problem reportedly said:

    Kolochenko said the flaw could have been used by an attacker in several ways, including stealing users' browser histories and their cookies. It could also have been used to inject HTML into a Web page and ask for people's personal details, a request that would appear to come from Nasdaq.

    In another kind of attack, Kolochenko said the XSS flaw could be used to plant a link within the Nasdaq site to a malicious website.

    Kolochenko said XSS flaws are common, and he has found ones in websites belonging to the BBC, Bloomberg and the Financial Times. Those organizations acknowledged the issues, but it was often a month or so before the websites were fixed, he said.

    Yes, so XSS vulnerabilities are bad and it's up to the website owner to ascertain how bad it could be for their website on a case by case basis. It sounds like this guy spends his days however just digging up XSS vulnerabilities and then spams the companies saying you have a problem, possibly to mine bug finding cash. I think you'll find that in each case, those companies notified of an XSS vulnerability addressed the issue within their usual change management process. This is a reasonable and logical approach.

    Now, if the guy said "XSS vulnerability crashes NASDAQ and steals your money!" or "NSA uses XSS vulnerabilities to steal your data." I'd be a bit more worried.

    Nothing to see here, move along and remain calm.

    --
    Harrison's Postulate - "For every action there is an equal and opposite criticism"
  16. Think about it by Tetetrasaurus · · Score: 2

    Of course NASDAQ could have fixed it immediately, but what very valuable information did they gather during the two-week window from first report to fixing it?

    Why, all the info on all the zero-day crackers giving it a go during that period -- a massive sort of honeypot operation.

    Think about it. Easy to plan the protocol for. Like flies to honey.

  17. A State Of Lies by Anonymous Coward · · Score: 0

    General Clapper States:
    "What we do not do," Clapper said in a statement, "is use our foreign intelligence capabilities to steal the trade secrets of foreign companies on behalf of - or give intelligence we collect to - U.S. companies to enhance their international competitiveness or increase their bottom line."

    This is a provable lie!

    Happy Birthday "General" Clapper.

  18. Reliability? by Errol+backfiring · · Score: 1

    Wait. The NASDAQ is a huge gambling institute where you can gamble with other people's money, right? What reliability are you speaking of?

    --
    Nae king! Nae laird! Nae yurrupiean pressedent! We willna be fooled again!