Slashdot Mirror

← Back to Stories (view on slashdot.org)

NVIDIA Breached

Posted by Soulskill on from the caught-in-the-crossfire dept.

jones_supa writes: Another day, another corporate network intrusion. NVIDIA has reportedly been breached in the first week of December, with the attack compromising personal information of the employees. There is no indication that other data has been compromised. This is according to an email sent out by the company's privacy office and Nvidia's SVP and CIO Bob Worwall on December 17th. It took NVIDIA a couple of weeks to pick up all the pieces and assess the incident. It appears that the issue was pinned down by an employee or several employees getting their personal data compromised outside of the company network. After that, the information was used to gain unauthorized access to the internal corporate network. NVIDIA's IT team has taken extensive measures since then to enhance the security of the network against similar attacks in the future.

39 of 59 comments (clear)

  1. "extensive measures" taken... by mlts · · Score: 3, Informative

    I wonder what that means, exactly.

    My hopes are that it means ensuring anyone on the outside is coming in via 2FA, internal and DMZ networks have a proper IDS/IPS in place that is tailored to the division in question (i.e. a bunch of point of sale terminals would sound an alarm if one of them decides to start making random connections to a site in Elbonia), there is an internal detection process so someone trying to brute force an account will make an audit trail and get a curious admin looking at why the events are happening.

    My hopes also include isolation of DMZ boxes so that unless they are intended to communicate with each other, they can't. Isolation between departments would be nice as well.

    Finally, my hopes include having remote access being more of using Citrix or RDP and having the remote machine be more of a dumb terminal, as opposed to an active VPN, making the remote machine a part of the corporate network.

    Of course, my fear is that "extensive measures" will be a domain admin logging on, popping up a command shell, typing in:

    dsquery user | dsmod user -mustchpwd yes

    and calling it a wrap.

    I'm hoping nVidia does more of the "hopes" portion.

    1. Re:"extensive measures" taken... by fuzzyfuzzyfungus · · Score: 3, Interesting

      Given that I have no particular personal stake in Nvidia's problems, I would hope, in the spirit of general benevolence, that they take actually effective action; but I would much, much, much more strongly hope, in my own interests and those of computer users generally, that they've taken effective measures surrounding control of their signing keys.

      Aside from a few *nixes that are actively hostile to proprietary drivers or simply don't do any integration work for Nvidia's, Nvidia is one of the hardware companies whose signature is pretty much universally trusted, without much question or notification, on a driver. If their signing infrastructure were to have been compromised, some very, very, interesting 'GPU drivers' might make it out into the wild and raise some hell.

      Unfortunately, this is true of other hardware outfits as well. I don't much care how they run things, though friendly advice would be to pay attention to the security geeks; but anyone who has a signing key that will get a driver right into the kernel of any windows system without comment(extra credit for getting it on Windows Update) is an active menace if they lose control of that.

    2. Re:"extensive measures" taken... by Solandri · · Score: 2

      My hopes also include isolation of DMZ boxes so that unless they are intended to communicate with each other, they can't. Isolation between departments would be nice as well.

      The problem with isolation is that some twit of an employee decides it's inconvenient, sneaks in a couple wifi routers, and sets up an authorized bridge.

      Proper security relies not just on IT locking everything down. IT has to be willing to go the extra mile to do so in a way which minimally hampers other employees from doing their jobs. Fail at that and you get employees setting up rogue bridges and networks. Unfortunately, some IT people I've encountered act like they're on a power trip and control "their" network with an iron fist. One place I visited wouldn't even let you plug in flash drives to transfer data (we tried to get authorization and were told it was never allowed). So that data ended up being sent as an oh-so-secure email attachment. Unencrypted since the system interpreted an encrypted zip file as malware and auto-deleted it.

      Putting a super-secure lock on a door is pointless if it makes getting around the workplace so difficult that everyone just uses a trash can to hold door open all the time. IT needs to make things secure while minimally impeding other employees from doing their jobs. No that's not easy. If it were easy, it wouldn't be a high-paying job.

    3. Re:"extensive measures" taken... by RingDev · · Score: 1

      "Finally, my hopes include having remote access being more of using Citrix or RDP and having the remote machine be more of a dumb terminal, as opposed to an active VPN, making the remote machine a part of the corporate network."

      Either way there are concerns. With Citrix (assuming Citrix Web, since you specifically call out no VPN), you wind up with the exact same issue as what happened here. As soon as someone's username/password is compromised, the would be hacker has full access to what ever is publicly exposed. So any applications or desktops available to the user in Citrix are fair game. You are now counting on the internal security of each of your applications...

      For RDP, I am unaware of an RDP clients that are trusted for use NOT coming in over a VPN. The thought of a non-VPN RDP solution on an enterprise network is somewhat frightening to me. Do you have a client in mind? I'd like to read more about it.

      As for VPNs, if done well, there is limited risk. For example, the laptop I have with my current employer has a built in software firewall and enterprise class virus prevention solution. I also do not have administrative access (only a small handful of people in the entire agency do and it is not on their standard domain accounts). Additionally, to connect to our VPN, you need not just the VPN software/configuration, and a username/password, but that user account must also be configured to allow for VPN access (again, limited to a subset of employees) on a specific machine (not sure if they use MAC or some other hardware ID tag), and when you connect you are prompted for a rotating security code that you get by checking an RSA dongle that has been issued to you.

      So yeah, VPNs can be cracked, especially if they aren't designed to be secure, but when done properly, they can be significantly more secure than just exposing a Citrix/RDP web solution.

      -Rick

      --
      "Most people in the U.S. wouldn't know they live in a tyrannical state if it walked up and grabbed their junk." - MyFirs
    4. Re:"extensive measures" taken... by mlts · · Score: 2

      The best thing they can do with signing keys is use HSMs.

      This limits the intruder to only being able to access and use the key if the HSM's users and roles are AD linked. With proper logging, it can be told what packages were signed, and when, and if the key or package IDs needs to be actively revoked.

      This happened to a Linux distro vendor, and they managed to do an effective job at limiting the damage.

      If nVidia isn't using HSMs for the key signing, they better start, as virtually every blackhat knows that getting their code to run under the signed nVidia flag is obtaining the key to the city, virtually everywhere.

    5. Re:"extensive measures" taken... by lgw · · Score: 1

      Well said. Security is not about being a control freak, because the more you tighten your grasp, the more systems will slip through your fingers (doubly so if you imagine you can police kernel devs). Instead, security is measured as follows:

      [Difficulty of unauthorized access] / [Difficulty of authorized access]

      Making authorized access harder reduces security because people. People will always make it easier fo themselves. In the world of physical security, the lesson is: "any door along the quickest path between where people work and the smoking area will be propped open - don't even try to fight it, instead make sure that doesn't compromise security".

      --
      Socialism: a lie told by totalitarians and believed by fools.
    6. Re:"extensive measures" taken... by mlts · · Score: 2

      Proper security requires a lot of factors, and as you said, this is no walk in the park. You are right that IT can't do everything. However, technical solutions are 90%-99% of what can be done and done right with minimal user inconvenience.

      However, from how the successful hacks were described, there are four things that would have slowed, if not stopped almost all of them:

      1: IDS/IPS. This shouldn't be something that a user should know/care about, unless they decide to run nmap from their PC and wonder why their box's port got dropped from the internal network. However, an IDS is like the ZoneAlarm of yore. Unless one knows what to set to have alert on the spot, and what to stuff a ball-gag in, it will be quite yappy. But there are plenty of books on this subject as well as Internet based documentation.

      2: Logging and responding to logs. For example, the Sony hack would have been mitigated by a tool like SolarWinds, SCOM, or Splunk that was configured to have thresholds, and if too many bad passwords happen, alert an admin. If AD locked a user out due to too many bad PW guesses, even if the lockout period is 1-5 seconds [1], it should generate an alert.

      3: 2FA. SecurID isn't cheap, but for a large business, it isn't that large an expense. It also provides good management, and is decently flexible. It also works with virtually anything.

      4: Measures to limit malware's influence. I brought up Citrix because a RAT could snap screenshots and capture keystrokes, but through a Citrix session, files would be a lot harder to steal, and if a user was using 2FA, as soon as the user logged out and the session key deactivated, there would be nothing a remote attacker could really do other than hope the user logs again, get screenshots, or maybe seize active control of the session (which likely will panic the user.)

      I have met some people in my field who had the mentality that they were the COs and users were the inmates. However, effective security can come without penalizing users. It just takes some common sense.

      [1]: AD's default is 20 minutes, but I've found reducing it to 3-5 minutes decent enough to not lock legit users out, but only provide an attacker 288 chances every 24 hour period to get into an account, once the lockout activates. With a sane password system (again, the default AD settings come to mind), guessing a PW of a user isn't going to be easy.

    7. Re: "extensive measures" taken... by fuzzyfuzzyfungus · · Score: 1

      There is also the much trickier; but potentially really unpleasant, matter of firmware. GPU cards have at least some flash onboard, not certain exactly how much probably varies by model; and they are a peripheral in a position of a great deal of power(big fat kernel driver, all the DMA they can eat, enough onboard RAM and computational capacity to really do interesting things with that). Certainly wouldn't want any bugged firmware sneaking around.

    8. Re:"extensive measures" taken... by khasim · · Score: 1

      [Difficulty of unauthorized access] / [Difficulty of authorized access]

      I would change that second part to

      "necessary access"

      . I'll explain in a moment.

      Making authorized access harder reduces security because people. People will always make it easier fo themselves.

      In my experience, the first problem is EGO. There is always some executive who bases his/her EGO on what exemptions he/she can get.

      I'm too important NOT to have access to X.
      From anywhere.
      Along with all my people.

      And then other executives have to have the same access because, otherwise, they are not as important. And IT can handle it, right?

      So you end up with too many people with too much access. And admin/root access to their machines. That they also use for non-work related activities because why shouldn't I have iTunes on my work laptop?

      So you end up with 100 people with VPN access to the HR servers and 95 of them don't even know it and 99 of them don't use it. BUT THEY ALL "HAVE" TO HAVE IT AND IT IS AUTHORIZED.

      In the world of physical security, the lesson is: "any door along the quickest path between where people work and the smoking area will be propped open - don't even try to fight it, instead make sure that doesn't compromise security".

      And with computer security, they bring the open doors with them. Wherever they go. And they are authorized to do so.

      But it is not necessary for them to have that authorization.

    9. Re:"extensive measures" taken... by WaffleMonster · · Score: 1

      My hopes are that it means ensuring anyone on the outside is coming in via 2FA

      What difference does it make where you are coming from? Majority of costly threats are inside jobs and or enabled by inside human error... All it takes is one marketing goon to get owned and the castle wall is breached.

      The very concept of network security is the most disastrous and perilous idea the security world has ever fabricated. If you want network security make IPSec mandatory across the enterprise everything short of this is worthless masturbation.

      and DMZ networks have a proper IDS/IPS in place that is tailored to the division in question

      IDS/IPS systems are nothing more than bureaucratic checkboxes completely worthless against actual tailored threats.

      i.e. a bunch of point of sale terminals would sound an alarm if one of them decides to start making random connections to a site in Elbonia

      If it were that simple then why are you granting the capability of point of sale terminals to make random connections to Elbonia in the first place?

      there is an internal detection process so someone trying to brute force an account will make an audit trail and get a curious admin looking at why the events are happening.

      If an online brute-force attacks ever succeed you've already failed. Doubtful requisite competence to catch this fact on the back-end even exists within such an organization.

      My hopes also include isolation of DMZ boxes so that unless they are intended to communicate with each other, they can't. Isolation between departments would be nice as well.

      Isolate those suckers or just replace them with bricks.

      Finally, my hopes include having remote access being more of using Citrix or RDP and having the remote machine be more of a dumb terminal, as opposed to an active VPN, making the remote machine a part of the corporate network.

      Turtles all the way down.

      Of course, my fear is that "extensive measures" will be a domain admin logging on, popping up a command shell, typing in:

      dsquery user | dsmod user -mustchpwd yes

      and calling it a wrap.

      I think this is about right. Attacker compromises domain, hooks pass filter API, make a scary noise and waits for everyone to change their passwords.

      I'm hoping nVidia does more of the "hopes" portion.

      I'm hoping the entire industry gets off its ass and does something other than continue to waste time, money and resources on pointless layers of worthless security just so they can pencil in a few checkboxes and CYA.

    10. Re:"extensive measures" taken... by tlhIngan · · Score: 1

      In my experience, the first problem is EGO. There is always some executive who bases his/her EGO on what exemptions he/she can get.

      I'm too important NOT to have access to X.
      From anywhere.
      Along with all my people.

      And then other executives have to have the same access because, otherwise, they are not as important. And IT can handle it, right?

      So you end up with too many people with too much access. And admin/root access to their machines. That they also use for non-work related activities because why shouldn't I have iTunes on my work laptop?

      So you end up with 100 people with VPN access to the HR servers and 95 of them don't even know it and 99 of them don't use it. BUT THEY ALL "HAVE" TO HAVE IT AND IT IS AUTHORIZED.

      Do what good IT does - give it to them for a week, and see who uses it. If no one logs in after the first day, then close it back down again. If someone asks, just re-enable and blame some software update or something thanks for noticing.

      Repeat as often as necessary.

      I can bet 99% of the time, no one will notice because really all requests get filtered through one person in the end who knows the system. Then they are the only ones that need access.

      Let them have their ego. Then close it and let them believe they still have it because you looked at the logs and saw they never touched it after the first day.

    11. Re:"extensive measures" taken... by lgw · · Score: 1

      No, "necessary" entirely misses the point. You're control-freaking. Users will find a way to do what they desire to do, and they'll find a way to make it easy to do so. That's humans for you - we're adaptive animals. Don't fight human nature.

      Instead, make it easier to hit Facebook without hurting corporate security than to do something "clever" to hit Facebook. Make it easy to IM securely, to trade work-related files securely, and so on. Get out in front of what users want to do and make the easiest way to do those things the secure way. Take what you think is "necessary" and shove it someplace necessary.

      --
      Socialism: a lie told by totalitarians and believed by fools.
    12. Re:"extensive measures" taken... by khasim · · Score: 1

      No, "necessary" entirely misses the point.

      No. That is the point.

      Users will find a way to do what they desire to do, and they'll find a way to make it easy to do so.

      Now think about a bank. Physical access to the money is controlled and verified and audited.

      Employees at a bank are NOT allowed to do whatever is easier for them. They do NOT prop open the secure doors.

      If they do so, they are fired.

      So why would Facebook be any different? Because people can SEE when the doors to the money are propped open. But they cannot SEE the network access.

      You are wrong.

    13. Re:"extensive measures" taken... by lgw · · Score: 1

      If you can't make it easy for employees to do what they desire, you're just not very good at your job since that is the job of IT. Keep trying to change human nature and you'll simply fail at security and be seen as an asshole control freak by the people who do the useful and productive work at the company. Especially if those people are driver devs and kernel hackers, who might make a sport of subverting controls.

      It's just a nasty combination of hubris and laziness that leads IT to try to change the users, rather than changing their own systems, to achieve security.

      You might also notice in a bank, the big vault door is actually propped open during working hours because it's a PITA. Inside there's usually a simple cage door, quick to open as any normal locked door, that provides security while employees need access. Funny that. Of course, most modern setups actually have an automatic cash dispenser for the teller that spits out specific amounts, making it even easier for the teller to do his job. That's good security: a system devised to make it quite easy for the teller while providing needed security.

      The recommended solution for the smoking area, BTW, is to have the secure perimeter outside the smoking area - a fence or somesuch - so that it doesn't actually hurt security for the door to be propped open during the day, since you can't really get to it from the street. Again, enabling the worker to do what he wants to, while providing needed security.

      --
      Socialism: a lie told by totalitarians and believed by fools.
  2. and yet... by slashmydots · · Score: 2

    And yet at my company I can't get the GM and president to let me implement a basic computer and security competency test for all new hires that use computers. I'm CIO by the way.

    1. Re:and yet... by Charliemopps · · Score: 1

      I agree with your GM and President. Not all positions require computer competency, and computer competency should have nothing to do with your security measures.

    2. Re:and yet... by Fire_Wraith · · Score: 1

      Not speaking to your suggested control in particular, I do think that in general the non-technical/MBA world, especially the older ones, simply do not take network security seriously enough to properly evaluate the tradeoff in risks to dollar figures. They see it as a cost center (which it is), but do not properly appreciate just how bad things can get. It's our job as network security professionals to make the case for this, but it's not easy when a lot of them still seem to have a view of the computer as a magic box. This is even before we get to the problem that good security can be tough for large enterprises even when you can shovel truckloads of money at it government style.

      I think that it's going to take a lot more breaches, and fired CEOs, before enough of it gets taken seriously.

    3. Re:and yet... by pr0fessor · · Score: 2

      We have yearly mandatory security training and testing for all employees. They like to recap recent incidents that hit the news and point out how following proper security policies could prevent a similar incident.

    4. Re:and yet... by bloodhawk · · Score: 2

      Why would you implement a test? We have mandatory basic security training and awareness, breaches of security policy range from being forced to attend the training again right up to being sacked, testing just shows what everyone already knows, I.e. that most people are unaware of security.

    5. Re:and yet... by slashmydots · · Score: 1

      We had a new salesman take a fake fedex cryptowall e-mail and forward it to the entire company. Explain that then.

  3. On a positive note... by MagickalMyst · · Score: 5, Funny

    Perhaps there will be some 'unexpected improvements' in open-source drivers for nVidia chipsets in the near future...

    --
    Political correctness is really just herd psychology pushed by insecure people who desperately seek social conformity.
    1. Re:On a positive note... by MagickalMyst · · Score: 1

      They won't; in and of themselves.

      The employee information was likely targeted in order to gain account information to access to critical systems with sensitive or confidential data.

      --
      Political correctness is really just herd psychology pushed by insecure people who desperately seek social conformity.
    2. Re:On a positive note... by Anonymous Coward · · Score: 2, Funny

      "Open source the driver or we... we'll make this public information PUBLIC"

    3. Re:On a positive note... by WaffleMonster · · Score: 1

      I can't see how employee's SSN's will improve the open source codebase.

      Why are SSN's permanent life-long secret codes only the people who represent them should know?

      I think in aggregate if everyone's SSN were made public it would prevent a vast sea of morons from relying on them for purposes for which they were neither intended or suitable and everyone would be better for it.

    4. Re:On a positive note... by Zontar+The+Mindless · · Score: 1

      Where I live, everyone's national ID number is a matter of public record.

      --
      Il n'y a pas de Planet B.
  4. Re:And here we go. by Anonymous Coward · · Score: 1

    Two California-based corporations suffer a big breach within a month of one another? Pelosi and Feinstein must be getting as wet as their old crotchety selves possibly can>

    *vomits*

  5. the real question by slashmydots · · Score: 2

    The #1 question on everyone's mind is, does the data leaked indicate that mangement at Nvidia also admits that the shield is a pointless, overpriced device with no market? I'm reeeeally dying to know. They're like a tablet mixed with a PSP mixed with a steambox but worse than all 3. I can't imagine anyone there is too happy with it.

  6. Was it time travellers? by wonkey_monkey · · Score: 2

    NVIDIA has reportedly been breached in the first week of December

    Bit of a mixed up tense there. Makes it sound like time travellers did (are doing) it.

    --
    systemd is Roko's Basilisk.
  7. Awareness by Gliscameria · · Score: 1

    Are these hacks happening more often or is it a mixture of actually catching the breaches now and more reporting on the breaches?

    --
    X
  8. my problem by Anonymous Coward · · Score: 1

    with things like this is that one finds out that enterprise IT admins frequently store passwords in plaintext... it's the only way they can tell that your new password is "not sufficiently different" from the previous one... eg at a previous employer that starts with A, i had a password like "App7!S@uCE". admittedly i was being lazy, but being told that "App7!S@uC3" was too similar (change last char from 'E' to '3', in this example) would only be possible if they were storing the plaintext rather than a hashed string. or they "hashed" it by applying something clever like ROT13... no, the guilty company was not Apple... anyway, moral of the story is use different passwords everywhere.

  9. Hrm... by Charliemopps · · Score: 4, Interesting

    Does anyone else find it ironic that every time one of these breaches happens... all the employee and costumer data walks right out the door. But their source code? Propitiatory corporate secrets? Oh, those are locked up tighter than a drum.

    It's not hard to prevent these "hacks" or "Leaks" they just only chose to actually spend money to protect what's valuable to them. After their employees or Customers personal info is out there, they throw some money at a Credit monitoring service and pretend like that means anything at all? What did it cost them? $1 a user? LOL

    We need federal liability laws. The feds do not need to dictate what they need to do to secure data like they've requested. They know, and we know that's a joke. The law will be out of date before it even takes effect. Simply make them liable for $100k per persons personal data they leak. They will quickly just flat out stop storing the data in the first place and we'll all be better off.

    1. Re:Hrm... by radarskiy · · Score: 1

      In the Libertarian paradise they have eliminated all external costs so you don't need to distinguish between a free market an an unregulated market. Therefore we can cut laws and regulations completely out of the problem!

  10. Re:And here we go. by zlives · · Score: 1

    Damn you Kim Jong, when will your terrorist activities be stopped. I beseech the military industrial complex and congress to provide whatever resources NSA needs to do what ever they want to do... for the children

  11. Security rings? by Midnight+Thunder · · Score: 1

    With all these issues, I am wondering whether beyond the firewall to the external network, internal portions of a corporate network should be firewalled too. For example HR related data should be on a sub-section of the network protected by its own firewall. I would imagine the chances of breaching multiple firewalls being low, unless the penetration into the network is either done by an insider or someone who has been able to lay low on the network for a while?

    This may already be the case in many organisations, but I don't know enough about security specifics?

    --
    Jumpstart the tartan drive.
    1. Re:Security rings? by ledow · · Score: 2

      That's covered by basic permissioning, surely?

      If the user you got access to has access to HR data, they have access to HR data. Anything else in the way is merely a hindrance (to you, and an intruder).

      But if you compromised a server and used them to get administrator access on the storage arrays, pretty much it doesn't matter what you've got in-between.

      The real solution, I think, would be proper encryption. But even there, you have the problem of key management that doesn't just hand out keys to the servers when they request them.

      Fact is, yes, it might be sensible to cordon-off a few of these things. You already have to have PCI-DSS stuff on isolated networks (to simplify their requirements), but it doesn't seem to stop this stuff. I know schools that, for years, separated off curriculum and admin - i.e. it was impossible for a child to compromise the deputy-head's login in the classroom and use it to get access to personnel data). I still stick to that mentality and move things to be physically apart wherever possible but nowadays it's considered old-hat and all the separation is virtual - VLAN's and permissioning on the servers and storage arrays.

      You put barriers in between but this kind of attack is more akin to the boy with his finger in the dyke. Your users (in HR, or Finance, or wherever) need that tiny hole to be open for them, and so piggybacking in on their connection automatically gives you access to this stuff.

      The problem is that intrusion detection / prevention, really, is nothing more than checking EVERY transaction back and forth and that's a huge undertaking, slow, requires lots of equipment, and has to be constantly updated by people as threats appear. It's not an easy problem to solve. Even governments are having slips with their classified data, because even though the networks are supposed to be completely isolated, it only takes one guy (malicious or not) to be compromised for that barrier to have a gaping hole in it for him to do his job through.

  12. I guess they missed the most important thing by Groo+Wanderer · · Score: 1

    It looks like they forgot to turn the Windows Firewall to "on" and set the Internet Security Zone to "High". That should prevent all hacks, right? /me ducks

  13. Re: And here we go. by aergern · · Score: 1

    Troll

    --
    Tell me what you believe...I'll tell you what you should see.
  14. Once it's out in the wild, it's game over! by Tasha26 · · Score: 1

    When you read that Stuxnet was an NSA/Israel creation and every month you get drip fed news about NSA's true illegal/terrorist side (like finding ways to hack popular email servers or backend links of cloud storage) and just now, cracking VPN services, you have to ask yourself this: "Who has opened Pandora's box? Who deserves to suffer from it [first]?"

  15. Comment removed by account_deleted · · Score: 1

    Comment removed based on user account deletion