Slashdot Mirror


Target's Internal Security Team Warned Management

david.emery writes "According to this story, Target's own internal computer security team raised concerns months before the retailer lost millions of credit card numbers in an attack. (Quoting a paywalled story in the Wall Street Journal.) Target's management allegedly 'brushed them off.' 'At least one analyst at the Minneapolis-based retailer wanted to do a more thorough security review of its payment system.' This raises a more general question for the Slashdot community: how many of you have identified vulnerabilities in your company's/client's systems, only to be 'brushed off?' If the company took no action, did they ultimately suffer a breach?"

39 of 236 comments (clear)

  1. Posting anonymously for obvious reasons... by Anonymous Coward · · Score: 5, Interesting

    Yes, there are horrible security flaws where I work. Things as basic as changing passwords on a regular basis have been brought up repeatedly, and the answer is always, "we can't make people do that", or "that's something to keep in mind for the future, but we have more important things to worry about"

    1. Re:Posting anonymously for obvious reasons... by ackthpt · · Score: 5, Informative

      Yes, there are horrible security flaws where I work. Things as basic as changing passwords on a regular basis have been brought up repeatedly, and the answer is always, "we can't make people do that", or "that's something to keep in mind for the future, but we have more important things to worry about"

      I've worked at two kinds of places - one, where it was pretty much as you described. The second sort was, upon orientation you are given your accounts and access and told they are your responsibility to use discretely and to notify the appropriate support should you even suspect they have been compromised. Failure, in the second case, was ground for discipline or termination of employment.

      Guess where things went more smoothly and security issues seldom elevated to crisis.

      --

      A feeling of having made the same mistake before: Deja Foobar
    2. Re:Posting anonymously for obvious reasons... by MightyMartian · · Score: 2

      Generally whomever I worked for took my security warnings to heart (the first production Linux server I ever built was put in place as a mail relay for a Windows-based mail server's SMTP daemon to prevent joe jobs and overcome some nasty security vulnerabilities, with the management's approval).

      I can tell you that other kinds of warnings have historically not been heeded. I had a boss who decided that because Windows 2000 Server supported disk mirroring on IDE drives, he didn't need to invest in decent hardware RAID. I warned him repeatedly that software RAID is better than nothing, but certainly not as efficient nor as effective as hardware RAID and that SCSI drives were infinitely superior on heavy load servers like our SQL and Exchange servers. Well, guess who was bitching about Outlook being a dog, and he just got really pissed off when I told them that at least the db server should be moved to appropriate equipment.

      --
      The world's burning. Moped Jesus spotted on I50. Details at 11.
    3. Re:Posting anonymously for obvious reasons... by Desler · · Score: 5, Insightful

      You do realize that making people change their passwords all the time simply leads to people using weaker passwords or writing them down, right? This type of policy though up by some self-proclaimed security expert amongst the IT monkeys almost always leads to worse security than not. And you don't even need to take my word for it:

      The downside of changing passwords is that it makes them harder to remember. And if you force people to change their passwords regularly, they're more likely to choose easy-to-remember -- and easy-to-guess -- passwords than they are if they can use the same passwords for many years. So any password-changing policy needs to be chosen with that consideration in mind.

      https://www.schneier.com/blog/...

    4. Re:Posting anonymously for obvious reasons... by Desler · · Score: 2

      Inplementing a boneheaded change password policy is not going to make your users act better. You are simply going to make no difference or make it worse.

    5. Re:Posting anonymously for obvious reasons... by plover · · Score: 4, Insightful

      Interesting that you should mention "changing passwords on a regular basis" as a "horrible security flaw". Have you considered that changing passwords generally introduces more risk than it guards against, and doesn't actually have an effect on most actual hack attacks?

      The attacker strikes with whatever credentials he finds, whenever he finds them. The second step of an attack is to create a separate back-door, so that if the first password is changed he's back in anyway. And how does an attacker find credentials? When someone's entering them, which includes changing them, or if someone's handling them. There is often a case when you have people who can't remember their newest recently cycled password who call the Help Desk. The phone drone resets it to something like "ForgottenPassword#1", then voicemails the chump with the temporary password. If a hacker's able to listen to their voicemail, he simply calls in a phony forgotten password request and it's Winner, Winner, Chicken Dinner!

      So what does changing the password every 30 days actually protect against? I suppose if you wrote the password on your blog, then in 31 days you're safe. Of course, if you wrote the password on your blog, I don't think password rotation should be your highest priority for fixing your security issues. Do you honestly think hackers have machines that can crack passwords in 31 days, but not 30? Either he can crack it in an hour or less, or he likely can't crack it at all and won't bother trying.

      Changing passwords periodically was only a good idea when there was one password shared by many people, and you had to exclude your former colleagues. But those days ended back with moats and longbowmen on the castle walls. In these modern days of electronic passwords that are never shared, it's a ritualistic holdover with negative consequences.

      --
      John
    6. Re:Posting anonymously for obvious reasons... by DarkOx · · Score: 2

      I hate people who insist that password changes are not a good thing. Look very very few organizations have proper identity and account management.

      Password rotation at least closes the hole of former employees still having access at some point in the future.

      Everyone's password ends up in a log file somewhere some time, in plain text just laying around. Usually its because they are in a hurry and enter it in a user name field. Password rotation ensures this password will at least at some point no longer be valid.

      People choose crappy passwords even when you don't force changes. Someone may well get access to an account by slowly guessing likely passwords over a long period of time. Password rotation reduces persistence of access to said account.

      All of these should be covered by other controls yes, but sometimes any given control can fail, especially in an organization where there is anything less than total maturity around IT processes (most) someone misses a step one time, and things can go terribly wrong. Good security is about layers. Changing of password is one layer. If someone claims to be a security professional and says you don't need at least some password rotation policy. They are a know nothing; who is just repeating someone equally incompetent's blog post to you and you should fire them.

      --
      Repeal the 17th Amendment TODAY! Also Please Read http://www.gnu.org/philosophy/right-to-read.html
    7. Re:Posting anonymously for obvious reasons... by Anonymous Coward · · Score: 2, Insightful

      Password rotation at least closes the hole of former employees still having access at some point in the future.

      No. If former employees still have access, that means the network admin folks are incompetent or the off-boarding procedure is broken.

      When an employee terminates, their account should be disabled. Problem solved.

      There should never be any anonymous or independent accounts that can cause damage (e.g.,, an FTP box could have anonymous access if nothing confidential is kept there, but it should never be allowed write access).

    8. Re:Posting anonymously for obvious reasons... by Penguinisto · · Score: 4, Insightful

      Ditto here... once you make the employees know that their screw-ups will end up costing them, they tend to not screw up as much, and tend to report things much, much faster should something go awry.

      That said, the Target penetration wasn't directly caused by a Target employee/user - the bad guys snuck in through a contractor that was given network access that they should have never had. This was more due to lazy architecture/vlan partitioning than it was $random_employee with a bad post-it note habit.

      If anything, the network admins should be facing the barrel before anyone else, followed very closely by most of the security admins, if not simultaneously (excepting the guy who shouted the warning and those who demonstrably supported him; that dude should be promoted post-haste.)

      --
      Quo usque tandem abutere, Nimbus, patientia nostra?
    9. Re:Posting anonymously for obvious reasons... by davidhoude · · Score: 3, Informative

      The point in changing passwords isn't to change user behavior. It is to ensure that any leaked credentials do not stay valid indefinitely.

    10. Re:Posting anonymously for obvious reasons... by l0n3s0m3phr34k · · Score: 5, Insightful

      At my job, I have three different VPN tokens, and at one time had at least 30 different passwords all over the globe I had to use...ours forces changes at various times, some are 30 days, some 90, some never...depending on the system. RSA admin software had a PIN too. We usually just keep it all in a spreadsheet. If you can't remember a single password...but you also need the Active ID token too. We potentially have deep access into the air line reservation system, although that system is so insanely complicated and cross-platform good luck finding anything of worth haha.

      It's kinda backwards in a way. Retail is always a huge target, the bigger the company the bigger the score. From a security design viewpoint, the "backend" and the "financial" systems should have been physically separated at all times, using some encrypted EDI to exchange whatever (inventory, overstock, per piece price, etc). The credit card terminals should have been "payment only" and not loaded down with all their SHIT like "cash back?" "cure cancer?" "are you sure?" "join our rewards / store card" and wtf other messages I have to tap on your stupid touchscreen a million times just to pay you. Some of them even have ads on them.

      Soon, Walgreens, CVS, Dollar whoever...the more sophisticated we make these terminals where our card touches their system, the more exploitable they will become. It's the slow feature creep, the "we need to upload new ad images at 2:50AM" by developers in a far-off land...pushed forward by managers who just want "shiney bright things" that make us give up even more information, waste our time more, and provide little real actual benefit.

    11. Re:Posting anonymously for obvious reasons... by cusco · · Score: 5, Interesting

      I've worked in the physical security field (cameras, key cards, alarm systems, etc.) for the past eight years, and can tell you that Target's HVAC vendor is in no way unusual. I know of a large security vendor that uses the same username/password combination on every every customer that they ever touch, nationwide, and at most of them they are administrators on the security server. At a lot of them they have remote access.

      --
      "Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
    12. Re:Posting anonymously for obvious reasons... by l0n3s0m3phr34k · · Score: 2

      The best is to fire them on a Friday. Cancel their access right before rolling up on their cube "yeah, we need to talk..." with security nearby when it eventually escalates. We've found that Iraq/Afghanistan vets with massive PTSD work the best, as they have the best "De-escalation" responses.

      I got fired from my last job on my day off, but got re-hired three weeks on a different team. We ran into a slight problem with my new email address because my old ID hadn't been archived yet, whatever that actually means. I'm assuming I could chase down some AD admin or there's some "form" somewhere on line to fix it all...but honestly my new email is better, but it gets a shit-ton of spam, that is totally insane...it was coming in other people's names but now I'm feeling the effect of having a four-teller . two letter .com, and it's like watching the internet "tubes" drain their digital sewage all into my inbox.

    13. Re:Posting anonymously for obvious reasons... by PrimaryConsult · · Score: 2

      The flaw in password lockout schemes that lack a timeout is that anyone can lock out anyone's account. I can imagine someone hammering every member of "Domain Admins", "Helpdesk Staff", etc with three fake attempts, and by the time anyone has realized it, it will be difficult to even find someone who can unlock the accounts.

  2. customer service portal by ironicsky · · Score: 5, Interesting

    Years ago I worked for one of the two big American cable companies currently merging. I identified a security flaw in the public facing side of their customer service portal, essentially giving access to all the config files, which contained admin credentials in plain text. I proposed simple solutions, like not allowing directory listings of folders, among others.

    They shrugged it off, and to the best of my knowledge, last year the vulnerability was still accesaible

    1. Re:customer service portal by Penguinisto · · Score: 2

      In such cases if you implemented the simple security solutions without telling them they would be none the wiser.

      Sometimes that breaks things.

      I worked at a certain software firm that had a nasty habit of requiring that a few service account user/pass sit right in plain text in an XML file on the front-facing web servers, else the whole thing wouldn't work. Their MSFT kool-aid drinking habit aside, I always found it hilarious that they preached security so hard, yet left such stupid flaws in place for many years (and many versions). Yelling about it got us approximately nowhere, and implementing a fix on our own w/o going through a very strict review and certification process meant that we'd violate at least a half-dozen laws (the industry they work in is regulated very heavily in that regard...) ...and yeah, the vuln is still there to this very day, every time I get bored and go look for it (from the outside, with a simple web browser).

      Let's just say that I avoid doing any business with any and all known clients of that company.

      --
      Quo usque tandem abutere, Nimbus, patientia nostra?
  3. I small lawsuit... by achbed · · Score: 2

    This has all the hallmarks of the beginnings of a civil suit for negligence, and if it can be proven that the flags were raised based on actual break-ins and were ignored, possibly criminal negligence. The only place in Target I'd want to be right now is in their legal office - they're gonna be putting in some overtime soon.

  4. Every single company by Tony+Isaac · · Score: 5, Insightful

    There are security concerns in every company, without exception. Obviously, even the NSA itself had inadequate security!

    Yes, many times security concerns are brought up, and brushed off. But this is not necessarily an indication of a problem. Every security risk must be weighed based on the likelihood of occurrence, and the severity of the impact, should it occur. Many of these calculations are inexact, and must be based on incomplete information.

    Should Target have protected themselves better? Probably. But hindsight is 20/20. The difficult part is to anticipate the problems that might occur, without crippling your organization through impossibly tight security.

    1. Re:Every single company by gtall · · Score: 2

      Which is very comforting to punters who must trust a company with their credentials in order to do business with it.

      One solution to mitigate risk is insurance. Companies should have to pay for security insurance. They cannot prevent every break in, but insurance companies have ways of evaluating an pricing risk. Customers would then at least have a shot at being made whole again.

    2. Re:Every single company by James-NSC · · Score: 2

      I’ll second that. When approaching management with security concerns, many of us fall short on being able to properly communicate with management regarding risk. While it’s helpful that management, specifically upper management, deal with risk every day the downside to that is, you have to present your risk to them in terms they can understand. Using the formula of:

      Cost of failure * rate of failure = total cost of failure is actually detrimental to this approach, most notably because the rate of failure for an undiscovered/undisclosed security defect is quite small and yields a total cost of risk that is well within norms for most companies.

      What you need to do is familiarize yourself with the upper management, specifically those through which you report up to the CEO, and understand the types of risk they deal with and – more importantly – the total costs of failure they find acceptable. Then, when approaching them – just by way of example - prepare a report which demonstrates this specific risk in terms they both understand and with a gravity that they appreciate. Never say “we could be hacked, it would be awful”, instead “when this defect is eventually discovered (include citations on the rate of remote network probes/scans), the resulting security breach will cost us $X to resolve, further (citations are handy) as this has been in the news lately, expect additional fallout in both news cycles and social media. Instead of facing $X in known risk, by investing $Y in prevention we can address this issue and improve (insert impact on project/product they are personally invested in).”

      Lastly, never leave the rate of risk ambiguous – never leave it at “might, may, could or worse still, one in a million” – always represent those uncertainties with math: number of remote attack attempts over time. If your perimeter is anything like mine, it will be read by management as an eventual certainty and *not* like something that can be safely ignored as an unlikely “storm of the century” type event.

  5. Close ties to the FBI by silas_moeckel · · Score: 2

    There is there problem they are fairly computer illiterate, I've dealt with many FBI computer forensic specialists whatever's that are dumbfounded by a .tgz, unix line endings. Hire out of the Secret Service they understand computers.

    --
    No sir I dont like it.
  6. Why are you surprised? by benjfowler · · Score: 2

    Stupid cookie-cutter MBA pindicks.

    They were the jocks in school who got ahead because of their aggro and ego, but not their brains.

    Guess what? They're now our bosses.

  7. You'd Be Amazed by The+Other+White+Meat · · Score: 5, Interesting

    Years ago I worked for a government IT department. A vendor wanted us to try out a product. The device plugs directly into the Internet connection, and monitors every packet, in real time, looking for strings matching an array of string that you provide. We ran queries against our internal databases, and compiled a list of SSNs and CCNs. The vendor programmed that data into their device, which from what I can tell used an FPGA to perform deep packet inspections.

    We expected that we might see maybe an email every week or two where someone accidentally sent that kind of information.

    First hit occurred 12 seconds after turning the device on.

    Second occurred .47 seconds later.

    Etc. Etc. Etc.

    Within an hour, we had overrun the quota on the network directory where we were logging this data.

    We found hundreds of separate systems that were transmitting this kind of data without authorization. We were planning a massive internal sweep to find and fix them all, when the following came down from management:

    Shut it down. Remove the device. Destroy all logs, emails, EVERYTHING. Offer the vendor a payment in return for signing an NDA. All employees required to sign secrecy docs (unenforceable at that level of govt, but still.)

    I believe this is how the acronym SNAFU came into existence.

    --

    --- Generation X: The first generation to have SIG lines inferior to their parents... ---
  8. Basically, yeah by Anonymous Coward · · Score: 4, Interesting

    I got my first job in the industry due to that sort of screw-up. A network administrator was "let go" following a server crash and loss of months' worth of data. The backup system hadn't been working. I was hired shortly thereafter to get things back in order.

    Now, that would be the end of the story, except that I was good friends with this administrator. The embarrassing subject of his dismissal didn't come up for about three years, but when it did, and I mentioned my surprise at a fairly intelligent guy allowing backups to lapse for that amount of time, he dug up an e-mail he'd sent to the president of the company, cc'ing the head of HR (who was more or less running the show, for some reason), pointing out the various problems they had - their "server," an old workstation, had been running for two years on a three-month evaluation copy of Windows Server 2000, there were no backup tapes working, and so on. The only excuse they could have had was that the backup thing was buried in a page-long list of serious issues. But when it blew up in their faces, they pinned it on the closest available peon. Assholes.

    1. Re:Basically, yeah by nobuddy · · Score: 5, Informative

      So... where do I know you from?

      You could have described my one and only firing ever, to the word.

      Me: "Boss, Beancounter- this backup system is broken and needs to be fixed. here is a cost breakdown for the fix and a loss analysis for failure to fix. It is genius and incorporates existing links and hardware to minimize cost and implement offsite backups for all sites!"
      Boss: "Shut up and go fix a printer somewhere."

      Fast forward a year- major crash of a POS server. Loss of customer records, $300,000 and 6 months predicted to be spend reconstructing the database from paper records.

      Boss: "You are fired for letting this happen."
      Me: "...."

  9. Happens all the time by dave562 · · Score: 2

    This is a frequent occurrence. I used to get upset about it. These days I have seen enough of these exact type of situations blow up that I am content to document my observations, report them to the appropriate people (always a direct supervisor), and then move on with my life. When things blow up, I am covered.

    Situations like this are why, although I understand security, I will never work in a security position. There is too much risk and liability, and not enough support.

  10. Blown Out of Proportion by organgtool · · Score: 2

    I'm sure that Target will address the issue by firing all of the management that brushed off the security researcher's concerns and will promote that security researcher to the head of a new task forced aimed at increasing their security and give him a huge pay increase (and maybe a pony).

  11. this is what you get with outsourcing / contractin by Joe_Dragon · · Score: 2

    When you have lot's of outsourcing / contracting / subcontracting they don't want to pay the costs of doing stuff right no they want fast / cheap.

  12. Re: Changing Passwords by Anonymous Coward · · Score: 2, Interesting

    Places where I've worked that users were required to change their password regularly invariably had the same password but with an incremented number at the end every time they needed to change the password. This allowed them to remember it more easily, be effectively meant they were using the same password.

    The more stringent that the password requirements become, the more likely it is that users are going to start writing them down somewhere or trying to come up with workarounds so that they can remember them. And in turn, you have another security issue.

    Everywhere I have worked has also have a review of brute force password hacking attempts. :-)

  13. It's okay to write them down. by khasim · · Score: 3, Insightful

    You do realize that making people change their passwords all the time simply leads to people using weaker passwords or writing them down, right?

    As long as you keep them in your wallet then writing them down is fine.

    You're MUCH more likely to be aware when someone steals your wallet than when someone steals your password. So keep your passwords in your wallet if you cannot remember them.

    Similar for home systems. Keep them safe at home. Criminals breaking into your home to steal stuff are not USUALLY going to be looking for a piece of paper with your passwords on it.

    1. Re:It's okay to write them down. by Ecuador · · Score: 2

      So, they steal your wallet and you quickly find out. Now what? They can log in, you can't without the password. Genius. No, the password changing policies are stupid. One of my banks requires a new password every 1-2 months, which also has to follow specific guidelines (guess how much less secure that makes the password by reducing the possible password space) and does not allow you to enter one similar to the previous passwords. So far I've had to call 3 times for their reset process... For my WiFi I have a password that is 3 very long foreign names. That is one kind of password that can't be brute-forced or guessed, but very few password policies allow something like that. For example I tried it for skype and it got rejected for lack of security, while a 7-letter lower case english word plus the number 1 was deemed fine! Go Google!

      --
      Violence is the last refuge of the incompetent. Polar Scope Align for iOS
  14. Typical Navy Response by Anonymous Coward · · Score: 2, Interesting

    As a former US Navy nuclear engineer, I informed management of material and procedural problems related to the nuclear reactor plant on board the USS La Jolla on a weekly basis. Have you ever gone to your boss with a technical manual that perfectly explains the "unexplainable problem" he's having, have him brush you off, and less than a week later that problem destroys a major system, causing millions of dollars in damage and endangering the entire ship? I have. I'm pretty sure none of my complaints were ever addressed except on the one or two occasions where I threatened to bypass management and complain to a newspaper. That's pretty standard Navy leadership. When you're dealing with a culture where everyone starts at the bottom, the best and brightest leave, and whatever's left gets promoted, that's the kind of technical management you get.

  15. They get their own network. by khasim · · Score: 2

    Then they may have a few fixed passwords / login's that they need to give out to all the people in the field it's much easier to have fixed one then giving each field tech own log in's that they may not even need day to day or even working at target all the time.

    So they get their own network that does not touch the production network.

    Probably just a *DSL/cable from a local ISP.

    With a firewall that you control. Heavily locked down. No need for them to hit Facebook from the HVAC, is there? No need for inbound access from 99.9% of the IP addresses out there, is there?

    Then paint it and label it and make sure no one else can touch it. Use super-glue on the ports.

    1. Re:They get their own network. by Joe_Dragon · · Score: 2

      and then some cost cutting cutting yoho says why does the HVAC need it's own network cabling and or DSL/cable line? or says we are not paying for cable when we get free directv / dish demo accounts and there is no DSL in the area.

  16. Re:Oh boy... Here we go... by nobuddy · · Score: 4, Informative

    document, document, document. And keep copies where you can get them once you are frog-marched out of the building wearing the scapegoat collar.

  17. Re:in this cases it may be out side vendors / cont by Anonymous Coward · · Score: 4, Interesting

    Which is a perfect example of incompetence.

  18. Re:but when you work with HVAC vendors who sub wor by khasim · · Score: 4, Insightful

    Someone please tell me what HVAC contractors could possibly need to do on a corporate network.

    *raises hand* ooh! ooh! Pick me! Pick me! Been there! Done that!

    Two things:
    1. It's not that they need access to the CORPORATE network. It's that they need access to the INTERNET so that the machinery can report back to the vendor when something starts to go wrong. That's usually in the service agreement. The sooner detected the sooner fixed without problem.

    2. For managers who like to look at stuff. There is usually an internal web server on the HVAC. You go there and it displays things like the temp and the humidity and blah blah blah.

    Thus, dumb managers (I've dealt with them) want them on the corporate network. It's easier for everyone.* Including the crackers who are looking for these exact vulnerabilities.

    *Security people are not included in this definition of "everyone" in this case.

  19. Looked for, found, reported, was fired. by The_Other_Kelly · · Score: 2

    I was the responsible IT manager, over all devs. admins, ops and security.

    Reviewed all contracts and implementations, upon taking over the job.

    Discovered some seriously, bad stuff.

    Developed plan to *quietly*, discretely, repair over short time period.
    "Rebury the bodies"

    Turned out the responsible party was the CEO's favorite, "baby shark".

    Got cardboard boxed. Out day after board presentation.

    So it goes.

    Interesting point:

    All of those devs, techs and security people who moan about the lack of management support?
    How many of you have ever supported or somehow defended *any* manager who tried to help you, to do the right thing?

    Speaking personally, I would guess ... None of you. "Not my problem" attitude, up and down.

    Maybe you have all been luckier.

    --
    (R)ule in Hell or (S)erve in Heaven [R]?
  20. Re:but when you work with HVAC vendors who sub wor by TarPitt · · Score: 2

    Wouldn't some big company like Target have someone on staff who knows how to firewall off a network just for the HVAC? Huh? Huh?

    They probably have several people who can do that. It requires some expertise but not a lot.

    Of course they have people who CAN do that. The better question is - do any of those people have the political clout to require Target to spend money and inconvenience managers and "essential" vendors to prevent a "theoretical" security attack.

    --
    If your children ever found out how lame you are, they'd murder you in your sleep