TN BlueCross Encrypts All Data After 57 Disks Stolen

Lucas123 writes "After dozens of hard disk drives were stolen from a leased facility in Chattanooga, potentially exposing the personal data of more than 1 million customers, BlueCross decided to go the safe route: they spent $6 million to encrypt all stored data across their enterprise. The health insurer spent the past year encrypting nearly a petabyte of data on 1,000 Windows, AIX, SQL, VMware and Xen server hard drives; 6,000 workstations and removable media drives; as well as 136,000 tape backup volumes."

Comments

I am impressed

[WindBourne]

Most insurance companies these days, are far more concerned with getting bonuses to the executives.

Re:I am impressed

[pla]

To bad its all protected with the same password.

But no one would ever guess "damnyouratbastardstohellihopearabidbadgerchewsyourballsoff" as the password for such a well loved and respected institution as a medical insurance company... So no worries!

/ that, or "bluecrossispants".

Re:I am impressed

[Enry]

12345

Re:I am impressed

[Lucas123]

That's amazing. I've got the same combination on my luggage.

Re:I am impressed

[Enry]

Idiot ;)

Re:I am impressed

[compro01]

what exactly is a "bid badger"?

Re:I am impressed

[mallyn]

A bid badger is the person who does the shill bids at an auction; the buddy of the seller for the purpose of driving up the price of that vintage osciloscope that I want so badly

Re:I am impressed

[WindBourne]

Thank you. Never heard that one before.

Re:I am impressed

[ArsonSmith]

Which is a win for all. Executives can't get bonuses if there isn't a decent amount of income to the company, there can't be a good amount of income to the company if there aren't high revenues. There can't be high revenues if there isn't a supply of something people want that can be produced for at least slightly less than they are willing to pay for it. They wont be willing to pay for it if it's cost is higher than it's value to the individual.

Everyone wins.

Re:I am impressed

[geekoid]

Executives can't get bonuses if there isn't a decent amount of income to the company,"

false.

Re:I am impressed

[SnarfQuest]

So that they don't forget it, they have printed it on sticky notes and attached it to all the monitors in the company. Take that, you data thieves!

Re:I am impressed

[JordanL]

i believe the correct concatenation was "rabid badger"

Re:I am impressed

[ArsonSmith]

well, i guess if you're an executive for a money counterfeiting organization. Otherwise that bonus has to come from somewhere.

You may be thinking of elected officials. They're the ones that get paid without having to show results.

Re:I am impressed

[darrylo]

Why? Some states require that companies notify people when their data is stolen, as well as sometimes requiring identity theft protection (e.g., credit reports or alerts) or somesuch. This can get pretty expensive, and so it's probably cheaper to just encrypt everything. They're not being altruistic -- they're saving money. It wouldn't surprise me if some executive got a bonus for saving the company money ...

Re:I am impressed

[Drugmath]

You seem to be forgetting the financial companies who were so fucking broke we had to give them money or the world would end. You know, the same companies who took our money, turned around and gave billions in bonuses to their employees, presumably for doing such a wonderful job

Re:I am impressed

[ArsonSmith]

And why we should have much lower taxes and smaller government and insure the government is never able to do a bailout like this ever again. Perhaps instead of taxes we could have bailout bonds issued so people could feel they were doing the right thing by buy the bank bailout bonds if they felt it was the right thing to do.

Encrypting data alone might be useless

[Zakabog]

This entire effort might be useless if they're not using good encryption. Is there one master passphrase to bypass all of the encryption? Also, they make no mention of how they plan to prevent physical theft of data again just that 'Well this time I put a password on my data, take that thieves!'

Re:Encrypting data alone might be useless

[ccguy]

It's ROT13. Good luck with those 136,000 tapes we've got.

TNBC chief of security.

Re:Encrypting data alone might be useless

[qxcv]

"Good luck, I'm behind SEVEN ROT13s!"

Re:Encrypting data alone might be useless

[Samantha Wright]

"When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl."

Re:Encrypting data alone might be useless

[Thud457]

"Good luck, I'm behind SEVEN ROT13s!"

demonstrably incorrect.

Re:Encrypting data alone might be useless

[Anonymous Coward]

Your missing the point... the reporting requirements are different if the data is encrypted and at the end of the day that is all that really matters.

Re:Encrypting data alone might be useless

[Unequivocal]

Yeah really. I thought the punch line to this story would be "..and then they promptly lost the private key that encrypted all that data."

Encryption doesn't solve much on its own - it's the process that surrounds the encryption (key and passphrase management, decryption environments, etc) that matter just as much..

Re:Encrypting data alone might be useless

[darrylo]

"Good luck, I'm behind SEVEN ROT13s!"

demonstrably incorrect.

I thought it was something like ROT26 or somesuch.

Re:Encrypting data alone might be useless

[White Yeti]

People communicate in Welsh?

(sorry...I've been busy)

very lame

[Anonymous Coward]

"We searched the country and were unable to find another company that has achieved this level of data encryption," Michael Lawley, vice president of technology shared services for BCBS, said in a statement.

He certainly did not search very hard. Less than 1PB encrytpted, we do more than that every single day. And I doubt we are unique.

Correct Response

[inglorion_on_the_net]

It is a pity that the data was stolen before adequate protection was put into place, but it seems to me TN BCBS took the right steps afterwards:

1. They sent out alerts to those affected, both current and former members

2. They now encrypt all their stored data

Of course, this will not prevent all possible leaks, but at least it shows they are taking protection of their customers' data seriously, and have put in serious work to protect that data. I wish more organizations did that. Way to go, BCBS of Tennessee!

You should be impressed

[somersault]

"I know I already shit on the floor, but I'm wearing a diaper now so it's all good!"

Re:You should be impressed

[rbrausse]

"I know I already shit on the floor, but I'm wearing a diaper now so it's all good!"

where is badanalogyguy?

so you're saying that one mistake (data loss; floor shitting) will render every countermeasure (encryption; diapering) invalid? nah, I don't think so. The insurance company handled the data loss quite competent - they disclosed it early (afaik) and implemented a regime that will make future data losses much harder.

Re:You should be impressed

[somersault]

It wasn't a perfect analogy, but I don't think they should be congratulated for closing the gate after the horse already bolted. They're just doing what they should have been doing all along. Really, they shouldn't let anything even get stolen.

Re:You should be impressed

[datapharmer]

no, it makes data losses just as easy as they were before. It prevents data theft as the records are now (theoretically) protected. Without proper off-site backups they are still screwed if someone steals their drives again.

Re:You should be impressed

[CraftyJack]

The counterpoint would be Sony:

"Oh, there I go again! And again! Well, I didn't see that comi-And again! Wow, this is quite a string of bad luck!"

Re:You should be impressed

[Sulphur]

"I know I already shit on the floor, but I'm wearing a diaper now so it's all good!"

where is badanalogyguy?

so you're saying that one mistake (data loss; floor shitting) will render every countermeasure (encryption; diapering) invalid? nah, I don't think so. The insurance company handled the data loss quite competent - they disclosed it early (afaik) and implemented a regime that will make future data losses much harder.

Does the insurance company have insurance for their data?

If the jelly does not cover the peanut butter on the PB&J pizza, then the PB gets hard and difficult to eat.

Re:You should be impressed

[rbrausse]

oops, thanks for the correction. Typical German that we use 'regime' for describing both a terror ~ and a medical treatment ~ :)

Re:You should be impressed

[geekoid]

" they shouldn't let anything even get stolen."

way to blame the victim.

Re:You should be impressed

[somersault]

Aren't the real victims their customers? If you have millions of customers' data, you should have enough resources to give it physical protection.

subject

[Legion303]

Well the new customers whose data hasn't already been stolen will be happy to hear it, I guess.

Re:subject

[Almandine]

Yes, I was worried until I saw that the breach happened in 2009.

$6 million?

[daktari]

I'm by no means a security expert but isn't $6 million a bit excessive for the effort?

TFA says "The company said it spent more than 5,000 man-hours on the encryption effort, which encompassed about 885TB of at-rest data." That equates to around $1200/hr. Perhaps I should become a security expert.

Re:$6 million?

[belthize]

I wouldn't take the $6M and 5000 man hours as directly coupled. The actual press release says:

BlueCross invested more than $6 million and 5,000 man-hours in the data encryption effort, which included:

- 885 Terabytes of mass data storage
- 1,000 Windows, AIX, SQL, VMWare and Xen server hard drives
- 6,000 workstation hard drives and removable media drives
- 25,000 voice call recordings per day
- 136,000 volumes of backup tape

The 5000 man hours may only reflect actual labor and not reflect all the hours of planning/scheduling etc. What ever hourly rate for labor double it for overhead, the cost of a person is about twice their salary, at $100/hour that's $1M in labor. Another 500K in planning. I have no clue what software they used but I'm pretty certain it wasn't a single package. Each system may well have required a different package + licenses + contractor time from the vendor. For example they may have had to out source the voice call recordings to who ever provides their phone system. I kind of doubt they slap all the recordings onto a single box and mass encrypt.

They're a very distributed organization so there's going to be a *lot* of duplication of effort, they may have had to do the phone bit at hundreds of sites.

I don't know if it could have been done for $3M or if $6M actually represents a relatively reasonable price compared to a lot of the $XXX Mllion dollar utter failure projects. It strikes me as fairly reasonable considering the scope of the problem and usefulness of the result (assuming it's not a $6M whitewash).

Re:$6 million?

[tecker]

Assuming 100% markup profit margin over baseline (common practice really) were looking at a baseline cost of $3 mil.

Now we need to factor in an encryption scheme that works across Windows, AIX, etc with enterprise support backing it up say $1.2 million to licence for all servers and locations (seem low but hey) and we have $1.8 million to spend.
Now we gotta pay people some prices to do that work so lets say $.5 million (500,000) so about $100 per man hour (bout right) and we have $1.3 to spend.
Now pay the electrical company for all that processing time (depending if they had THEM process it or they did it on their servers) at about $.5 million and we have $.8 million (800k) to explain.
Throw in some training for a few Ks to ensure the techs know how to handle the system lets say 100,000k for that (ouch! hey that is specializations ya know) 700K to go.
Maybe a little software rework (even if it wasnt really necessary) for another 100k and we have 600K to explain.
Opps forgot the "maintenance contract" which is often 10% of the sale price so 600k and lookie there, 6million blown pretty quick.

Thanks for shopping.

Re:$6 million?

[daktari]

True. I might just hang on to my current skill set a bit longer then.

Re:$6 million?

[elsurexiste]

Other people did a breakdown before me of the costs. Lucky thing: it's expensive to start but cheap to keep it, just remind people every 6 months that they should use the software. Oh, and check very often that you can restore your backups: there's nothing funny in working your whole weekend because an encrypted backup has locked itself in.

Re:$6 million?

[geekoid]

Obviously, and no it isn't.

Re:$6 Million to check a checkbox?

[MikeB0Lton]

Trolls... Good luck implementing BitLocker on entire VMFS datastores. Not everything is based on Windows Vista/7.

Re:$6 Million to check a checkbox?

[GameboyRMH]

Damn I would have personally gone around and done it on all their computers for $50k. I'd even pay my own airfare.

And then they can pay me again to switch to TrueCrypt when BitLocker falls off the Microsoft upgrade treadmill :-P

"Safe route"

[_0xd0ad]

So, they're locking the barn door after the horse has bolted...

dozens of hard disk drives were stolen from a leased facility in Chattanooga, potentially exposing the personal data of more than 1 million customers

The data is gone... and now they're encrypting.

Re:"Safe route"

[MysteriousPreacher]

I don't think the barn door saying means what you think it does. It suggests pointless action taken after the event. The original data was stolen but encryption to hinder future theft of data seems sensible.

Re:"Safe route"

[cavreader]

Even with the best commercially available encryption if someone steals the hardware storing the encrypted data they have all the time in the world to try and access it. The disks were in the possession of a 3rd party at the time of the theft so a security audit of their premises and security procedures might be in order to help raise awareness and prevent future incidents.

Re:"Safe route"

[isorox]

So, they're locking the barn door after the horse has bolted...

dozens of hard disk drives were stolen from a leased facility in Chattanooga, potentially exposing the personal data of more than 1 million customers

The data is gone... and now they're encrypting.

They've locked the barn dor after 1 horse bolted. There's hundreds more left in the barn.

Re:"Safe route"

[sys_mast]

Your analogy, while not perfect has a valid point. However, remember that they now have a new horse in that barn. (all the customers that have since the data loss) What would you say about the farmer that lost his horse, got a new one, and still leaves the door open?

Perhaps the lesson here should be to all the IT people (does anyone in IT still read slashdot?) take this type of preventive action BEFORE you have data stolen. (yes, i know it's really up to the C-something-O to fund and order such an operation)

Re:"Safe route"

[MysteriousPreacher]

Definitely no arguments there.

Cheap, but what about ongoing costs?

[plsuh]

$6 million is pocket change to a company that has $5.2 billion in annual revenue. However, the true cost is really higher, as encrypting everything means that things like disk corruption are no longer repairable, lost passwords can't be reset without losing data, and the like. It'd be interesting to see just what the ongoing costs are.

That said, I would like to compliment Tennessee BC/BS for doing the right thing, in spite of it costing money.

--Paul

Re:Cheap, but what about ongoing costs?

[blueg3]

How is disk corruption less repairable when you encrypt?

The lost-passwords problem is already well-solved for decent systems.

Re:Cheap, but what about ongoing costs?

[maxume]

If you use the password to encrypt the key, you can store a copy of the key somewhere else.

So if the password is lost, to reset, you grab the key from the escrow and encrypt it with the new password.

Re:Cheap, but what about ongoing costs?

[Himring]

$6 million is pocket change to a company that has $5.2 billion in annual revenue.

Right, but any money spent on IT is a waste to the stuffed shirts, until something blows up, which, inevitably, gets them off the fence. Telling the COs in a meeting, "our worst possible downtime with the current allotted budget might be as bad as 3 days," makes them all look at each other with satisfaction and approval, seemingly, ok with being down 3 days in theory. Then, after 3 hours of downtime, they are talking about outsourcing all of IT for 10 times the amount of budget they barely allowed that caused the downtime....

Short of it:
Pre-disaster: IT should be cheap if not free.
Post-disaster: IT will get all the money it needs, but a new crew.

Re:Cheap, but what about ongoing costs?

[horza]

I think he meant less recoverable rather than repairable. Which is true, you can't simply dump the disc and extract the fragments by hand if necessary if encrypted.

Phillip.

Re:Cheap, but what about ongoing costs?

[Lieutenant_Dan]

My personal experience with a couple of mainstream commercial enterprise solutions, is their data recovery tools leave a LOT to be desired and seem to only work for us about a third of the time. Features and management tools get the attention; auditing and recovery are after-thoughts in most products.
In a few instances where we had to engage a data recovery service, they charge quite a bit more when they find out that they're dealing with an encrypted disk (i.e. when we're going after a specifc folder or a bunch of files)

Anyways, it got to the point where one of my clients is now looking at expanding their archiving solution rather than spending the cash (and time!) to attempt to recover data on encrypted media.

Re:Cheap, but what about ongoing costs?

[bill_mcgonigle]

Which is true, you can't simply dump the disc and extract the fragments by hand if necessary if encrypted.

If you have a properly layered solution (e.g. LUKS), you can open the crypto volume, and then dump the unencrypted block device for manual recovery.

Re:Cheap, but what about ongoing costs?

[mrheckman]

I work for a company where data is subject to HIPAA (United States' Health Insurance Portability and Accountability Act - a law whose provisions also address the security and privacy of health data). Our data has been encrypted -- at rest and in transit -- for years. The loss of private health information, like what Blue Cross did, is a serious crime under HIPAA and subject to major fines (in this case, at least tens of millions of dollars, probably, given how large the breach was). The initial cost to encrypt and any ongoing expenses will be pocket change compared to the fines that Blue Cross is potentially facing, with increased fines for repeat offenses.

In practice, once you have disk-level encryption set up for data at rest, and network encryption for transmitted data, your on-going costs are pretty minimal. There's some central administration and IT support to administer and maintain the tools, and your ISO needs to do some compliance reviews and risk assessments to make sure that things stay encrypted, but after installation they are pretty transparent.

Blue Cross should have been doing this all along. Nothing like a large fine to focus the mind.

Adage

[SirDice]

In the Netherlands we have a adage that seems fitting, "De put pas dempen als het kalf al verdronken is.". Which roughly translates to "Closing the well after the calf already drowned.".

Re:Adage

[Random Destruction]

The equivalent american english idiom is "closing the barn door after the horse has bolted"

Hold on...

[Syberz]

They have the personal details (health records, bank info, addresses, etc.) of millions of people and they just now decided to encrypt the data? WTF?

Re:Hold on...

[Sloppy]

It sounds reasonable on the surface, since people think of drive theft as very exceptional and something you can physically defend against. But then .. these people never had a drive fail and then RMAed it? Am I supposed to believe that when there's a mechanical failure and they're unable to erase the drive, they destroy it rather than mailing it back to a vendor or manufacturer?

!first post

[orange47]

jryy vg jbhyq unir orra svefg cbfg vs vg jrera'g sbe rapelcgvba bireurnq.

leased facility = cloud so this is what you get fr

[Joe_Dragon]

leased facility = cloud so this is what you get from going to the cloud the data can be in a place that can range from a nice data center to a small room in a office building. Also the people ruining the cloud can just have real low prices and then sell data to the highest bidder.

Re:leased facility = cloud so this is what you get

[Anonymous Coward]

Leased facility != cloud. In a leased facility, you can find out the operational conditions and the level of physical security. You can make them part of the lease contract if you care enough. You can't do that in a cloud.

Lets congratulate them for doing the right thing

[damn_registrars]

... even if it is far too late. And of course, the customers will pay for the cost of the failure, plus the cost of the fix. The company made a bad choice, and the consequences of that bad choice will be born by .. the customers. The executives will still get their usual multimillion dollar "performance" bonuses as if nothing was ever wrong.

It works much better

[nedlohs]

If you encrypt it before it gets stolen.

what?

[damn_registrars]

Most insurance companies these days, are far more concerned with getting bonuses to the executives.

You don't honestly think that the executives will end up with smaller bonuses as a result, do you? We all know that isn't how this game works.

The company will cover these costs by raising premiums and/or reducing payments. It is very likely that the executives will see larger bonuses after this, as a self-congratulatory measure for "proactively correcting the situation".

Re:what?

[RenHoek]

That's part of the fun right?

I mean, as a customer, first you get screwed over by having your medical records out in public. Then the company gets fined and leverages that fine on its customers, thusly getting screwed a second time. Finally, costs are incurred for getting up to standards, and guess who is paying for those costs?

Re:what?

[davester666]

There. All done encrypting every hard drive and backup tape.

Um, does anybody remember the password we used? Surely somebody wrote it down?

Re:what?

[Darinbob]

So I'm getting screwed three times, without ever once getting flowers or dinner out of it!

usless

[Charliemopps]

If you've got the drive... you have unlimited attempts to crack it. Someone with a couple of video cards and a few days on their hands and their encryption is pointless.

Re:usless

[horza]

Easy slip to say a few days rather than a couple of billion years.

Phillip.

Re:usless

[Charliemopps]

It's the government. They "encrypted" the drives. What do you think that means? Do you really think they did it properly? Or do you think they Bought some licenses form Symantec and clicked next next next? Randomly generate passwords? Seriously?

If they were properly secure in the first place, the would not need encryption. Encryption is for data that leaves your network. If physical media is leaving your network, you're doing it wrong.

Irrelevant in the long-run, but...

[John Napkintosh]

These drives were likely part of various RAID volumes. Doesn't that mean they're pretty well useless outside their hosts? Is someone really going to go to the level of forensic data recovery to elevate from property theft to identity theft? That stuff isn't cheap, so the ROI is probably going to be really low.

Re:Medical data can't be encrypted

[TheRaven64]

My mind is boggling at the level of ignorance and stupidity in that post. Even a moment of thinking would let you realise that this can't possibly be correct.

Re:Medical data can't be encrypted

[elsurexiste]

+1. The only problem is that I usually recognize people because of their sigs, not their user names...

Standard Procedure?

[DarthVain]

Is it just me, or shouldn't this be standard fscking procedure for companies dealing with sensitive information such as medical and financial records?

Re:Standard Procedure?

[qwijibo]

Should be, but generally isn't. Security costs money, and most companies have been in a cost cutting mode for years. Security is one of the first things to go since it's invisible until you're compromised.

Re:Standard Procedure?

[DarthVain]

Generally I think most companies don't need it. Some only need the basics. You got my personal information, or credit cards? Just securely encrypt those sources. Sure some might slip out here and there, but you won't lose your whole database of 300,000 customers or whatever.

I just mean if your a bank, financial institution of some description, or someone that handles my medical information, get on the encryption boat and set sail. Seriously. I mean it is one thing if someone gets my VISA number... its usually protected anyway.

However your right it is a cost thing. And until companies are held responsible in court financially they will not take it seriously. Once some CEO's start getting the boot for allowing a catastrophic lawsuit to take place, change will happen.

Re:Medical data can't be encrypted

[compro01]

You should read the rest of his post history.

Re:$6 Million to check a checkbox?

[jimicus]

And then they can pay me again to switch to TrueCrypt when BitLocker falls off the Microsoft upgrade treadmill :-P

Firstly, as someone else has already said, not everything is based on Windows.

Secondly, I cannot think of a product I should be less inclined to use than TrueCrypt to deal with such a problem. Reason I say this is simple - in every large business you always have the occasional helpdesk call to reset a forgotten password - usually when someone's just come back off holiday. How exactly are you going to deal with the problem when the answer to a helpdesk call for a lost TrueCrypt password is "please send the laptop in for reimaging"?

What solution?

[Lieutenant_Dan]

Looked around the stories including their "infographic", not clear what they are using and how they've implemented it.

Do servers have pre-boot enabled? How did they change they operational processes? Are these HW-encrypted drives? What is the failure rate on the process?

Details like this are important. As it stands, they spent the cash and a lot of time, but no indication that they've implemented it properly. I wouldn't feel much safer.
5,000 hours is nothing to be honest for even a mid-size company. That's 2-3 techs working a whole year on it. Big deal. They could be just sitting in front of the monitor watching the progress bar.

Re:Knee Jerk Reaction?

[mlts]

What is ironic that any enterprise tool has encryption built in if it was made in recent times:

The EMC devices have Powerpath encryption for LUNs. Someone hacks the SAN, nothing available on the server other than trashing the LUNs.

IBM storage arrays check if they can boot off a key server, and then unlock their encrypted drives in hardware. If this isn't enabled, AIX has EFS (different from Windows's EFS) to ensure that only the user with the right key can attach a directory.

Linux has so many tools, there is a supported solution somewhere. LUKS, TrueCrypt, EncFS, gpg, various userlevel tools accessed via FUSE, PGP, etc.

Windows has plenty of tools. BitLocker, EFS, third party tools like PGP, TrueCrypt, and document level tools like LockLizard or Microsoft's IRM.

Backup programs can encrypt data to tape using hardware encryption and SPIN/SPOUT SCSI commands, or the backup client can deduplicate on its end and send encrypted stuff up, so the backup server is not the weakest link.

Applications can encrypt on a table basis in almost all RDBMS programs. Store the value and a nonce as a salt. This way, even if a table had repeating values, an attacker couldn't discern what repeated and what didn't.

Everything supports two-factor authentication, so even though RSA Security may have had issues, having a token and a password is better than nothing. If someone doesn't want SecurID, there are plenty of other two factor products, such as VASCO's stuff they OEM to Blizzard, SOE, and eBay.

The encryption tools are there, and likely sitting around ready to be configured. It will take some time making a recovery scenario, because key management can be hairy, but if done right, encryption will be pretty much set and forget.

Yeah but still...

[AlfaMike]

They should get some credit for spending money encrypting their data but it's still another case of a company that only does the right thing AFTER shit hits the fan.

how are they encrypted?

[markhahn]

when one of their machines reboots, where does the key come from? such sites usually spend as much money as possible on the theory that mauve is better, which in this case probably means FC SANs. but at which level does the encryption happen? and doesn't disk encryption just mean that you need to take the enclosure or client box too?

Re:leased facility = cloud so this is what you get

[geekoid]

And now, Samuel L. Jackson will read a line from his up coming movie: "English Lesson"
Punctuation motherfucker, learn it.

And the master key

[ThatsNotPudding]

is written on a post-it stuck to the monitor of the secretary for the CEO.

Barn analogy

[Nyder]

It only took them 57 horses getting stolen before they decided to lock the barn door.

good job! way to keep on top of things.

6 million for 5000 man hours?

[Lando]

So they are spending 1200 dollars a man hour? Total machines seem to be about 6000, so each machine is costing a grand to encrypt? Seems pretty expensive.