US Homeland Security Employees Locked Out of Computer Networks (reuters.com)

Dustin Volz, reporting for Reuters: Some U.S. Department of Homeland Security employees in the Washington area and Philadelphia were unable to access some agency computer networks on Tuesday, according to three sources familiar with the matter. It was not clear how widespread the issue was or how significantly it affected daily functions at DHS, a large government agency whose responsibilities include immigration services, border security and cyber defense. In a statement, a DHS official confirmed a network outage that temporarily affected four U.S. Citizenship and Immigration Services (USCIS) facilities in the Washington area due to an "expired DHS certificate." Reuters first reported the incident earlier Tuesday, which a source familiar with the matter said also affected a USCIS facility in Philadelphia. Employees began experiencing problems logging into networks Tuesday morning due to a problem related to domain controllers, or servers that process authentication requests, which could not validate personal identity verification (PIV) cards used by federal workers and contractors to access certain information systems, according to the source.

Comments

Security focused

[Fire_Wraith]

DHS is the primary government agency responsible for protecting the country's civilian infrastructure, including the internet and computer networks. I feel so much better knowing that they're so good at keeping their own systems secure, that even their own workers can't access them.

Re:Security focused

[Anonymous Coward]

What's so insecure about denying access due to an expired certificate? Isn't that an example of security measures working as expected?

Re:Security focused

[sycodon]

I work for one of the largest Defense companies in the nation. In the last year we have had two major network outages. One related to provider issues and the other related to firewall changes gone bad.

This shit happens. Creating/Managing/Upgrading huge networks like this a very complicated and delicate task.

Re:Security focused

[Notabadguy]

Plot twist, the government doesn't manage their own networks anymore, for a while now they've been getting rid of military trained personnel and replacing them with civilian contractors.

Keep in mind that Department of Homeland security != Military; the Department of Defense (military) is a separate department. And many DHS personnel are unskilled, uneducated workers. TSA and all the security theater is part of DHS. This news article is as special as "Exxon gas station cashiers locked out of computer network."

Baggage handlers, X-Ray viewers, clerks, and even janitorial staff proudly introduce themselves in public as "I'm with Homeland Security." It sounds a lot better than "I'm a baggage handler at the airport."

Re:Security focused

[GeekBird]

On the systems I administer, we have an alert that checks the certificate expiration once a day, and alerts it plenty of time to get it renewed.

But a lot of people don't do that, they just mark it on a calendar somewhere, or expect the certificate issuer to notify them. For the latter, often the contact email is to a person no longer with the organization, or in a different role, so it is ignored. That's why my current $Employer insists that certificate emails go to an email list for a group, rather than just to one person.

It wouldn't be quite as funny if it wasn't so very common.

Doing more with less..

[lionchild]

I think I'd like to take this opportunity to point out that this is what happens as we do more and more with IT on less and less staff. While I understand sometimes we think of IT as a cost-center and not a revenue generator, it probably needs to be thought of as more like a utility; because without the lights, water, phones...and internet, you can't do business very effectively these days.

That being said, this happens more and more. Someone is responsible for renewing certificates, but as we renew them for longer and longer periods, that means we simply start to forget about them. Then with the certificate issuer sends out an notification to that IT staffer who used to do that, but was 'right sized' a year and a half ago...no one gets the email. So, the certificate expires and this happens. Same song, different, louder verse, apparently when it happens to DHS, and likely more embarrassing.

Bottom line: Doing more with less, isn't always in everyone's best interest.

Re:Doing more with less..

[TechyImmigrant]

That being said, this happens more and more. Someone is responsible for renewing certificates, but as we renew them for longer and longer periods, that means we simply start to forget about them.

An alternative viewpoint is that this is one of the ludicrously bad failings of PKI. Requiring someone to remember to do an infrequent and short task at a point 1 or 2 years in the future, or the whole system collapses when they forget or leave or get booted. We could fix (I.E. delete and replace) PKI and this specific failure would not happen, so the overworked IT staff can go back to deploying Windows NT patches.

Re:Doing more with less..

[EndlessNameless]

Requiring someone to remember to do an infrequent and short task at a point 1 or 2 years in the future

Bullshit.

I could write a PowerShell script in maybe 10 minutes that will list all of the computers in the domain, connect to them, and check for expiring certificates. I can get a reminder in advance---90 days, 30 days, a week, whatever I want. All I have to do is one thing: understand my job.

Alternatively, some tools (like Nessus, which is FOSS) have audits which automatically check for expiring certificates. They can be configured to email a report, and you can notified every day/week/month if you have expiring certs.

This is a stupid, incompetent failure. You can build or buy a tool to avoid this problem very easily. Compared to using passwords, the only reasonable complaint is that you require decent sys admins.

Cert expiration == not a surprising cause

[ErichTheRed]

The interesting part of the article isn't about who is affected, but the "certificate expiration" aspect. I've recently started doing the legwork necessary to learn about public key infrastructure (for our company's internal consumption) and have found that there are 3 prevalent camps out there:
- Developers who just say "here's my credit card, VeriSign, make my customers' browser address bars turn green."
- Admins who get just enough of a PKI background to make the certificate errors go away, then run away screaming -- or worse yet, had it implemented a decade ago by a consultant and have NO CLUE how it works or how to fix it
- Auditors who just say "lock icon, green browser windows, check. Congrats, you're PCI compliant."

For something so critical like certificates, there really is a dearth of resources out there that isn't aimed at hardcore security programmers or one of these three groups. Cert expirations have figured prominently in many outages -- Azure had a partial outage a few years ago because of that very reason. I'm seriously considering writing a "PKI for non-dummies" series of blog posts or something because the amount of misinformation out there is scary!

Re:Stop the presses! Someone in IT fucked up!

[Archangel Michael]

All News is fake depending on who is reporting and who is the reader/viewer.

Kind of like "Planned Parenthood doesn't use public funding for abortion services". Technically "accurate", but really not even close to being accurate.

A woman comes in for an abortion, but gets six other "tests" and diagnostics done. Pregancy test, Pap smear .... etc. All those other "tests" are paid for by government money, none of which are part of the actual "abortion" procedure. Since that Planned Parenthood clinic provides mostly abortion related services, they are "government funded" and would fold if they didn't get any other funding. They subsidize the Abortion with federal monies, using loopholes.

Technically it is "true" that PP doesn't use federal dollars for "abortion". Realistically it is fully subsidized procedure using loopholes. Both sides are considered "alternative facts" by the other side. And the reason we can't have civil discourse about anything any more.

And watch this get modded "Troll" since I used the inflammatory "Planned Parenthood / Abortion" example by people who can't actually debate the actual topic.

First Rule of IT

[IWantMoreSpamPlease]

Always install a backdoor.

For times like this.... ...and for "other" times, as needed.

Re:NOT NEWS!

[TheGratefulNet]

timecube guy.

4 simultaneous days.

something along those line. details are unimportant.

Re:Stop the presses! Someone in IT fucked up!

[Archangel Michael]

1) Yeah, which is why I did it. Inflamatory subject using rational thought. Imagine that.

2) If you ran a Hamburger Restaurant and said that you're not a "Hamburger" place because only 33% of your business was "Hamburgers", would you be telling the truth, or telling a lie?

You sell Hamburger, fries, and a soda, and count that as 1/3, 1/3, 1/3 you'd technically be correct. But everyone in the world would understand that you're in the "hamburger" business. Right?

3) So, yeah, Abortion procedure itself is only 12%. Technically correct using the metric as applied by PP, which is VERY similar to how I explained above. How about you ask the question differently. How many people visiting PP are there to get an abortion vs "other" services they offer. BTW, those "other" services are fairly limited to .... being an abortion provider.

They claim to be "women's health" but they do not offer Prenatal anything ... except abortion. They don't offer Mammograms like they continue to claim (as in NONE). They don't do .... a whole lot of things related to "women's health". (Fact Check article is technically correct: Lies, Damn Lies and Statistics)

BUT I have an idea, I've suggested a number of times. Planned Parenthood can keep all the funding it gets now, if they stop providing abortion or referrals for abortion. Lets see how much of their Business is Abortion. I'll bet it is like a Hamburger shop not being able to actually sell burgers. Just fries, cokes .... And yes, this would settle the case, once and for all. Their primary business is abortion. They can't exist without it.