Slashdot Mirror

← Back to Stories (view on slashdot.org)

US State Department Suffers Worldwide Email Outage (usatoday.com)

Posted by EditorDavid on from the speaking-of-government-shutdowns dept.

An anonymous reader quotes USA Today: The U.S. State Department's email system underwent a worldwide outage Friday, affecting all its unclassified communications within and outside of the department. The system was fully restored by Friday afternoon [after 12 hours], said a State Department official briefed on the incident who was not authorized to speak publicly and requested anonymity.

It was not clear what caused the early morning outage, but spokeswoman Heather Nauert told reporters it was not "any external action or interference."

69 comments

  1. Logical conclusion: by Archtech · · Score: 4, Informative

    '... spokeswoman Heather Nauert told reporters it was not "any external action or interference."'

    So just incompetence, then.

    --
    I am sure that there are many other solipsists out there.
    1. Re:Logical conclusion: by omnichad · · Score: 4, Insightful

      If they don't know what caused it, I don't really know how they can fully rule out external actors being involved.

    2. Re:Logical conclusion: by El+Cubano · · Score: 4, Informative

      So just incompetence, then.

      I came here to say just that. You might be surprised at just how bad it is.

      In most cases, government IT is a dumpster fire. I could go on at length, but the absolute best example of it is the failed FBI Virtual Case File project. IEEE did a very in depth write up of it several years ago.

      If I recall correctly, the objective was that the FBI tried implement a system after 9/11 that would allow its field agents to collaborate more effectively than phone calls and non-government email (apparently most field agents used things like Yahoo! and HotMail because they either couldn't get government email accounts or they sucked so badly). After years of ill-defined and constantly changing requirements and $300+ million spent, the project was terminated as a failure and the government actually got nothing out of it. Go read the IEEE story on it for more details.

      The problem is that this sort of thing happens constantly with projects of all sizes and scopes. Of course, if you've worked in corporate IT, you will recognize that similar things happen there as well.

    3. Re:Logical conclusion: by MoarSauce123 · · Score: 4, Funny

      I bet on Windows updates going haywire.

    4. Re:Logical conclusion: by Anonymous Coward · · Score: 0

      The shit show continues

      They are probably running with a skeleton IT crew, just like the rest of the administration runs with gaping holes. But nothing to worry about I'm sure.

    5. Re:Logical conclusion: by Neuronwelder · · Score: 1

      When you don't pay people a fairly, and cut back on the skeleton crew - what do you expect?

    6. Re:Logical conclusion: by Anonymous Coward · · Score: 0

      So just incompetence, then.

      I came here to say just that. You might be surprised at just how bad it is.

      Maybe not directly individual incompetence, but incompetence arising from a dysfunctional organisation. And large organisations always have some degree of dysfunction. Of course, a well-run one will have orders of magnitude less of it than a poorly-run one. Large government departments aren't particularly well-run in the best of cases.

      The problem is that this sort of thing happens constantly with projects of all sizes and scopes. Of course, if you've worked in corporate IT, you will recognize that similar things happen there as well.

      It is fascinating to see how and why that sort of thing goes awry. E.g. why can't you provide email by saying "use any client you like, as long as it speaks IMAP4 properly with our servers", then provide a web-based client for good measure? Provisioning can be made similarly straight-forward, iff you're the kind of bureaucrat that can keep his pants on while making policy.

    7. Re:Logical conclusion: by Salgak1 · · Score: 1

      And those of us inside TRYING to put the dumpster fire out are stymied at every turn. I have a program that refuses to produce ANY documentation, "because we're agile". Another one had lost all track of a bunch of equipment. Until I was assigned to close out the program. . .and found it was all at the last location listed on the 4-year-old documentation.

      Security patches get waived by programs "because we don't risk downtime". Even if the patching was planned to occur during scheduled downtime

      As one of the nuns at my elementary school was known to remark. . . it makes Baby Jesus cry. . .

      And as for the FBI and the Virtual Casefile. . . .they had also bought an entire warehouse of gear to support it. Which sat there for years, in boxes, until they finally killed the program. And then PAID someone to take the unused gear off their hands. . .

    8. Re: Logical conclusion: by Anonymous Coward · · Score: 0

      IT Intern: "MX Record? Da fuqs dat do?"

    9. Re:Logical conclusion: by rbrander · · Score: 1

      I was just in a municipal government, but the dysfunctionality really was amazing.

      I know I'm an anti-authoritarian personality, so most problems look to me to be caused by authoritarianism, but I'm pretty sure about this one. I watched it grow for 20 years.

      Every large corporate (and this goes double for "municipal corporations" and other governments) IT shop that goes back decades started off as a *mainframe* shop, with all IT totally Under Control. As they took over the PCs, everything there had to be brought totally Under Control, which included things like Only One Application for every need, followed 2 years later by Everything Must Be Microsoft.

      And they could back this up, and hold up the rest of the corporation for any sum they demanded for the simplest applications, because IT was *their* turf and no other department could do their own, hire their own contractors, nothing.

      Whenever they developed a service or application, NOTHING could be done until EVERY ramification had been explored and the perfect One Application For All Needs Of All Stakeholders was developed. This made every application a huge project. Indeed, it soon became clear that they hated small projects and those would either not be done, or be made large.

      I retired feeling that until the power of centralized IT departments is broken, that IT in large corporate bodies will be like the Soviet central-command economy: hopelessly broken and incapable of producing the simplest things even at huge cost mark-ups, with frequent spectacular failures.

      This isn't about government-vs-private, it's just centralized-vs-flexible. I compared notes with an engineer at our phone company, which sells all sorts of services now, TV and Internet, etc. They do have a business imperative that government services do not: they have had to constantly innovate to develop and grow these new businesses lately. So their IT department was forced to change away from the centralized mainframe shop of yore. It's a chaotic mess, alas, because they never acknowledged they'd have to do this, so it all kind of slipped away on them with no planning for it. But, while it's a chaotic mess, they are functioning and making money; even in the worst case scenario for control-vs-chaos, the chaos is not as bad as the over-control.

    10. Re:Logical conclusion: by Anonymous Coward · · Score: 0

      IIRC, the gist of the IEEE article was that there was no one on staff at FBI who had enough IT skills to understand what was going on, while suits insisted on meeting impossible deadlines with impossible change requests, resulting essentially in fake code produced by probably imho low-skill coders. The author went on to point out that part of the problem is that the govt doesn't have a job classification for the proper IT skills. The classification for that job is the same as say for a computer operator or such that doesn't have the background to oversee a large IT project. I doubt this has anything to do with pay.

    11. Re: Logical conclusion: by Anonymous Coward · · Score: 0

      It's not a problem if the intern asks that. He's there to learn, after all. And it's no more than expected that a kid fresh out of university knows nothing of value. All he was in there for was the expensive bit of paper, after all.

      (You can blame this on the "wisdom" that managers automatically dismiss anyone from consideration who doesn't have formal education. We get the situation that any formal education will do, as long as it's formal. Education entirely optional, of course.)

      It is a problem if, say, the IT across the campus gets centralised, so all computers are now classified as one of three kinds of windows and nothing else can be supported. (Curiously, the "CS" department didn't mind. The "industrial design" department didn't either, even though it put their graduates on a disadvantage within the industry. The physics department, however, did. A lot.) It's an even worse problem when they flattened out all the subdomains because supporting them in the same vendor's collaboration server (it's not actually an email server, though many pretend) was "too hard". That's the professionals at work. On a university world-renowned for its strong technical slant, or so it would like to think.

    12. Re:Logical conclusion: by Anonymous Coward · · Score: 2, Insightful

      If they don't know what caused it, I don't really know how they can fully rule out external actors being involved.

      It's actually exceptionally simple, but I'll explain it in detail for you.

      Say you are the sysadmin in charge of a server, which goes down one day and so far all you have discovered is that pressing the power button appears to do nothing at all.

      Within a minute or two of you discovering this, your boss is asking for a comprehensive detailed report on what is wrong.
      You tell him the server won't power up, but you aren't sure why.
      Boss asks if it was Russian hackers, to which you look at him like he's a moron for even asking and you say "No, I just said the server isn't powering on"

      A moment later your boss sends out a company wide email stating that the server administrators do not know what caused the outage, but have ruled out external actors being involved.

      Reporters are similar to your boss in the above made-up situation. They don't care about details and tech stuff and how things work. Their eyes totally glaze over.
      The "report" also has the primary goal of being short and concise, without a lot of qualifiers and details, just simple and to the point at the cost of being accurate in ways most reading the report won't care about.
      They also likely don't care about attributing words to you that you did or did not say.

      This all combines to get the result you see and are confused by, and is almost always what happens when your words get filtered through other people like this.

    13. Re:Logical conclusion: by davester666 · · Score: 0

      Well, Trump was in the server room, and was quoted as saying "This doesn't look so hard to do."

      --
      Sleep your way to a whiter smile...date a dentist!
    14. Re:Logical conclusion: by __aaclcg7560 · · Score: 1

      Or, more likely, they're switching everyone over to Office 365 and decided to migrate all the accounts at the same time. That's usually a recipe for disaster. The three-letter agency that I work for is taking months to complete the migration, resulting in minor issues for some people and a whole lot of bitching by everyone else over the new interface.

    15. Re:Logical conclusion: by haruchai · · Score: 1

      '... spokeswoman Heather Nauert told reporters it was not "any external action or interference."'

      So just incompetence, then.

      Looks like Trump's plan to stop the leaks is working

      --
      Pain is merely failure leaving the body
    16. Re:Logical conclusion: by haruchai · · Score: 1

      ".they had also bought an entire warehouse of gear to support it. Which sat there for years, in boxes, until they finally killed the program. And then PAID someone to take the unused gear off their hands"

      Not just government who messes up like that. Way back BP decided to implement WiFi at all their sites in North America & decided to buy ALL the access points before the project was ready to roll. Except this was when 802.11b was the standard and just before 802.11a/g was ready, which happened after the purchase but long before they were about to start installing the APs.
      In the end, they bought all new equipment and never opened the 802.11b gear. My boss at the time - we were doing outsourced IT support for them - visited their Houston Westlake campus in 2007 and saw a storage room with 15000 boxed APs.

      --
      Pain is merely failure leaving the body
    17. Re:Logical conclusion: by CaptainDork · · Score: 1

      Fortunately, we took care of this back in 2009.

      Meet the Military's Cyber-Security Forces
      Posted by CmdrTaco on Thursday January 28, 2010 @11:41AM from the can't-wait-to-see-the-ad-campaign dept

      --
      It little behooves the best of us to comment on the rest of us.
    18. Re:Logical conclusion: by Dutch+Gun · · Score: 1

      "Hey, why am I on this mailing list? Remove me please."
      "Yeah, me too"
      "Why am I on this thread?"
      "What mailing this?"
      "Everyone, stop responding to all"
      "Autoreply: Mark G is out of the office until next Tuesday"
      "Lol, you first!"
      .
      .
      .
      Bedlam!"

      --
      Irony: Agile development has too much intertia to be abandoned now.
    19. Re: Logical conclusion: by Anonymous Coward · · Score: 0

      "What does this button do? I can't resist pressing big buttons. General Kelly wrestled me to the floor when i tried it in the White House. That guy's such a joker."

    20. Re:Logical conclusion: by Anonymous Coward · · Score: 0

      You barbarian. I bet you are a fucking user aren't you. You would have the inmates running the asylum. It would be Bedlam! You uncircumcised BYOD heathens cannot be trusted to care for yourselves. It is for your own good. Keep the cork on the fork.

      Sincerely,
            IT

  2. Massive coverup! by Anonymous Coward · · Score: 0

    Obviously the Russians did it

  3. This is what I can guarantee... by bogaboga · · Score: 1, Insightful

    The U.S. State Department's email system underwent a worldwide outage Friday, affecting all its unclassified communications within and outside of the department.

    There will be mention of Russia in relation to this unfortunate event, partly as an effort to hide our [inherent] incompetency, sadly.

    1. Re:This is what I can guarantee... by omnichad · · Score: 1

      This is the Russian^H^H^H^H^H^H^H Trump presidency. They will blame Ukraine.

    2. Re:This is what I can guarantee... by Anonymous Coward · · Score: 0

      Well, your guarantee isn't worth the electrons it took to transmit it. They've already said it wasn't due to any external action or interference.

    3. Re:This is what I can guarantee... by Anonymous Coward · · Score: 0

      Uh, well, ok, then? Glad to see you could find a way to inject Russia into yet another totally irrelevant thread, bogaboga!

    4. Re:This is what I can guarantee... by Anonymous Coward · · Score: 0

      There will be mention of Russia in relation to this unfortunate event

      Aha! Look up... Beat ya this time!

  4. BUTTERY MALES! by Anonymous Coward · · Score: 0

    MAGA

  5. Uh, so.... by sconeu · · Score: 4, Funny

    Hilary's server is still up, right?

    [DISCLAIMER: I dislike both parties]

    --
    General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
    1. Re:Uh, so.... by Anonymous Coward · · Score: 1

      No it was taken down and wiped. Like with a cloth I think.

    2. Re:Uh, so.... by jruschme5184 · · Score: 5, Funny

      I was thinking the opposite... someone shut down a Clinton server, not realizing it was the backbone for the entire State Department.

    3. Re:Uh, so.... by Anonymous Coward · · Score: 0

      Maybe Bannon redirected all State Department's mail to it as a goodbye prank as he left from the White House?

    4. Re:Uh, so.... by Avantare · · Score: 1

      Nope. The FCC has them in the NN comments left by the public.

  6. Coincidentally... by Anonymous Coward · · Score: 0

    .... The State Department can no longer locate any of Clinton's Benghazi emails hosted on State Dept servers that they were ordered to retrieve on Aug 4th. It's like they were wiped with a cloth or something.

  7. Gmail by warewolfsmith · · Score: 1

    Gmail just works, ask Hillary.

  8. And no one noticed. by biggaijin · · Score: 2

    However, the global temperature dropped 0.3 degrees Celsius because of reduced hot air emissions.

  9. Understaffed by Kenshin · · Score: 1

    Maybe the one person left working in the State Department doesn't know how to restart a server.

    --

    Does it make you happy you're so strange?

  10. Tech industry to the rescue by rholtzjr · · Score: 1

    Hey I know, why don't they get their friends at Google to help them transition to their email system.

  11. Did Hillary's server crash again? by Anonymous Coward · · Score: 0

    Oh not again, did Hillary's server crash again?

  12. The worlds a safer place for the outage by Anonymous Coward · · Score: 0

    Hopefully it will last years...

  13. But no big loss. by 140Mandak262Jamuna · · Score: 3, Funny

    The Secretary of State remained in complete and continuous contact with all people using a server in the basement of the Secretary's residence. The value of precedent set by a previous secretary of state was greatly appreciated by the Administration.

    --
    sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
    1. Re:But no big loss. by Anonymous Coward · · Score: 1

      First, there has been no contact between SecState and the Department employees to date, so there was no change in that due to the email outage.

      Second, Jared was still on-line, so the administration's foreign affairs needs were consider well in hand by the President.

      What a shit-show.

    2. Re:But no big loss. by Anonymous Coward · · Score: 0

      "The Secretary of State remained in complete and continuous contact with all people using a server in the basement of the Secretary's residence."

      Hey now, that's fake news. Everyone knows Rex is a one-person state department and has no staff to talk to.

  14. Reliable servers. by Anonymous Coward · · Score: 1

    This points out the necessity of Secretaries of State in having their own reliable email servers. But that goes over the heads of everyone.

    So, let me comment with the mentality of most people:

    Thanks Trump! You Nazi!

  15. OopsSpace by Anonymous Coward · · Score: 0

    Oops they ran out of space on those 15 year old SCSI drives...

  16. Comment removed by account_deleted · · Score: 1

    Comment removed based on user account deletion

  17. Dang... by __aaclcg7560 · · Score: 1

    That's what you get for upgrading everyone to Office 365 at the same time.

    1. Re:Dang... by Anonymous Coward · · Score: 0

      Hard to believe that a three-letter government agency can have an IT problem when I have it on reliable evidence that they have a muscular miracle worker on their staff.

      Oh, I just heard he's organizing empty candy bar wrappers by brand in the storage closet.

    2. Re:Dang... by __aaclcg7560 · · Score: 1

      Hard to believe that a three-letter government agency can have an IT problem when I have it on reliable evidence that they have a muscular miracle worker on their staff.

      I don't work for the State Department.

      Oh, I just heard he's organizing empty candy bar wrappers by brand in the storage closet.

      I cannot confirm nor deny the existence of Hillary's backup email server behind the U-No candy wrappers in the interns IT closet. Go ask Bill.

    3. Re:Dang... by Anonymous Coward · · Score: 0

      Wow, have you had your medication adjusted? You're funny and almost ... pleasant.

    4. Re:Dang... by __aaclcg7560 · · Score: 1

      Wow, have you had your medication adjusted? You're funny and almost ... pleasant.

      I forgot my vitamins this morning. Thanks for reminding me!

    5. Re:Dang... by Anonymous Coward · · Score: 0

      Well, these "vitamins" sure aren't doing anything for your memory, huh. And if it's multi-vitamins, you realize that all they achieve is expensive urine?

    6. Re:Dang... by __aaclcg7560 · · Score: 1

      And if it's multi-vitamins, you realize that all they achieve is expensive urine?

      I'm shocked — shocked! — that you would suggest that I piss away cheap urine! My gawd... The nerve of some people.

    7. Re:Dang... by Anonymous Coward · · Score: 0

      As we know, anti-psychotic medications cause weight gain, Chris. Looking at ten year old pictures of yourself, you seemed a lot smaller back then.
      I think you just revised your youth in order to deny that you've gained a lot of weight from the medication.
      You seem to go through cycles of batshit insanity, followed by almost mellow sociability.

      So, which is it?

      Benperidol, Bromperidol, Droperidol, Haloperidol, Timiperone, Diphenylbutylpiperidine, Fluspirilene, Penfluridol, Pimozide?

  18. It definitely could possibly be the Russians... by Flytrap · · Score: 1

    Definitely Russian hackers...

    Probably Russian hackers... all the evidence points to this being the case...

    It must be Russian hackers... who else could it be...

    The Cyrillic character found in the code is proof that the Russians could have possibly did it... maybe... definitely...

    The computer logs are obviously classified... and that proves that the Russians may have definitely had a hand in possibly directing the hackers that did it...

    What more proof does anyone need... there is no doubt in my mind that the Russians directed this hacking group to bring the State Department email down...

    All the evidence leads directly to the Kremlin... all the way to the top...

    :
    :
    :

  19. But her emails! by Anonymous+Codger · · Score: 1

    But her emails!

    --
    No sig? Sigh...
  20. Obviously... by hsthompson69 · · Score: 1

    ...they were violating the ToS for their email provider, and were shut down because they were emailing about topics not approved by #Goolag and the other alt-left service providers :)

  21. Hillary the IT Specialist by Anonymous Coward · · Score: 0

    Hillary was doing some server maintenance.

    These servers looked a little dirty so I figured I'd scrub them with some bleach. [Hillary Laugh]

  22. Hmm by butchersong · · Score: 1

    Strangely, nothing of value was lost.

  23. Government! by HEMI426 · · Score: 1

    I'm a Fed, but not for the State Department. It's nice to know that their e-mail fiascos are news-worthy, but my Service being unable to provide reliable e-mail access---even after farming it out to Microsoft---reliably is just par for the course. Add in our software-distribution system being down for weeks, and...Yeah, it's definitely incompetence. Government "efficiency" at its finest.

  24. article E-Marketing by osamaalqaisi · · Score: 1

    http://anshrnow.com/article/10... http://anshrnow.com/article/10... http://anshrnow.com/article/14... http://anshrnow.com/article/13... http://anshrnow.com/article/13... http://anshrnow.com/article/13... http://anshrnow.com/article/13... http://anshrnow.com/article/13... http://anshrnow.com/article/13...

    --
    http://anshrnow.com
  25. I've always heard... by Hentai007 · · Score: 1

    Aren't they running on lotus notes running on some old NT Era hardware? No joke I had always heard that from folks.

    1. Re:I've always heard... by __aaclcg7560 · · Score: 1

      IBM used to provide help desk services for larger corporations and government entities. I worked for them for three years after the dot com bust. I've encountered many former IBM shops over the years. You could always tell with the presence of IBM/Lenovo laptops and/or the requirement of Lotus Notes in the job description (I've never seen it in the field). The three-letter agency I work for now used to be an IBM shop, but Dell laptops have completely taken over in the last few years.

  26. Microsoft!!!! by Anonymous Coward · · Score: 0

    Microsoft. Let's put the blame where it belongs.

    Anyone tried to upgrade 200 Exchange servers?

    They need to be running postfix with dovecot, clearly.

    And I'd wipe Outlook from any desktops too. The only larger virus I know is iTunes.

    1. Re:Microsoft!!!! by Anonymous Coward · · Score: 0

      I was the email admin at a small Gov agency. I had 2500 users on Exchange which work quite well and I basically administered for about half my time. I had around 100 Unix based mail users that took 2 admin working full time to support.

  27. "Suffers"? by No+Longer+an+AC · · Score: 1

    I would consider it a blessing if my email stopped working.