Slashdot Mirror

← Back to Stories (view on slashdot.org)

US State Department Suffers Worldwide Email Outage (usatoday.com)

Posted by EditorDavid on from the speaking-of-government-shutdowns dept.

An anonymous reader quotes USA Today: The U.S. State Department's email system underwent a worldwide outage Friday, affecting all its unclassified communications within and outside of the department. The system was fully restored by Friday afternoon [after 12 hours], said a State Department official briefed on the incident who was not authorized to speak publicly and requested anonymity.

It was not clear what caused the early morning outage, but spokeswoman Heather Nauert told reporters it was not "any external action or interference."

40 of 69 comments (clear)

  1. Logical conclusion: by Archtech · · Score: 4, Informative

    '... spokeswoman Heather Nauert told reporters it was not "any external action or interference."'

    So just incompetence, then.

    --
    I am sure that there are many other solipsists out there.
    1. Re:Logical conclusion: by omnichad · · Score: 4, Insightful

      If they don't know what caused it, I don't really know how they can fully rule out external actors being involved.

    2. Re:Logical conclusion: by El+Cubano · · Score: 4, Informative

      So just incompetence, then.

      I came here to say just that. You might be surprised at just how bad it is.

      In most cases, government IT is a dumpster fire. I could go on at length, but the absolute best example of it is the failed FBI Virtual Case File project. IEEE did a very in depth write up of it several years ago.

      If I recall correctly, the objective was that the FBI tried implement a system after 9/11 that would allow its field agents to collaborate more effectively than phone calls and non-government email (apparently most field agents used things like Yahoo! and HotMail because they either couldn't get government email accounts or they sucked so badly). After years of ill-defined and constantly changing requirements and $300+ million spent, the project was terminated as a failure and the government actually got nothing out of it. Go read the IEEE story on it for more details.

      The problem is that this sort of thing happens constantly with projects of all sizes and scopes. Of course, if you've worked in corporate IT, you will recognize that similar things happen there as well.

    3. Re:Logical conclusion: by MoarSauce123 · · Score: 4, Funny

      I bet on Windows updates going haywire.

    4. Re:Logical conclusion: by Neuronwelder · · Score: 1

      When you don't pay people a fairly, and cut back on the skeleton crew - what do you expect?

    5. Re:Logical conclusion: by Salgak1 · · Score: 1

      And those of us inside TRYING to put the dumpster fire out are stymied at every turn. I have a program that refuses to produce ANY documentation, "because we're agile". Another one had lost all track of a bunch of equipment. Until I was assigned to close out the program. . .and found it was all at the last location listed on the 4-year-old documentation.

      Security patches get waived by programs "because we don't risk downtime". Even if the patching was planned to occur during scheduled downtime

      As one of the nuns at my elementary school was known to remark. . . it makes Baby Jesus cry. . .

      And as for the FBI and the Virtual Casefile. . . .they had also bought an entire warehouse of gear to support it. Which sat there for years, in boxes, until they finally killed the program. And then PAID someone to take the unused gear off their hands. . .

    6. Re:Logical conclusion: by rbrander · · Score: 1

      I was just in a municipal government, but the dysfunctionality really was amazing.

      I know I'm an anti-authoritarian personality, so most problems look to me to be caused by authoritarianism, but I'm pretty sure about this one. I watched it grow for 20 years.

      Every large corporate (and this goes double for "municipal corporations" and other governments) IT shop that goes back decades started off as a *mainframe* shop, with all IT totally Under Control. As they took over the PCs, everything there had to be brought totally Under Control, which included things like Only One Application for every need, followed 2 years later by Everything Must Be Microsoft.

      And they could back this up, and hold up the rest of the corporation for any sum they demanded for the simplest applications, because IT was *their* turf and no other department could do their own, hire their own contractors, nothing.

      Whenever they developed a service or application, NOTHING could be done until EVERY ramification had been explored and the perfect One Application For All Needs Of All Stakeholders was developed. This made every application a huge project. Indeed, it soon became clear that they hated small projects and those would either not be done, or be made large.

      I retired feeling that until the power of centralized IT departments is broken, that IT in large corporate bodies will be like the Soviet central-command economy: hopelessly broken and incapable of producing the simplest things even at huge cost mark-ups, with frequent spectacular failures.

      This isn't about government-vs-private, it's just centralized-vs-flexible. I compared notes with an engineer at our phone company, which sells all sorts of services now, TV and Internet, etc. They do have a business imperative that government services do not: they have had to constantly innovate to develop and grow these new businesses lately. So their IT department was forced to change away from the centralized mainframe shop of yore. It's a chaotic mess, alas, because they never acknowledged they'd have to do this, so it all kind of slipped away on them with no planning for it. But, while it's a chaotic mess, they are functioning and making money; even in the worst case scenario for control-vs-chaos, the chaos is not as bad as the over-control.

    7. Re:Logical conclusion: by Anonymous Coward · · Score: 2, Insightful

      If they don't know what caused it, I don't really know how they can fully rule out external actors being involved.

      It's actually exceptionally simple, but I'll explain it in detail for you.

      Say you are the sysadmin in charge of a server, which goes down one day and so far all you have discovered is that pressing the power button appears to do nothing at all.

      Within a minute or two of you discovering this, your boss is asking for a comprehensive detailed report on what is wrong.
      You tell him the server won't power up, but you aren't sure why.
      Boss asks if it was Russian hackers, to which you look at him like he's a moron for even asking and you say "No, I just said the server isn't powering on"

      A moment later your boss sends out a company wide email stating that the server administrators do not know what caused the outage, but have ruled out external actors being involved.

      Reporters are similar to your boss in the above made-up situation. They don't care about details and tech stuff and how things work. Their eyes totally glaze over.
      The "report" also has the primary goal of being short and concise, without a lot of qualifiers and details, just simple and to the point at the cost of being accurate in ways most reading the report won't care about.
      They also likely don't care about attributing words to you that you did or did not say.

      This all combines to get the result you see and are confused by, and is almost always what happens when your words get filtered through other people like this.

    8. Re:Logical conclusion: by __aaclcg7560 · · Score: 1

      Or, more likely, they're switching everyone over to Office 365 and decided to migrate all the accounts at the same time. That's usually a recipe for disaster. The three-letter agency that I work for is taking months to complete the migration, resulting in minor issues for some people and a whole lot of bitching by everyone else over the new interface.

    9. Re:Logical conclusion: by haruchai · · Score: 1

      '... spokeswoman Heather Nauert told reporters it was not "any external action or interference."'

      So just incompetence, then.

      Looks like Trump's plan to stop the leaks is working

      --
      Pain is merely failure leaving the body
    10. Re:Logical conclusion: by haruchai · · Score: 1

      ".they had also bought an entire warehouse of gear to support it. Which sat there for years, in boxes, until they finally killed the program. And then PAID someone to take the unused gear off their hands"

      Not just government who messes up like that. Way back BP decided to implement WiFi at all their sites in North America & decided to buy ALL the access points before the project was ready to roll. Except this was when 802.11b was the standard and just before 802.11a/g was ready, which happened after the purchase but long before they were about to start installing the APs.
      In the end, they bought all new equipment and never opened the 802.11b gear. My boss at the time - we were doing outsourced IT support for them - visited their Houston Westlake campus in 2007 and saw a storage room with 15000 boxed APs.

      --
      Pain is merely failure leaving the body
    11. Re:Logical conclusion: by CaptainDork · · Score: 1

      Fortunately, we took care of this back in 2009.

      Meet the Military's Cyber-Security Forces
      Posted by CmdrTaco on Thursday January 28, 2010 @11:41AM from the can't-wait-to-see-the-ad-campaign dept

      --
      It little behooves the best of us to comment on the rest of us.
    12. Re:Logical conclusion: by Dutch+Gun · · Score: 1

      "Hey, why am I on this mailing list? Remove me please."
      "Yeah, me too"
      "Why am I on this thread?"
      "What mailing this?"
      "Everyone, stop responding to all"
      "Autoreply: Mark G is out of the office until next Tuesday"
      "Lol, you first!"
      .
      .
      .
      Bedlam!"

      --
      Irony: Agile development has too much intertia to be abandoned now.
  2. This is what I can guarantee... by bogaboga · · Score: 1, Insightful

    The U.S. State Department's email system underwent a worldwide outage Friday, affecting all its unclassified communications within and outside of the department.

    There will be mention of Russia in relation to this unfortunate event, partly as an effort to hide our [inherent] incompetency, sadly.

    1. Re:This is what I can guarantee... by omnichad · · Score: 1

      This is the Russian^H^H^H^H^H^H^H Trump presidency. They will blame Ukraine.

  3. Uh, so.... by sconeu · · Score: 4, Funny

    Hilary's server is still up, right?

    [DISCLAIMER: I dislike both parties]

    --
    General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
    1. Re:Uh, so.... by Anonymous Coward · · Score: 1

      No it was taken down and wiped. Like with a cloth I think.

    2. Re:Uh, so.... by jruschme5184 · · Score: 5, Funny

      I was thinking the opposite... someone shut down a Clinton server, not realizing it was the backbone for the entire State Department.

    3. Re:Uh, so.... by Avantare · · Score: 1

      Nope. The FCC has them in the NN comments left by the public.

  4. Gmail by warewolfsmith · · Score: 1

    Gmail just works, ask Hillary.

  5. And no one noticed. by biggaijin · · Score: 2

    However, the global temperature dropped 0.3 degrees Celsius because of reduced hot air emissions.

  6. Understaffed by Kenshin · · Score: 1

    Maybe the one person left working in the State Department doesn't know how to restart a server.

    --

    Does it make you happy you're so strange?

  7. Tech industry to the rescue by rholtzjr · · Score: 1

    Hey I know, why don't they get their friends at Google to help them transition to their email system.

  8. But no big loss. by 140Mandak262Jamuna · · Score: 3, Funny

    The Secretary of State remained in complete and continuous contact with all people using a server in the basement of the Secretary's residence. The value of precedent set by a previous secretary of state was greatly appreciated by the Administration.

    --
    sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
    1. Re:But no big loss. by Anonymous Coward · · Score: 1

      First, there has been no contact between SecState and the Department employees to date, so there was no change in that due to the email outage.

      Second, Jared was still on-line, so the administration's foreign affairs needs were consider well in hand by the President.

      What a shit-show.

  9. Reliable servers. by Anonymous Coward · · Score: 1

    This points out the necessity of Secretaries of State in having their own reliable email servers. But that goes over the heads of everyone.

    So, let me comment with the mentality of most people:

    Thanks Trump! You Nazi!

  10. Comment removed by account_deleted · · Score: 1

    Comment removed based on user account deletion

  11. Dang... by __aaclcg7560 · · Score: 1

    That's what you get for upgrading everyone to Office 365 at the same time.

    1. Re:Dang... by __aaclcg7560 · · Score: 1

      Hard to believe that a three-letter government agency can have an IT problem when I have it on reliable evidence that they have a muscular miracle worker on their staff.

      I don't work for the State Department.

      Oh, I just heard he's organizing empty candy bar wrappers by brand in the storage closet.

      I cannot confirm nor deny the existence of Hillary's backup email server behind the U-No candy wrappers in the interns IT closet. Go ask Bill.

    2. Re:Dang... by __aaclcg7560 · · Score: 1

      Wow, have you had your medication adjusted? You're funny and almost ... pleasant.

      I forgot my vitamins this morning. Thanks for reminding me!

    3. Re:Dang... by __aaclcg7560 · · Score: 1

      And if it's multi-vitamins, you realize that all they achieve is expensive urine?

      I'm shocked — shocked! — that you would suggest that I piss away cheap urine! My gawd... The nerve of some people.

  12. It definitely could possibly be the Russians... by Flytrap · · Score: 1

    Definitely Russian hackers...

    Probably Russian hackers... all the evidence points to this being the case...

    It must be Russian hackers... who else could it be...

    The Cyrillic character found in the code is proof that the Russians could have possibly did it... maybe... definitely...

    The computer logs are obviously classified... and that proves that the Russians may have definitely had a hand in possibly directing the hackers that did it...

    What more proof does anyone need... there is no doubt in my mind that the Russians directed this hacking group to bring the State Department email down...

    All the evidence leads directly to the Kremlin... all the way to the top...

    :
    :
    :

  13. But her emails! by Anonymous+Codger · · Score: 1

    But her emails!

    --
    No sig? Sigh...
  14. Obviously... by hsthompson69 · · Score: 1

    ...they were violating the ToS for their email provider, and were shut down because they were emailing about topics not approved by #Goolag and the other alt-left service providers :)

  15. Hmm by butchersong · · Score: 1

    Strangely, nothing of value was lost.

  16. Government! by HEMI426 · · Score: 1

    I'm a Fed, but not for the State Department. It's nice to know that their e-mail fiascos are news-worthy, but my Service being unable to provide reliable e-mail access---even after farming it out to Microsoft---reliably is just par for the course. Add in our software-distribution system being down for weeks, and...Yeah, it's definitely incompetence. Government "efficiency" at its finest.

  17. article E-Marketing by osamaalqaisi · · Score: 1

    http://anshrnow.com/article/10... http://anshrnow.com/article/10... http://anshrnow.com/article/14... http://anshrnow.com/article/13... http://anshrnow.com/article/13... http://anshrnow.com/article/13... http://anshrnow.com/article/13... http://anshrnow.com/article/13... http://anshrnow.com/article/13...

    --
    http://anshrnow.com
  18. I've always heard... by Hentai007 · · Score: 1

    Aren't they running on lotus notes running on some old NT Era hardware? No joke I had always heard that from folks.

    1. Re:I've always heard... by __aaclcg7560 · · Score: 1

      IBM used to provide help desk services for larger corporations and government entities. I worked for them for three years after the dot com bust. I've encountered many former IBM shops over the years. You could always tell with the presence of IBM/Lenovo laptops and/or the requirement of Lotus Notes in the job description (I've never seen it in the field). The three-letter agency I work for now used to be an IBM shop, but Dell laptops have completely taken over in the last few years.

  19. "Suffers"? by No+Longer+an+AC · · Score: 1

    I would consider it a blessing if my email stopped working.