Slashdot Mirror
State Dept E-mail Crash After "Reply-All" Storm
twistah writes "It seems that a recent 'reply-all storm' at the State Department caused the entire e-mail infrastructure to crash. A notice sent to all State Department employees warned of disciplinary actions which will be taken if users 'reply-all' to lists with a large amount of users. Apparently, the problem was compounded by not only angry replies asking to be taken off the errant list, but by the e-mail recall function, which generated further e-mail traffic. One has to wonder if capacity planning was performed correctly — should an e-mail system be able to handle this type of traffic, or is it an unreasonable task for even the best system?"
http://msexchangeteam.com/archive/2004/04/08/109626.aspx
Again...
-Ghostis
Computer Science is all about trying to find the right wrench to bang in the right screw. -T.Cumbo?
What an irony that they decided to mass mail when they've warned their employees not to do so. What they should have done if they were concerned about their load [which evidently they should have] was to warn their employees in blocks, perhaps 10% at a time with space between to take care of the massive response... However, judging by the nature of their work [it is the state department after all] I don't believe it unreasonable that there could be events in their future requiring such mass mailings again and having the whole system crash under the load would be no doubt very bad in emergencies.
Sigs are too short to say anything truly profound so read the above post instead.
Of the reply all button. Please do not respond with the reply all button. What they need is a reply some button.
Whoever wrote the headline for this summary needs to have their slashdot editor privileges revoked.
TFA states "an e-mail storm nearly knocked out one of the State Department's main electronic communications systems", and "a major interruption in departmental e-mail". The problem is clearly spelled out as "e-mail queues, especially between posts, back up while processing the extra volume of e-mails".
This is simply the queues backing up, not the servers crashing. Nowhere does TFA state anything to suggest that there was a "State Dept E-mail Crash", which the summary's headline boasts. The proper headline should read "Large E-mail Queues at State Dept After Reply-All Storm".
No, I'm not new here. That's why I'm fed up with the sensationalist "journalism" that is getting worse and worse here.
I remember my first year of college when I wanted to send Xmas greetings to 'everyone'. I remember, the IT director of the college running from computer lab to computer lab looking for student number xxyz.
Fun times.
If, like approximately 1/2 of the American population, you currently had no health care at all, your attitude would probably be different.
And you might want to remember that the current financial and industrial collapse was given to us by the finest and most highly educated examples of stupid, greedy, incompetent, short sighted, overpaid, negligent, and possibly criminal management that private enterprise has been able to produce and promote.
a) Maintaining large list by copying all recipients into the hrader is a fucked up idea at best (because there is no way this list will be kept updated), and a informaiton leak at worst (because somebody eralier on a non-updated list may get information which he should not get - e.g. former employees). Why do governmental institutions still us it?
b) Why in the world do modern e-mail clients still allow reply all to hundreds of recipients without an additional safety question. I would expect my program would warn me before sending an emails to thousand people.
Dear state department
I'm sorry to hear about your recent trouble
There is a brand new invention on the internet which have the ability to ease the strain on your mailservers. it is called maillist managers. one is called mailman and can be found at: http://www.gnu.org/software/mailman
There are several others, some free, and some non free, but they exist for most server platforms. If you don't have the expertice in house to set it up corrctly, you can get any number of consultancy companies to help you out.
Yours faithfull
Almost anonymous coward
OpenNet, by a very quick look on google, seem to be their network name for the non-classified bits and pieces. Supposedly Microsoft + Cisco stuff.
Feel free to disagree, but please provide a URL reference to the OpenNet email server software vendor if doing so.. ;-)
To all the mods, please don't destroy all my Karma. I really do hate that Reply All button.
According to this article, they were migrating to Exchange in 2001. If it was set up by admins who knew what they were doing, they could have set the perms on the distribution list so only authorized users could use it.
Good to know that rigorous competition in the marketplace has totally eliminated misuse of 'reply-all' in the private sector. I look forward to continuing to have a lower life expectancy and higher infant mortality than Canadians and Swedes.
Why is that funny?
Exchange has a feature your email client didn't support. Ha ha ha!! IT'S HILARIOUS!!!!
Comment of the year
The really good thing though is that you are not bitter at all.
And in even further news, corporations are not perfect.
I take it you're not familiar with how enterprises plan. They plan for regular load, not aberrant once-in-a-blue-moon load. This is bog standard behavior for a system responding to people doing stupid things. If you think this is restricted to the US government, you've never worked in corporate IT.
Those who can, do. Those who can't, sue.
yes its exchange internally
openNet is what they brand it as
feel free to correct me with evidance that it was not the case any more but I know 2 exchange servers there and this say's otherwise
exchange has the recall ability and so does lotus notes
most other servers do not have this feature for very good reasons l
regards
John Jones
www.johnjones.me.uk my blog about email and digital communication
I've seen it so many times over the years. I wonder why it's so hard to add an administrative setting that limits Reply All to a certain number of users? Set at 100, it would only send the first 100, then ask the user if they wanted to send the next 100. Or 300 or 400 or whatever.
I can't count the number of people sending a hasty and blistering reply to thousands of people. Not only committing public suicide but accounting for who knows how many unproductive man hours while the entire organization stopped to read their spew. It's just crazy.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
http://www.hanselman.com/blog/HowToEasilyDisableReplyToAllAndForwardInOutlook.aspx
2 simple lines that you can include in your Outlook client to prevent this action internally on your exchange server.
Note this does not include any macros in the email.
This just in, President-Elect Obama Assassinated! Oh, don't take it so literally. I was just trying to capture your attention in a short amount of time. Obama wasn't killed, silly. There was just some CHARACTER assassination against him on a late night talk show.
No, this is more like a free ride when you already paid.
I use the reply-all button frequently, for ad-hoc small group discussions. If I have a document I want two people to review, I send it to both of them, and they send their comments back to both me and the other person I sent it to with reply-all, so we're all on the same page.
If the same group of people is frequently collaborating you can set up a mailing list, but it's a real pain in the ass to set up a mailing list every time you want a group of 3 or 4 people to exchange 5-10 emails.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
The problem is the message replied to having - RTFA - several thousand addresses in the To: and CC: fields. This is what BCC is for . Allowing people to put several thousand addresses in to the headers will eventually result in a mail storm, whether someone hits Reply To All or not. The first time someone opens a virus laden attachment that goes through their (archived by law, this being a federal agency) emails, it will send itself out to thousands of equally clueless people. One of them will run the attachment, which will send another copy to several thousand people. And so on. This happened where I work once, by people who should have known better. Before it was done, I was getting two hundreds copies of the virus per day.
Whoever sent out the message replied to should be fired and criminally prosecuted for deliberately sabotaging the State Department's email system. But since the article doesn't mention this at all, I'm assuming it was some dumbass boss somewhree who is immune to any form of disclipline for anything, up to and including murder.
I saw a weird variant on that back in university.
One of the engineering departments had a room full of (at the time) fairly high end sun workstations, and these were used both interactively and for people running longer compute jobs overnight.
To facilitate overnight jobs, the admins had set up a round robin dns alias that updated every couple of seconds to point to the machine reporting the lowest load average.
One of the students in my class had the bright idea of "If put 'ssh lowest' in my bashrc file, every time i open a terminal window it'll automatically pick the least loaded machine".
Fast forward a few minutes and we've got 80 sun workstations which have all systematically ssh'd to each other and none of which will accept any new connections...
What like about this post is its calm matter-of-fact delivery... not a hint of anger to be found. Party on Garth!
If, like approximately 1/2 of the American population, you currently had no health care at all, your attitude would probably be different.
A) Nobody has "no health care". There's always the emergency room -- in the US they can't turn you away for lack of ability to pay. This isn't ideal but it disproves your statement of "no health care at all"
B) There's ~47,000,000 Americans without health insurance. Out of a population of ~300,000,000. That's 15.67%, not 50%
Neither of those are ideal but if you are going to post on a subject at least get your facts straight.....
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
"The term Deltic (meaning in the form of the Greek letter Delta) is used to refer to both the Deltic E.130 opposed-piston high-speed diesel engine designed and produced by D Napier & Son, and the locomotives produced by English Electric using these engines, including their demonstrator locomotive named DELTIC and the production version for British Railways, who designated these as (TOPS) Class 55."
For a train your English is quite good.
Not a Twitter sockpuppet... but I wish I was.
This is the government we're talking about. That would be above their pay grade.
And that's why we don't have the Army, Navy, Air Force, or Marines any more. We just have private contractors, because they are far more patriotic, and loyal. Plus they don't TK as often. We should have gotten rid of those government socialist military types BEFORE we had to spit on them for fucking up Vietnam.
Now THAT is a troll.
Fascism trolls keeping me up every night. When I starts a preachin', he HITS ME WITH HIS REICH!
When somebody sends out an email with incorrect information? Oops, I just noticed that I said 3 pm for the meeting, but I forgot about the timezone change so it's actually 2 pm. Or I just mistyped. Whatever.
Do a recall and replace - that way you don't have people thinking the second email was just a duplicate.
I don't read AC A human right
The guy who wants to quit but doesn't because he'll only get unemployment benefits if he's fired :)
Um...which goes to show how little you know about unemployment. At least in MA, you don't get shit if it is "termination with cause", ie fired. If you're laid off, great- but even then, your employer gets a phone call from the unemployment department asking whether you were fired or laid off. Nothing stops them from lying and saying you were fired with cause- and then you've got a legal battle on your hands, which you can't afford.
Other fun facts about unemployment in MA: you don't get paid for two full weeks after you FILED- not after you were laid off, but after you FILED. You get a pittance compared to your normal salary; you'd be lucky to make rent on a studio apartment in Boston based off an entire month's unemployment checks.
Any income is deducted from your UA check. Say for example you find a 2-3 hour consulting thing on CL and make $150 helping someone fix their computer. Guess what? Your unemployment check for that week will be $150 smaller. This basically means that you have no incentive to find any kind of income while you're on UA.
Last but certainly not least: you have to pay taxes, medicare, medicaid, etc on your unemployment benefits. It's not bad enough that you're basically on welfare- you have to fork over a portion of the money the government is giving you, BACK to the government. Cute, eh?
Please help metamoderate.
Message recall. Oh dear.
Years ago, I wrote the bulk of this feature. It is not an Exchange feature, but an Outlook feature. It works by sending a custom MAPI message that Outlook recognizes and processes. Of course, this only works if all recipients are using Outlook. It also, after we did some usability testing, only deletes unread email, or email that has not been moved to a subfolder (the original version was quite determined and would hunt down and kill the message even if it had been moved to a subfolder, renamed or entered the email protection program). In this way, it did not violate the UI dictum that the computer move things around when you haven't given it instructions to do so.
So yes, it is Outlook only. If sent to a non-Microsoft mail system, it degrades to a simple notification that the message is being recalled. And it does not a good choice for getting rid of flames you shouldn't have been sending. But within its expected use as a feature - correcting mistakes in email that should have been caught before pressing send, it works fairly well.
But because it is client based, rather than an Exchange feature, it does cause a new mail message to be sent to each original recipient and, combined with a send-all storm, could greatly exacerbate things.
And, preemptively, for those who have philosophical objections to me having written the code in the first place, I'll just have to live with your disapproval and hope my steady paycheck somehow sooths my guilty conscience.
I am on a list of bidders for potential contracts with the Washington Metropolitan Transit Authority, which operates the Metrobus and Metrorail for Washington, DC and the nearby suburbs in Maryland and Virginia. The annual budget for the Authority is in excess of a billion dollars; it's larger than the budget of the entire State of Montana, for example.
One time I got a message with more than 25 recipients on it regarding a change in the way they were operating their procurement website. Well, I suspected that it was some spammer pretending to be from the Authority, because one of the "red flag" signs of being spammed is more than 10 recipients on the same messsage. But I discovered that it really was from the Transit Authority, it was simply an ordinary announcement with no url links and nothing but the announcement. But instead of simply either making the recipients BCC recipients, and sending it to an internal transit authority e-mail address as To:, or sending individual messages to each potential supplier, the contracting agent had simply sent it out To: listing all persons who were registered as bidders with the authority.
My e-mail address was one of these potential suppliers along with a few other people.
1,627 other people to be precise. This was the longest To: list on an e-mail message I have ever seen on a piece of e-mail that wasn't spam; 1,628 contacts. No, I didn't reply all, but I couldn't think of a way to refer to this incident as a "Send All" message and tie into this story. The other half of this incident was that the procurement agent had also just given all potential suppliers to the Authority, every other supplier's e-mail address, too.
The lessons of history teach us - if they teach us anything - that nobody learns the lessons that history teaches us.
There's ~47,000,000 Americans without health insurance. Out of a population of ~300,000,000. That's 15.67%, not 50%
I don't know anything about those source numbers, so I'll just go ahead and believe them, but I've gotta call you on those sig figs there. 15.67? 4 sig figs? How about just 20%.
(I'm not sure whether to thank or to blame all of my physics teachers for drilling us in sig figs)
coding is life
and of course, off-topic from TFA, I signed up with a Product Testing Place. They email me once every six months and see if I want to test some new gadget or something and I get paid $75.
I signed a confidentiality agreement with them.
I am not allowed to discuss ANYTHING about the product or reveal I am testing it or anything. I was never there, I am nobody.
Last year I got an email - From The President of The Testing Company - personally thanking me for all the help in the last year.
He also thanked everyone else who "helped" last year as well and I could see who they were because apparently the President (or the secretary) just put all our emails into the TO: field and let it fly.
Lots of Identifiable people on the list because they used their WORK email, like john.doe@largecorporation.com So it was easy to see who else was part of that big Butt Plug testing program.
I did a REPLY to ONLY the President and laid into him about the confidentiality agreement and told him if he didn't know how to use email to stay away from the computer.
Later that day we all got another email from the President, this time apologizing for revealing all our personal emails, never happen again etc etc. And apparently he figured out how to use BCC!
So yelling at someone does seem to work to change behaviour.
Also- this is a dupe comment, I posted this once before on Slashdot someplace, but since this is Slashdot I didn't think a dupe would be a problem.
I like microcars
Looks like the pathetic one is you, and the Submitter. If you RTFA, it clearly says
He said the result was "effectively a denial of service as e-mail queues, especially between posts, back up while processing the extra volume of e-mails.
Never says the actually crashed, merely that the high volume generated large queues, exactly what you would expect to happen in a properly engineered system. But hey, this is Slashdot, so making up reasons to hate Exchange (and there are plenty of LEGITIMATE reasons to hate exchange) is the norm.
Yes, and if the management there knew what they were doing they could have used BCC instead of threatening their employees with repercussions for touching the Reply-All button. Full rant here
Show a man some news, distract him for an hour. Show a man some mod points, distract him for the rest of his life.
And, preemptively, for those who have philosophical objections to me having written the code in the first place, I'll just have to live with your disapproval and hope my steady paycheck somehow sooths my guilty conscience.
Sir, your conscience should enable you to sleep as a baby. This feature alone has provided me with countless hours of entertainment as people suddenly realize the error of their ways as they make bold statements (often of a derogatory nature) in an e-mail which happens to be sent to the wrong person in CC. As they hastily move to send out the recall of their mail only to receive a mail from someone who was offended.
This feature combined with the nature of some people to act before they think has proven quite entertaining indeed. Please enjoy your steady paycheck to its fullest, your code has amused me countless times.
Do a recall and replace - that way you don't have people thinking the second email was just a duplicate.
[Corrected] Send a followup with a changed subject line so that people know something has changed.
And next I expect you're going to try and teach them to edit their quotes and to stop top posting ?
I'm close to giving up on educating users with email, it's pretty hopeless I think.
Top posting is easier for most users to understand. For business users, its best to top post by default, unless you are going to counter a long email point by point. In that case please be sure to top post the words "see below."
On open mailing lists (anything not run by your employer where you decide to subscribe) I bottom post and edit posts. At work, I top post. It gives a complete linear history of a conversation, which is good because most outlook users just sort email by date.
Some people just can't handle reading properly formatted reply emails, let alone writing them.
--- Justin Dearing http://www.justaprogrammer.net/ We're just programmers.
They used exchange and got screwed, just like -everyone- who uses exchange.
This happens all the time just most companies cover up stuff like this because it's not good for the share price.
Am I the only one to think that it is quite peculiar that it is happening 9 days before the Government turns over? I mean, how much difficult could it be to say that some sensitive/embarrasing mails got lost during this crash? I think this should be looked into in more detail and make double sure that no mail was "lost".
I recently (as in, within the last week) gave in to breaking netiquette and dumbing down my emails by top posting. Why? I used to respond through the email, making sure everything is properly indented (for HTML users) or prefixed with a >. I also edit out extraneous content with [snip].
And yet, every time I get into discussions with clients who aren't VERY computer savvy via email and respond to each point in order (as one SHOULD to make it EASIER to understand) they miss it.
Why?
They skim. They don't look for anything beyond the top of the email. If it's not contained in the top paragraph, it's obviously not important enough to worry about.
Proper netiquette saved my behind in corporate America (the company for the specific company in question shall rename nameless. Let's just say that it's a Waltham-based HR software company which was recently bought out. They're not SAP or Oracle so you may not know of them unless you're with Fidelity, State Farm, Sears, etc. HR departments). VPs used to come back to me and demand to know why my team didn't find certain obvious showstopper defects (I personally found the ones in question, analyzed the potential show-stopping, contract-voiding effects the bugs might have, and argued for their resolution and was vetoed by the COO and CEO personally). Well, as it turns out, they denied ever hearing about the defects, and the blame was on me since I was directing Quality Assurance. Fortunately for me (or perhaps unfortunately because I remained at that company for 2.5 years after that incident because I foolishly believed their lies about stock options, etc. - in the recent buyout "preferred stock" holders got NOTHING but the common stock holders (mainly the CEO and CIO) made millions - I could have moved on to any of the much better offers that came my way during that time) I archive ALL my email. I don't delete email unless it's spam or jokes, etc.
These were bugs I brought up to the director of client services, the COO, and the CEO (I went up the food chain properly) and while the director of client services wanted it fixed; she immediately saw the potential ramifications, the COO and CEO flat-out rejected it, citing the costs involved in fixing it (it was an architectural issue which would have required 3 to 6 days of dedicated time for the chief architect, myself, and two Sr. software engineers).
The issue blew up at a client site. The client spent months and months developing content (in our English-like business logic language), assuming that plans would display to the employee as our Sales and Support staff claimed it would, and didn't set up the complex tests I did to verify. Silly client, they assumed that the software works as advertised! They discovered after hiring many temps and contractors to develop their HR portal that data inheritance was completely broken. They had to reimplement 6+ months of work. Well, needless to say, the shit hit the fan at that point.
The CEO and COO came by my desk (I moved on to Release Engineering at that point, wanting to do more coding and and playing less political games since the executives were morons, in denial about our being a software company despite our sole product is software and our sole service is designed on selling seats of product and number of subjects and plans, not hours/days/resources for implementation. It was a very product-driven model and the customers were treated as a product-driven company would operate.). They wanted to know why these defects were not found under my watch. Fortunately while I was director not only did I personally read every single defect to classify and prioritize them properly and ensure they were assigned to the correct software engineer, but I also happened to be the one who discovered and analyzed that defect, suspecting it was broken when I was digging through some old spaghetti code we had in place driving that module.
So, I found the email thread in outlook in about one minute. I also happened to have follow up messages citing our con
The Christian Right is Neither (Christian nor right). See: Matthew 23, Matthew 25, Ezekiel 16:48-50
Microsoft has described this in excruciating detail before, because at one point even they managed to crash their Exchange server - through mail list reply all spam.
http://msexchangeteam.com/archive/2004/04/08/109626.aspx
Sounds like the State Department might not have upgraded to Exchange 2003.
For a site about things like basic rights, Slashdot users sure do like to censor "dissent".
The very fact that there is a "recall" feature shows a lack of understanding of how email works.
What's the phrase? Trying to take something off the internet is like trying to remove piss from a pool.
One can argue that internally they should have complete control, but even then one is racing against time to delete the message *on the client* before the person reads it. What's the point?
It's better to vote for what you want and not get it than to vote for what you don't want and get it.
- E. Debs
Good stuff, university being used for what it's there for - education. I'm sure that guy won't ever make that mistake again.
That post wasn't rambling. It only took me a minute to read. I'm pretty sure his business emails contained just enough needed data to get the point across.
Perhaps people should learn how to read again without going "tl;dr" everytime they see more than 1 paragraph. Which is why he had to start top-posting, because that's what people do. More than one paragraph, they don't read it because that would tired their little brains out too much to spend 1-3 minutes reading.
"We need to get over this notion, that, for Apple to win... Microsoft must lose." - Steve Jobs, 1997