Slashdot Mirror
State Dept E-mail Crash After "Reply-All" Storm
twistah writes "It seems that a recent 'reply-all storm' at the State Department caused the entire e-mail infrastructure to crash. A notice sent to all State Department employees warned of disciplinary actions which will be taken if users 'reply-all' to lists with a large amount of users. Apparently, the problem was compounded by not only angry replies asking to be taken off the errant list, but by the e-mail recall function, which generated further e-mail traffic. One has to wonder if capacity planning was performed correctly — should an e-mail system be able to handle this type of traffic, or is it an unreasonable task for even the best system?"
Sounds like nearly the exact same situation. The problem here is that the average user is just going to click the first "reply" button he sees, and if that happens to be Reply All, nothing's going to stop him. Perhaps the mail client should have a feature enabled by default that warns if an exceptionally large number of messages are being sent and allow the option to cancel.
Whoever wrote the headline for this summary needs to have their slashdot editor privileges revoked.
TFA states "an e-mail storm nearly knocked out one of the State Department's main electronic communications systems", and "a major interruption in departmental e-mail". The problem is clearly spelled out as "e-mail queues, especially between posts, back up while processing the extra volume of e-mails".
This is simply the queues backing up, not the servers crashing. Nowhere does TFA state anything to suggest that there was a "State Dept E-mail Crash", which the summary's headline boasts. The proper headline should read "Large E-mail Queues at State Dept After Reply-All Storm".
No, I'm not new here. That's why I'm fed up with the sensationalist "journalism" that is getting worse and worse here.
If, like approximately 1/2 of the American population, you currently had no health care at all, your attitude would probably be different.
And you might want to remember that the current financial and industrial collapse was given to us by the finest and most highly educated examples of stupid, greedy, incompetent, short sighted, overpaid, negligent, and possibly criminal management that private enterprise has been able to produce and promote.
To all the mods, please don't destroy all my Karma. I really do hate that Reply All button.
Maybe someone could introduce them to the concept of a BCC.
Change that to 'that warns if an exceptionally large number of messages are being sent and smack the user over the head with a LART if they don't click cancel' and i'll agree with you.
A large company should have an internal mailing list and/or intranet system that individual users can post messages to. Letting individual users send email to more than a few thousand users in one hit is madness. Especially if they are anything like our customers where they think it is a good idea to send a 10MB attachment to 500 users...
The problem is the morons who send email with "everybody.all.everwhere" (or whatever) in the To: or CC: list. If they were smart enough to put them in the BCC: field, it would be impossible for people to clog up the system with Reply All. Alas.
Comment of the year
Why is that funny?
Exchange has a feature your email client didn't support. Ha ha ha!! IT'S HILARIOUS!!!!
Comment of the year
What they should have done if they were concerned about their load [which evidently they should have] was to warn their employees in blocks, perhaps 10% at a time with space between to take care of the massive response...
No. What they should have done was installed a mailing list manager, created a read-only list called "employees", and posted to it. Voila - n-thousand workers get announcements with no ability to reply to the whole list. Problem solved.
Dewey, what part of this looks like authorities should be involved?
This just in, President-Elect Obama Assassinated! Oh, don't take it so literally. I was just trying to capture your attention in a short amount of time. Obama wasn't killed, silly. There was just some CHARACTER assassination against him on a late night talk show.
And then they threaten disciplinary action if someone uses them the wrong way. Wouldn't it be so much easier to just lock them down? It's what most companies do.
The problem is the message replied to having - RTFA - several thousand addresses in the To: and CC: fields. This is what BCC is for . Allowing people to put several thousand addresses in to the headers will eventually result in a mail storm, whether someone hits Reply To All or not. The first time someone opens a virus laden attachment that goes through their (archived by law, this being a federal agency) emails, it will send itself out to thousands of equally clueless people. One of them will run the attachment, which will send another copy to several thousand people. And so on. This happened where I work once, by people who should have known better. Before it was done, I was getting two hundreds copies of the virus per day.
Whoever sent out the message replied to should be fired and criminally prosecuted for deliberately sabotaging the State Department's email system. But since the article doesn't mention this at all, I'm assuming it was some dumbass boss somewhree who is immune to any form of disclipline for anything, up to and including murder.
What the fuck? Don't do that. Reply all has a valid use case. In fact it's the way everyone at my company most commonly replies to email messages. Why? Because the CC list is there for a reason - those are people who are supposed to know what's going on in that email thread.
How about just educating your users on checking who they're sending an email to, every single time they send one.
It's rare that you're presented with a knob whose only two positions are Make History and Flee Your Glorious Destiny.
If you were my admin and you did this to me I would hunt you down and kick you in the shins.
In the year 2009 we now often work in teams. We often communicate as teams. We often 'think' as... you guessed it... teams.
But by all means I'm sure whatever company you're working at people only talk to one person at a time. You have no group discussions and the only interaction that occurs between employees is by the watercooler and in meetings.
At our company however more than one recipient is the norm. Especially when you want to keep a project manager 'in the loop' of a conversation with a vendor. In fact our most common occurance is to have to say "oops - sorry looks like I dropped so and so from this conversation". Not "Ooops, I accidentally killed our mail server while talking to 4 people."
So go ahead and remove Reply-All in the classic System Administrator "I don't care how my users want to use my network. It's mine and I'll do as I please." dick move. Because that's what it is. It's a Dick move and expect irate emails from users who suddenly find their email doesn't work very well anymore.
This is the government we're talking about. That would be above their pay grade.
I have direct experience that whenever a popup is presented reading something like.
Are you sure you want to do this stupid thing ?
pops up, people universally click "OK" without a second thought.
People have just been blasted by too many of these warnings to take any proper note any more.
Nullius in verba
How about conducting discussions on discussion boards rather than email?
Great Idea!
"Dear Vendor,
Since several people will need to be in on this conversation please visit our website at:
www.webboards.com and create a user account and password.
When you've registered you will need to visit the topic "Vendor discussions/XYZ Project" and pay special attention to the topic "How do we implement X without breaking Y or can we live without Y?".
We would really appreciate your input.
Signed,
Employee who's about to be fired."
Brilliant idea! A webboard! Now... I just need to send a mass email to everyone involved to let them know there is a new topic open and how to get to it. That sounds far more intuitive than just sending an email to 5 people!
There was at least one employee who actually spammed everyone for his direct marketing stuff... He got everyone which included the bosses ;).
That said, I think there actually should be a distribution list for the entire company - it can be useful for some stuff.
However the actual name should be hard to guess, and secret.
Then you set up the "everyone" list for people to send to which actually goes to a moderator.
If the moderator thinks the email should go out, it is sent out via the "secret-real-everyone-list", otherwise it isn't.
If the email indicates that the sender has significant lack of discretion or intelligence, the moderator may wish to pass it to the Bosses concerned so that they can take necessary measures.
In one of the places I worked for "everyone" actually went to the Big Boss(es), and I think it worked reasonably well.
It doesn't need to be secret if there are controls on who can send messages to the list. It is so trivial to do for any competent email admin no matter what software they use.
And next I expect you're going to try and teach them to edit their quotes and to stop top posting ?
I'm close to giving up on educating users with email, it's pretty hopeless I think.
Top posting is easier for most users to understand. For business users, its best to top post by default, unless you are going to counter a long email point by point. In that case please be sure to top post the words "see below."
On open mailing lists (anything not run by your employer where you decide to subscribe) I bottom post and edit posts. At work, I top post. It gives a complete linear history of a conversation, which is good because most outlook users just sort email by date.
Some people just can't handle reading properly formatted reply emails, let alone writing them.
--- Justin Dearing http://www.justaprogrammer.net/ We're just programmers.
Am I the only one to think that it is quite peculiar that it is happening 9 days before the Government turns over? I mean, how much difficult could it be to say that some sensitive/embarrasing mails got lost during this crash? I think this should be looked into in more detail and make double sure that no mail was "lost".