Slashdot Mirror
State Dept E-mail Crash After "Reply-All" Storm
twistah writes "It seems that a recent 'reply-all storm' at the State Department caused the entire e-mail infrastructure to crash. A notice sent to all State Department employees warned of disciplinary actions which will be taken if users 'reply-all' to lists with a large amount of users. Apparently, the problem was compounded by not only angry replies asking to be taken off the errant list, but by the e-mail recall function, which generated further e-mail traffic. One has to wonder if capacity planning was performed correctly — should an e-mail system be able to handle this type of traffic, or is it an unreasonable task for even the best system?"
http://msexchangeteam.com/archive/2004/04/08/109626.aspx
Again...
-Ghostis
Computer Science is all about trying to find the right wrench to bang in the right screw. -T.Cumbo?
What an irony that they decided to mass mail when they've warned their employees not to do so. What they should have done if they were concerned about their load [which evidently they should have] was to warn their employees in blocks, perhaps 10% at a time with space between to take care of the massive response... However, judging by the nature of their work [it is the state department after all] I don't believe it unreasonable that there could be events in their future requiring such mass mailings again and having the whole system crash under the load would be no doubt very bad in emergencies.
Sigs are too short to say anything truly profound so read the above post instead.
Of the reply all button. Please do not respond with the reply all button. What they need is a reply some button.
Whoever wrote the headline for this summary needs to have their slashdot editor privileges revoked.
TFA states "an e-mail storm nearly knocked out one of the State Department's main electronic communications systems", and "a major interruption in departmental e-mail". The problem is clearly spelled out as "e-mail queues, especially between posts, back up while processing the extra volume of e-mails".
This is simply the queues backing up, not the servers crashing. Nowhere does TFA state anything to suggest that there was a "State Dept E-mail Crash", which the summary's headline boasts. The proper headline should read "Large E-mail Queues at State Dept After Reply-All Storm".
No, I'm not new here. That's why I'm fed up with the sensationalist "journalism" that is getting worse and worse here.
I remember my first year of college when I wanted to send Xmas greetings to 'everyone'. I remember, the IT director of the college running from computer lab to computer lab looking for student number xxyz.
Fun times.
a) Maintaining large list by copying all recipients into the hrader is a fucked up idea at best (because there is no way this list will be kept updated), and a informaiton leak at worst (because somebody eralier on a non-updated list may get information which he should not get - e.g. former employees). Why do governmental institutions still us it?
b) Why in the world do modern e-mail clients still allow reply all to hundreds of recipients without an additional safety question. I would expect my program would warn me before sending an emails to thousand people.
Dear state department
I'm sorry to hear about your recent trouble
There is a brand new invention on the internet which have the ability to ease the strain on your mailservers. it is called maillist managers. one is called mailman and can be found at: http://www.gnu.org/software/mailman
There are several others, some free, and some non free, but they exist for most server platforms. If you don't have the expertice in house to set it up corrctly, you can get any number of consultancy companies to help you out.
Yours faithfull
Almost anonymous coward
OpenNet, by a very quick look on google, seem to be their network name for the non-classified bits and pieces. Supposedly Microsoft + Cisco stuff.
Feel free to disagree, but please provide a URL reference to the OpenNet email server software vendor if doing so.. ;-)
Good to know that rigorous competition in the marketplace has totally eliminated misuse of 'reply-all' in the private sector. I look forward to continuing to have a lower life expectancy and higher infant mortality than Canadians and Swedes.
Why is that funny?
Exchange has a feature your email client didn't support. Ha ha ha!! IT'S HILARIOUS!!!!
Comment of the year
And in even further news, corporations are not perfect.
I take it you're not familiar with how enterprises plan. They plan for regular load, not aberrant once-in-a-blue-moon load. This is bog standard behavior for a system responding to people doing stupid things. If you think this is restricted to the US government, you've never worked in corporate IT.
Those who can, do. Those who can't, sue.
yes its exchange internally
openNet is what they brand it as
feel free to correct me with evidance that it was not the case any more but I know 2 exchange servers there and this say's otherwise
exchange has the recall ability and so does lotus notes
most other servers do not have this feature for very good reasons l
regards
John Jones
www.johnjones.me.uk my blog about email and digital communication
This is the sort of thing that listservs seem to do pretty well.
I just wish I could convince more of my users to use them. I have one winner who sends a list using 300+ CC's. The anti-spam system on the mail server slows that list to a crawl (deliberately). They wonder why it takes 3 hours to send, and I tell them to use the list server that we set up, but it's different and they don't want to be bothered. I think I'll make it take 6 hours next time.
This just in, President-Elect Obama Assassinated! Oh, don't take it so literally. I was just trying to capture your attention in a short amount of time. Obama wasn't killed, silly. There was just some CHARACTER assassination against him on a late night talk show.
The problem is the message replied to having - RTFA - several thousand addresses in the To: and CC: fields. This is what BCC is for . Allowing people to put several thousand addresses in to the headers will eventually result in a mail storm, whether someone hits Reply To All or not. The first time someone opens a virus laden attachment that goes through their (archived by law, this being a federal agency) emails, it will send itself out to thousands of equally clueless people. One of them will run the attachment, which will send another copy to several thousand people. And so on. This happened where I work once, by people who should have known better. Before it was done, I was getting two hundreds copies of the virus per day.
Whoever sent out the message replied to should be fired and criminally prosecuted for deliberately sabotaging the State Department's email system. But since the article doesn't mention this at all, I'm assuming it was some dumbass boss somewhree who is immune to any form of disclipline for anything, up to and including murder.
I saw a weird variant on that back in university.
One of the engineering departments had a room full of (at the time) fairly high end sun workstations, and these were used both interactively and for people running longer compute jobs overnight.
To facilitate overnight jobs, the admins had set up a round robin dns alias that updated every couple of seconds to point to the machine reporting the lowest load average.
One of the students in my class had the bright idea of "If put 'ssh lowest' in my bashrc file, every time i open a terminal window it'll automatically pick the least loaded machine".
Fast forward a few minutes and we've got 80 sun workstations which have all systematically ssh'd to each other and none of which will accept any new connections...
"The term Deltic (meaning in the form of the Greek letter Delta) is used to refer to both the Deltic E.130 opposed-piston high-speed diesel engine designed and produced by D Napier & Son, and the locomotives produced by English Electric using these engines, including their demonstrator locomotive named DELTIC and the production version for British Railways, who designated these as (TOPS) Class 55."
For a train your English is quite good.
Not a Twitter sockpuppet... but I wish I was.
The guy who wants to quit but doesn't because he'll only get unemployment benefits if he's fired :)
Um...which goes to show how little you know about unemployment. At least in MA, you don't get shit if it is "termination with cause", ie fired. If you're laid off, great- but even then, your employer gets a phone call from the unemployment department asking whether you were fired or laid off. Nothing stops them from lying and saying you were fired with cause- and then you've got a legal battle on your hands, which you can't afford.
Other fun facts about unemployment in MA: you don't get paid for two full weeks after you FILED- not after you were laid off, but after you FILED. You get a pittance compared to your normal salary; you'd be lucky to make rent on a studio apartment in Boston based off an entire month's unemployment checks.
Any income is deducted from your UA check. Say for example you find a 2-3 hour consulting thing on CL and make $150 helping someone fix their computer. Guess what? Your unemployment check for that week will be $150 smaller. This basically means that you have no incentive to find any kind of income while you're on UA.
Last but certainly not least: you have to pay taxes, medicare, medicaid, etc on your unemployment benefits. It's not bad enough that you're basically on welfare- you have to fork over a portion of the money the government is giving you, BACK to the government. Cute, eh?
Please help metamoderate.
Message recall. Oh dear.
Years ago, I wrote the bulk of this feature. It is not an Exchange feature, but an Outlook feature. It works by sending a custom MAPI message that Outlook recognizes and processes. Of course, this only works if all recipients are using Outlook. It also, after we did some usability testing, only deletes unread email, or email that has not been moved to a subfolder (the original version was quite determined and would hunt down and kill the message even if it had been moved to a subfolder, renamed or entered the email protection program). In this way, it did not violate the UI dictum that the computer move things around when you haven't given it instructions to do so.
So yes, it is Outlook only. If sent to a non-Microsoft mail system, it degrades to a simple notification that the message is being recalled. And it does not a good choice for getting rid of flames you shouldn't have been sending. But within its expected use as a feature - correcting mistakes in email that should have been caught before pressing send, it works fairly well.
But because it is client based, rather than an Exchange feature, it does cause a new mail message to be sent to each original recipient and, combined with a send-all storm, could greatly exacerbate things.
And, preemptively, for those who have philosophical objections to me having written the code in the first place, I'll just have to live with your disapproval and hope my steady paycheck somehow sooths my guilty conscience.
and of course, off-topic from TFA, I signed up with a Product Testing Place. They email me once every six months and see if I want to test some new gadget or something and I get paid $75.
I signed a confidentiality agreement with them.
I am not allowed to discuss ANYTHING about the product or reveal I am testing it or anything. I was never there, I am nobody.
Last year I got an email - From The President of The Testing Company - personally thanking me for all the help in the last year.
He also thanked everyone else who "helped" last year as well and I could see who they were because apparently the President (or the secretary) just put all our emails into the TO: field and let it fly.
Lots of Identifiable people on the list because they used their WORK email, like john.doe@largecorporation.com So it was easy to see who else was part of that big Butt Plug testing program.
I did a REPLY to ONLY the President and laid into him about the confidentiality agreement and told him if he didn't know how to use email to stay away from the computer.
Later that day we all got another email from the President, this time apologizing for revealing all our personal emails, never happen again etc etc. And apparently he figured out how to use BCC!
So yelling at someone does seem to work to change behaviour.
Also- this is a dupe comment, I posted this once before on Slashdot someplace, but since this is Slashdot I didn't think a dupe would be a problem.
I like microcars
Looks like the pathetic one is you, and the Submitter. If you RTFA, it clearly says
He said the result was "effectively a denial of service as e-mail queues, especially between posts, back up while processing the extra volume of e-mails.
Never says the actually crashed, merely that the high volume generated large queues, exactly what you would expect to happen in a properly engineered system. But hey, this is Slashdot, so making up reasons to hate Exchange (and there are plenty of LEGITIMATE reasons to hate exchange) is the norm.
And next I expect you're going to try and teach them to edit their quotes and to stop top posting ?
I'm close to giving up on educating users with email, it's pretty hopeless I think.
Top posting is easier for most users to understand. For business users, its best to top post by default, unless you are going to counter a long email point by point. In that case please be sure to top post the words "see below."
On open mailing lists (anything not run by your employer where you decide to subscribe) I bottom post and edit posts. At work, I top post. It gives a complete linear history of a conversation, which is good because most outlook users just sort email by date.
Some people just can't handle reading properly formatted reply emails, let alone writing them.
--- Justin Dearing http://www.justaprogrammer.net/ We're just programmers.