Slashdot Mirror
State Dept E-mail Crash After "Reply-All" Storm
twistah writes "It seems that a recent 'reply-all storm' at the State Department caused the entire e-mail infrastructure to crash. A notice sent to all State Department employees warned of disciplinary actions which will be taken if users 'reply-all' to lists with a large amount of users. Apparently, the problem was compounded by not only angry replies asking to be taken off the errant list, but by the e-mail recall function, which generated further e-mail traffic. One has to wonder if capacity planning was performed correctly — should an e-mail system be able to handle this type of traffic, or is it an unreasonable task for even the best system?"
http://msexchangeteam.com/archive/2004/04/08/109626.aspx
Again...
-Ghostis
Computer Science is all about trying to find the right wrench to bang in the right screw. -T.Cumbo?
Of the reply all button. Please do not respond with the reply all button. What they need is a reply some button.
Whoever wrote the headline for this summary needs to have their slashdot editor privileges revoked.
TFA states "an e-mail storm nearly knocked out one of the State Department's main electronic communications systems", and "a major interruption in departmental e-mail". The problem is clearly spelled out as "e-mail queues, especially between posts, back up while processing the extra volume of e-mails".
This is simply the queues backing up, not the servers crashing. Nowhere does TFA state anything to suggest that there was a "State Dept E-mail Crash", which the summary's headline boasts. The proper headline should read "Large E-mail Queues at State Dept After Reply-All Storm".
No, I'm not new here. That's why I'm fed up with the sensationalist "journalism" that is getting worse and worse here.
I remember my first year of college when I wanted to send Xmas greetings to 'everyone'. I remember, the IT director of the college running from computer lab to computer lab looking for student number xxyz.
Fun times.
Dear state department
I'm sorry to hear about your recent trouble
There is a brand new invention on the internet which have the ability to ease the strain on your mailservers. it is called maillist managers. one is called mailman and can be found at: http://www.gnu.org/software/mailman
There are several others, some free, and some non free, but they exist for most server platforms. If you don't have the expertice in house to set it up corrctly, you can get any number of consultancy companies to help you out.
Yours faithfull
Almost anonymous coward
OpenNet, by a very quick look on google, seem to be their network name for the non-classified bits and pieces. Supposedly Microsoft + Cisco stuff.
Feel free to disagree, but please provide a URL reference to the OpenNet email server software vendor if doing so.. ;-)
Maybe someone could introduce them to the concept of a BCC.
yes its exchange internally
openNet is what they brand it as
feel free to correct me with evidance that it was not the case any more but I know 2 exchange servers there and this say's otherwise
exchange has the recall ability and so does lotus notes
most other servers do not have this feature for very good reasons l
regards
John Jones
www.johnjones.me.uk my blog about email and digital communication
What they should have done if they were concerned about their load [which evidently they should have] was to warn their employees in blocks, perhaps 10% at a time with space between to take care of the massive response...
No. What they should have done was installed a mailing list manager, created a read-only list called "employees", and posted to it. Voila - n-thousand workers get announcements with no ability to reply to the whole list. Problem solved.
Dewey, what part of this looks like authorities should be involved?
This just in, President-Elect Obama Assassinated! Oh, don't take it so literally. I was just trying to capture your attention in a short amount of time. Obama wasn't killed, silly. There was just some CHARACTER assassination against him on a late night talk show.
I saw a weird variant on that back in university.
One of the engineering departments had a room full of (at the time) fairly high end sun workstations, and these were used both interactively and for people running longer compute jobs overnight.
To facilitate overnight jobs, the admins had set up a round robin dns alias that updated every couple of seconds to point to the machine reporting the lowest load average.
One of the students in my class had the bright idea of "If put 'ssh lowest' in my bashrc file, every time i open a terminal window it'll automatically pick the least loaded machine".
Fast forward a few minutes and we've got 80 sun workstations which have all systematically ssh'd to each other and none of which will accept any new connections...
I have direct experience that whenever a popup is presented reading something like.
Are you sure you want to do this stupid thing ?
pops up, people universally click "OK" without a second thought.
People have just been blasted by too many of these warnings to take any proper note any more.
Nullius in verba
Having been a witness to the incident in question, here's what happened:
1) Around December 30th a blank e-mail (with receipt request) went out to almost all users. Apparently it was from a single user with some malware etc. (we didn't get any further details).
2) The next day, the same blank message was sent out again (from the same user).
3) As people came back from vacations, we got a few "Please remove me from this list", and "What is this message" send as reply-all.
4) Then, followed with a bunch of "Me Too".
5) Then, a bunch of "Please, don't reply all" (sent, of course, reply-all).
6) Followed by a bunch of "remove me from this list".
and so on, and so forth, with no end in sight...
The initial message didn't have any virus or other "payload"; just a blank message that caused a bunch of confusion. The whole incident was actually pretty hilarious to watch.
Message recall. Oh dear.
Years ago, I wrote the bulk of this feature. It is not an Exchange feature, but an Outlook feature. It works by sending a custom MAPI message that Outlook recognizes and processes. Of course, this only works if all recipients are using Outlook. It also, after we did some usability testing, only deletes unread email, or email that has not been moved to a subfolder (the original version was quite determined and would hunt down and kill the message even if it had been moved to a subfolder, renamed or entered the email protection program). In this way, it did not violate the UI dictum that the computer move things around when you haven't given it instructions to do so.
So yes, it is Outlook only. If sent to a non-Microsoft mail system, it degrades to a simple notification that the message is being recalled. And it does not a good choice for getting rid of flames you shouldn't have been sending. But within its expected use as a feature - correcting mistakes in email that should have been caught before pressing send, it works fairly well.
But because it is client based, rather than an Exchange feature, it does cause a new mail message to be sent to each original recipient and, combined with a send-all storm, could greatly exacerbate things.
And, preemptively, for those who have philosophical objections to me having written the code in the first place, I'll just have to live with your disapproval and hope my steady paycheck somehow sooths my guilty conscience.