Slashdot Mirror
The NHS's 1.2 Million Employees Are Trapped in a 'Reply-All' Email Thread (businessinsider.com)
An anonymous reader shares a Business Insider report:The NHS's 1.2 million employees are currently trapped in a "reply-all" email hell. A "test" email was accidentally sent to everyone who works at the UK health service - prompting a series of reply-all responses from annoyed recipients going out to all 1-million-plus employees of the organisations. An NHS employee told Business Insider that there have been at least 120 replies so far -- meaning that more than 140 million needless emails have been sent across the NHS's network today. As a result, they said, its email systems are running "very slow today." The NHS Pensions department is currently warning people on Twitter that "if contacting us by email please be aware that there may be delays in responding due to an issue currently affecting all NHS mail."
Please remove me from this distribution, I don't know how I got on it.
I survived Bedlam DL3
The story of when Microsoft themselves fell victim to the same issue, and how it was resolved.
...email was God's gift to business. Transformative, empowering, a paradigm-shift.
It's Satan that added Reply-all, and then BCC just to continue the general fuckery.
-Styopa
Obligatory Dilbert:
http://dilbert.com/strip/2003-04-06
I've worked at a couple of companies that have sent out mass emails like this. My guess is that 1% of employees are stupid enough to hit "reply all" when requesting being removed from the email thread.
I seriously doubt that only 120 people in the NHS have hit reply all. My guess is that there will eventually be a few thousand who do this. That's assuming the NHS has above average intelligence employees.
I don't work for a company with over a million employees but I see this happen frequently. People reply all out of rote habit, not even consciously. It's so annoying. I know one company who customized their distribution of Outlook to not have a reply all button. Short of that, my recommendation is to either protect the large distribution group so that only a select few can email it and/or making use of BCC for that group when original emails are sent so that even if people do reply to all there is no further waste of time. I tried individually shaming people when they did it for a while, sending them dumbass instructions on the location of the Reply button versus Reply to All, but that had a limited impact other than for more people to know I'm an asshole. Oh well...
The simple solution is to lock down the "all" distribution list so only certain (very few) senders can send mail to it. I am not sure why this would not have been done... I am actually surprised that this hasn't happened before if everyone has send access to this list....
If course, if the distribution list was enumerated on the client prior to sending, that is a different story. But I can't imagine any client that would work reasonably with a million individual recipients.
My eyes reflect the stars and a smile lights up my face.
I have seen it few times in big corporations I worked in. Somebody sends email to wrong group by accident and then we have 3 waves of attack:
1) Clueless people hitting 'reply all' asking for removal from mailing group
2) Even more clueless people hitting 'reply all' asking people to not 'reply all'
3) "Champions" trying to save a day by putting all in BCC and telling people to not reply all, unless you put it in BCC [1]
And then, few hours later, next timezone wakes up and things start again.
Why is it useful? After it is obvious what is happening, you create folder called 'idiots' and redirect all these emails into that group by outlook/whatever rule. After that, if you need to deal with somebody in your organization, first check if he/she is in idiots folder and approach accordingly.
BTW, 120 replies seems very low. I have seen mailstorms with group of 10k recipients (it was not 'all' group, just some subset of company) generate over 600 replies total in these 3 waves. 120 replies from 1.2 million looks to be technical limitation (or, maybe, there was some hero in IT department who pulled the plug fast enough...)
[1] - My favorite is self correcting champion, which first sends 'reply all' and then does reply to that with everybody in BCC saying he should have put everybody in BCC in first place...
they have ...
https://www.theguardian.com/so...
still will take time though
IT contractors eh!
who where what when now?
Comment removed based on user account deletion
Emailing a group like that should have been limited to a very small number of people. Everyone else should have got a message that they were not authorized to use that email message.
This is email server admin 101 level knowledge.
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
It doesn't have to be of unlimited length. In Outlook/Exchange, at least, it's possible to have a distribution group that is handled by the server, so including the group name in the To: (or CC: or BCC:) field will send it to everyone in that group, no matter how big. My organization has a #Everyone group that does actually go to everyone. I don't know if #Everyone is protected, but there are certainly some very large distribution groups- around 1/3 of the organization- that anyone is allowed to send to.
There's no point in questioning authority if you aren't going to listen to the answers.
A bean counter at a local hospital sent out an email to everyone that explained the cost in lost productivity for each "reply all" email was $0.08 per person. Not surprisingly, someone hit the "reply all" button to respond that the bean counter's email cost the hospital $800 in lost productivity. It went downhill from there. Not sure if the tab was $80,000 or $800,000 in lost productivity when everyone stopped hitting "reply all" button. Executive management wasn't amused.
Which requires an Emergency Change Request (which takes hours to get approved for a Sev 2).
"I don't know, therefore Aliens" Wafflebox1
i worked at NG for 7 years, and there were several instances (the last one being in 2010) where email system company-wide was crippled or knocked offline by an email that was sent to the wrong mailing list. apparently, there was one available that included literally every single person in the system (probably about 100,000 people).
i remember one morning in about 2008 or so, getting an email addressed to some team i wasn't a part of, seeing the "CC" list was several miles long, and i knew instantly what was going to happen. i guess the "first post" instinct in me acted up for the first time ever. i knew we were all already doomed, so i hit reply-all and simply posted: "oh no, not again." i did manage to be first, but before i could blink, i had over a thousand new emails all saying some variation of "WHAT IS THIS?" "REMOVE ME FROM THIS LIST" and "STOP REPLYING FOR GODS SAKE". my new emails hit 30,000 in a few minutes.
the entire NG email system was down for more than a day.
two days later i got called into my boss's office and he explained that top-tier management at NG had demanded that i be fired. my "oh no not again" was the last email most people saw before the system exploded. a very heated conversation between my supervisors and NG executives apparently just barely saved my job, but my supervisors were not pleased either and mentioned this would go on my permanent record (i thought that was just a high school thing). it didn't matter that i didn't actually do anything to cause the crash. i had merely made myself visible at the wrong time, and NG wanted someone's head.
so glad i don't work there now.
i could live a little longer in this prison
Probably because the CTO knew that it is his head that should roll since the email system that allowed such fuckery happened on his watch.
Got to find a low level scapegoat.
I'm a good cook. I'm a fantastic eater. - Steven Brust
I get an email for a division-wide thing.
I get a copy from my VP's admin, specifically targeting my team in case we weren't on the original distribution list.
I get a copy of this from my manager, since he doesn't want me to miss this.
I also get a copy of the original from a corporate level special interest group I'm part of.
Then I get a copy from a former team member. Just in case I was left of their distribution, since they left our team but believe they may be getting team emails that current team members are not.
And a copy from their manager, with a note to be sure the distribution list I cannot administer is properly updated to get these mails from the list they should not be and indeed are not part of. Just in case.
Then I get a copy from an interested team leader who wants to make sure we are in the loop.
And another from their #1 team member, who looks out for us.
And finally my cubicle mate leans over and tell me 'hey, did you get the email from......'
And so I have a 14GB .OST that i cannot backup locally due to GPO. and i get warnings occasionally that my file will be groomed back to an unspecified maximum size. Some day, real soon now. Right after they encrypt my files for no apparent reason, in accordance with some policy I cannot get a copy of.
I'm not bitter, really. I feel for the corporate security and cloud services guys. They can't fix stupid.
deleting the extra space after periods so i can stay relevant, yeah.
If /. was a British site I would assume the UK Govt health service, but its American, and the first thing that came up when I googled was National Honor Society
Theres also quite a few schools using that TLA and the National Highway System
Yes, but you see, Mailman is an old-fashioned piece of software for an operating system that comes from the '70's.
Modern, advanced software from Microsoft (tm) will ensure emails will be delivered to everybody, under every circumstance.
-- Sometimes you have to turn the lights off in order to see.
They would have saved money (read: time) at this point if they just distributed memoranda on paper.
To err is human.
To really fuck up requires a computer.
If you allow your large distribution lists to be used by all, you are handing your own users a loaded weapon with a round chambered. You'd think professional email sysadmins would know this, sigh
Nothing is simple when you have a million users. Sure, it's quite easy to do a quick search and delete on a small number of mailboxes but for a million? It's a shit ton of servers and mailboxes to search through. At least it works better than it used to... Still, i'm surprised that they didn't restrict who could send to the distribution group (I"m pretty sure they are using MS Exchange.) There are lots of examples for scripts to do the search & delete but it probably has to be broken up into lots of smaller batches to avoid killing servers or hitting memory limits. Get-Mailbox -OrganizationalUnit "domain.local/Users" -resultsize unlimited | Search-Mailbox -SearchQuery 'Subject:"PDF version of memo" ' , 'From:"email@domain.com” ',”Sent:03/02/2015" –DeleteContent -TargetMailbox “target@domain.com” -TargetFolder “Delete” -loglevel full
Back in the 90s I was in college and the CS department implemented some mailing-lists per course and one for the whole department (this was on a Unix server). Someone was smart enough to subscribe the mailing-list to itself (and they didn't saw it coming). He successfully mail bombed all the accounts in the department and crashed the server :). Evidently the CS department was not amused, proclaiming they would certainly find the prankster but back in those days you could just probably have done this by telnet-ting the smtp server so nope ...
:-)
Another classic in big companies (and has happened several times at one I worked for) is people trying to send an e-mail to a department in the company like HR, finding a list named something like "HR_Dallas" in the global address listing not knowing this is not the HR Dallas department but rather the list that sends an e-mail to all Dallas employees. Yep, you just send your private confidential mail to all the employees