Slashdot Mirror
The NHS's 1.2 Million Employees Are Trapped in a 'Reply-All' Email Thread (businessinsider.com)
An anonymous reader shares a Business Insider report:The NHS's 1.2 million employees are currently trapped in a "reply-all" email hell. A "test" email was accidentally sent to everyone who works at the UK health service - prompting a series of reply-all responses from annoyed recipients going out to all 1-million-plus employees of the organisations. An NHS employee told Business Insider that there have been at least 120 replies so far -- meaning that more than 140 million needless emails have been sent across the NHS's network today. As a result, they said, its email systems are running "very slow today." The NHS Pensions department is currently warning people on Twitter that "if contacting us by email please be aware that there may be delays in responding due to an issue currently affecting all NHS mail."
Please remove me from this distribution, I don't know how I got on it.
I survived Bedlam DL3
The story of when Microsoft themselves fell victim to the same issue, and how it was resolved.
This sort of thing happens all the time. You can mediate it a bit by setting the permissions on some of the global type DLs so that only certain people can use them. But people are stupid. Heck, all the time you get people putting a conference call on hold (causing music to everyone on the call). People that stupid always do a reply all.
...email was God's gift to business. Transformative, empowering, a paradigm-shift.
It's Satan that added Reply-all, and then BCC just to continue the general fuckery.
-Styopa
Most MUAs won't accept To/CC/BCC of unlimited length.
They should send out an email to everyone advising them not to reply.... Oh and don't reply to this one either or we might be forced to send another in a similar vain.
Obligatory Dilbert:
http://dilbert.com/strip/2003-04-06
I've worked at a couple of companies that have sent out mass emails like this. My guess is that 1% of employees are stupid enough to hit "reply all" when requesting being removed from the email thread.
I seriously doubt that only 120 people in the NHS have hit reply all. My guess is that there will eventually be a few thousand who do this. That's assuming the NHS has above average intelligence employees.
Mailman has a "Pause all mailing lists" button for flame wars.
I don't work for a company with over a million employees but I see this happen frequently. People reply all out of rote habit, not even consciously. It's so annoying. I know one company who customized their distribution of Outlook to not have a reply all button. Short of that, my recommendation is to either protect the large distribution group so that only a select few can email it and/or making use of BCC for that group when original emails are sent so that even if people do reply to all there is no further waste of time. I tried individually shaming people when they did it for a while, sending them dumbass instructions on the location of the Reply button versus Reply to All, but that had a limited impact other than for more people to know I'm an asshole. Oh well...
The simple solution is to lock down the "all" distribution list so only certain (very few) senders can send mail to it. I am not sure why this would not have been done... I am actually surprised that this hasn't happened before if everyone has send access to this list....
If course, if the distribution list was enumerated on the client prior to sending, that is a different story. But I can't imagine any client that would work reasonably with a million individual recipients.
My eyes reflect the stars and a smile lights up my face.
I have seen it few times in big corporations I worked in. Somebody sends email to wrong group by accident and then we have 3 waves of attack:
1) Clueless people hitting 'reply all' asking for removal from mailing group
2) Even more clueless people hitting 'reply all' asking people to not 'reply all'
3) "Champions" trying to save a day by putting all in BCC and telling people to not reply all, unless you put it in BCC [1]
And then, few hours later, next timezone wakes up and things start again.
Why is it useful? After it is obvious what is happening, you create folder called 'idiots' and redirect all these emails into that group by outlook/whatever rule. After that, if you need to deal with somebody in your organization, first check if he/she is in idiots folder and approach accordingly.
BTW, 120 replies seems very low. I have seen mailstorms with group of 10k recipients (it was not 'all' group, just some subset of company) generate over 600 replies total in these 3 waves. 120 replies from 1.2 million looks to be technical limitation (or, maybe, there was some hero in IT department who pulled the plug fast enough...)
[1] - My favorite is self correcting champion, which first sends 'reply all' and then does reply to that with everybody in BCC saying he should have put everybody in BCC in first place...
they have ...
https://www.theguardian.com/so...
still will take time though
IT contractors eh!
who where what when now?
Comment removed based on user account deletion
Emailing a group like that should have been limited to a very small number of people. Everyone else should have got a message that they were not authorized to use that email message.
This is email server admin 101 level knowledge.
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
A bean counter at a local hospital sent out an email to everyone that explained the cost in lost productivity for each "reply all" email was $0.08 per person. Not surprisingly, someone hit the "reply all" button to respond that the bean counter's email cost the hospital $800 in lost productivity. It went downhill from there. Not sure if the tab was $80,000 or $800,000 in lost productivity when everyone stopped hitting "reply all" button. Executive management wasn't amused.
And this highlights one of the many uses of BCC
i worked at NG for 7 years, and there were several instances (the last one being in 2010) where email system company-wide was crippled or knocked offline by an email that was sent to the wrong mailing list. apparently, there was one available that included literally every single person in the system (probably about 100,000 people).
i remember one morning in about 2008 or so, getting an email addressed to some team i wasn't a part of, seeing the "CC" list was several miles long, and i knew instantly what was going to happen. i guess the "first post" instinct in me acted up for the first time ever. i knew we were all already doomed, so i hit reply-all and simply posted: "oh no, not again." i did manage to be first, but before i could blink, i had over a thousand new emails all saying some variation of "WHAT IS THIS?" "REMOVE ME FROM THIS LIST" and "STOP REPLYING FOR GODS SAKE". my new emails hit 30,000 in a few minutes.
the entire NG email system was down for more than a day.
two days later i got called into my boss's office and he explained that top-tier management at NG had demanded that i be fired. my "oh no not again" was the last email most people saw before the system exploded. a very heated conversation between my supervisors and NG executives apparently just barely saved my job, but my supervisors were not pleased either and mentioned this would go on my permanent record (i thought that was just a high school thing). it didn't matter that i didn't actually do anything to cause the crash. i had merely made myself visible at the wrong time, and NG wanted someone's head.
so glad i don't work there now.
i could live a little longer in this prison
And only a small division with a thousand or so employees.
There had to be threats of discipline from upper management to get it to stop.
I'm a good cook. I'm a fantastic eater. - Steven Brust
Probably because the CTO knew that it is his head that should roll since the email system that allowed such fuckery happened on his watch.
Got to find a low level scapegoat.
I'm a good cook. I'm a fantastic eater. - Steven Brust
I get an email for a division-wide thing.
I get a copy from my VP's admin, specifically targeting my team in case we weren't on the original distribution list.
I get a copy of this from my manager, since he doesn't want me to miss this.
I also get a copy of the original from a corporate level special interest group I'm part of.
Then I get a copy from a former team member. Just in case I was left of their distribution, since they left our team but believe they may be getting team emails that current team members are not.
And a copy from their manager, with a note to be sure the distribution list I cannot administer is properly updated to get these mails from the list they should not be and indeed are not part of. Just in case.
Then I get a copy from an interested team leader who wants to make sure we are in the loop.
And another from their #1 team member, who looks out for us.
And finally my cubicle mate leans over and tell me 'hey, did you get the email from......'
And so I have a 14GB .OST that i cannot backup locally due to GPO. and i get warnings occasionally that my file will be groomed back to an unspecified maximum size. Some day, real soon now. Right after they encrypt my files for no apparent reason, in accordance with some policy I cannot get a copy of.
I'm not bitter, really. I feel for the corporate security and cloud services guys. They can't fix stupid.
deleting the extra space after periods so i can stay relevant, yeah.
Back in I think 1996, when I was in the Air Force, a contractor sent to everybody on base the dancing baby animation. I think it was several megabytes in size. Nobody even had to reply to it for the mail system to crash.
I'm curious what email client they use at the DHS. I feel like Gmail or Outlook would totally explode if you tried cc-ing 1.2M people.
Fast Federal Court and I.T.C. updates
this is why i simply ignore these emails
a lot simpler than getting outraged and emailing everyone to be taken off
yeah, when i was at Vicenza, Italy in the US Army around this time we did the same thing. only we did it on purpose to try to crash the email system
There is no earthly reason why everyone should be allowed to send emails with 1.2 million recipients. Any email with more than some reasonable number (say maybe 100ish? Certainly not more that 1000 even with that big if an org) should not be allowed to be sent except from a very small list of authorized users.
Unless of course management caught a bad case of the dumbass and forced the admins to leave that setting on. In which case the managers should be fired.
I blame the email provider (Microsoft, I assume in this case.) The "Reply all" button should have never been included without some sort of throttle/warning mechanism. Something as simple as a pop-up saying "You are about to Reply all to x people. Are you sure you want to do this?" with the message and threshold number configurable by an administrator. Really hard core admins could set the message to something like "Management has determined that all useless emails will be counted, and the offender terminated when the count reaches xxx. Have a nice day."
If /. was a British site I would assume the UK Govt health service, but its American, and the first thing that came up when I googled was National Honor Society
Theres also quite a few schools using that TLA and the National Highway System
Yes, but you see, Mailman is an old-fashioned piece of software for an operating system that comes from the '70's.
Modern, advanced software from Microsoft (tm) will ensure emails will be delivered to everybody, under every circumstance.
-- Sometimes you have to turn the lights off in order to see.
This only applies to places big enough to have an "email server admin". In a lot of small shops, there is only one IT guy doing all the work from installing ethernet cables, helping users configure their OS and programs, setting up WordPress and email accounts. He can't know everything on all subjects.
They would have saved money (read: time) at this point if they just distributed memoranda on paper.
To err is human.
To really fuck up requires a computer.
They have 1.2 million users. That's big enough to have an "email server admin." In fact, it's big enough to have two. They should get some. Not having any hasn't worked out too well for them.
Or so those of us that pay for it are lead to believe.
Sent from my ASR33 using ASCII
"Having used the NHS in the UK, and the US medical system, I can unequivocally say ..."
Who cares, anecdote dude.
Fucking annoying. Had to change my email address because of those whiny shits.
You could have unsubscribed. No need to play the victim game.
If you allow your large distribution lists to be used by all, you are handing your own users a loaded weapon with a round chambered. You'd think professional email sysadmins would know this, sigh
Comment removed based on user account deletion
Comment removed based on user account deletion
Nothing is simple when you have a million users. Sure, it's quite easy to do a quick search and delete on a small number of mailboxes but for a million? It's a shit ton of servers and mailboxes to search through. At least it works better than it used to... Still, i'm surprised that they didn't restrict who could send to the distribution group (I"m pretty sure they are using MS Exchange.) There are lots of examples for scripts to do the search & delete but it probably has to be broken up into lots of smaller batches to avoid killing servers or hitting memory limits. Get-Mailbox -OrganizationalUnit "domain.local/Users" -resultsize unlimited | Search-Mailbox -SearchQuery 'Subject:"PDF version of memo" ' , 'From:"email@domain.com” ',”Sent:03/02/2015" –DeleteContent -TargetMailbox “target@domain.com” -TargetFolder “Delete” -loglevel full
Comment removed based on user account deletion
If you've successfully unsubscribed, you need to share your method with the world.
Scroll down to the bottom of the email, click on the unsubscribe link, and follow the directions on the webpage. That simple.
Or, you know, Exchange also has a bunch of tools allowing you to do the same thing, but incompetent IT will be incompetent regardless of the tools (or manglement if they overrode IT decisions about it).
Back in the 90s I was in college and the CS department implemented some mailing-lists per course and one for the whole department (this was on a Unix server). Someone was smart enough to subscribe the mailing-list to itself (and they didn't saw it coming). He successfully mail bombed all the accounts in the department and crashed the server :). Evidently the CS department was not amused, proclaiming they would certainly find the prankster but back in those days you could just probably have done this by telnet-ting the smtp server so nope ...
:-)
Another classic in big companies (and has happened several times at one I worked for) is people trying to send an e-mail to a department in the company like HR, finding a list named something like "HR_Dallas" in the global address listing not knowing this is not the HR Dallas department but rather the list that sends an e-mail to all Dallas employees. Yep, you just send your private confidential mail to all the employees
Testing, testing, testing
https://youtu.be/_Cmv58lTWgU
I worked at a place where someone went on maternity leave and had an Out of Office set up that somehow ended up causing an infinite loop as soon as someone sent her an email. I forget how it happened, but it was fun watching hundreds and thousands of emails appearing in everyone's inbox for a few minutes until they pulled the plug on it.
Two things, and Email System that has 1.2 Million people should have a few Email Admins at various levels of authority. The other thing is, a small shop (for various definitions of "small" ) wouldn't have nearly the problem a system with 1.2 Million accounts has, so getting it right wouldn't be nearly the same issue has NIH has.
That being said, It is a five minute configuration on just about any system to setup security to prevent this sort of thing. Size of shop doesn't actually matter in how much time it takes to setup. And the headaches prevented, large or small, is worth the time to do it right the first time.
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
Our study abroad administrator didn't understand how email worked, didn't know how email *lists* worked, and didn't know you could suppress the email field via BCC.
She hand-typed the email address of every single student into a standard CC email field at a time when we only had something like 300KB of space for our *entire* email. The header alone was larger than that, given that we had over 2000 students. And *that* was before the "Reply-Alls" started rolling in. You could still send mail with your email storage full, it just wouldn't save the outgoing message, so the entire server filled up in minutes. Response time went through the floor. It took IT all afternoon to sort the whole thing out.
And then, two days later, she did it again.
Actually they have 1.2 million Employees, only 850,000 have an Email address.
Apocalypse Cancelled, Sorry, No Ticket Refunds
No single raindrop believes it is to blame for the flood. poster
The underlying premise is the Government knows how to orchestrate all this health activity better than ordinary people.
You see the government can spot the wastefulness of giving cataract medication to someone before they go blind in one eye. (A true NHS policy, btw).
And the government understands how politicians should get large lump sums of money for the more people get put on the kill lists. (Part of the VA/Walter Reed thing).
And if everyone was enlightened as the Government, they would email bomb each other in waves of sophisticated confusion everyday, just like the NHS here.
But ordinary people are not as enlightened as the Government.
They have 1.2 million users. That's big enough to have an "email server admin." In fact, it's big enough to have two. They should get some. Not having any hasn't worked out too well for them.
Well....if they're using MS Exchange they'd need a team of 4 or 5 at least. So yes, someone should have known how to set the groups properly so that this could not happen. Of course, that's assuming it went to "Everyone" or a similar all-encompassing group. It could just as easily been a single email that contains a bunch of groups too...read TFA to find out (I would assume).
Truth is like the sun. You can shut it out for a time, but it ain't goin' away. - Elvis Presley (source: imdb.com)
It's obviously impossible to specify 1M+ recipients on a To: or Cc: header, so there must be an alias or list expansion that goes to all. Simply disable that alias until things calm down. It's not rocket science.
Sometime in the 2000's an email made the rounds with a large power point slide show of pictures depicting a foam fire suppressant system test that went better than expected. I want to say that the attachment was 35 megs but that seems a bit over the top. Anyways the unit from whence the photos originated was the subject of a lot of mockery among airmen everywhere though there wasn't any re-alls going on with it that I saw. Within a day or two though an engineer from that group sent out an email seemingly across the entire Air Force insisting the test went perfectly as planned and that everyone should stop laughing about it and wasting Air Force resources forwarding the email.
Apocalypse Cancelled, Sorry, No Ticket Refunds
The NHS and the WHO collaborate, but they are different organizations. Both have doctors, Dr NHS and Dr WHO is what they are called.
__
Men with no respect for life must never be allowed to control the ultimate instruments of death.
GW Bu
I perticularly liked the help desk emails going out to the same distribution list saying 'Your incident has been raised with the Help Desk. Your incident number is 5235" ..
Anyone else want to check my math, but assuming the average email address is 20 characters long (that's on the low side from what I can find), the list of recipients on each email should be about 24 Megabytes? That's a lot of bandwidth even without these ridiculous quantities. Or is it just a mailing list name on the "TO" line and the mail-server looks up all 1.2 million addresses from its own table?
Comment removed based on user account deletion
It was sent to a distribution list, which to any reasonable person would be expected to have a very small distribution. The list was called "CroydonPractices", so presumably intended for the primary care practice managers in the Croydon region of London.
It is not at all clear who set up the distribution list, or whether it was the same person who sent the test e-mail. The issue was that this particular distribution list for some reason included all users.
I don't know what precautions the admin tools have to prevent mass replies. I think that some of the later versions of outlook will warn users sending e-mail to distribution lists if the number of recipients is greater than a certain number. However, according to wikipedia the backend for NHSmail is a customised version of outlook 2007, so it may be lacking some of the more modern features.
Emailing a group like that should have been limited to a very small number of people. Everyone else should have got a message that they were not authorized to use that email message.
This is email server admin 101 level knowledge.
Then your user gets frustrated and starts manually adding names to the To: or Cc: and voila, instant meltdown again, just without distro lists.
I have seen To: lines with hundreds, possibly thousands of email addresses in it. You would think there would be some sort of limit on the amount of individual addresses... but meh.
"Someone needs to talk to the tree of liberty about its ghoulish drinking problem." by ohnocitizen
The 4 star was on the distro and replied all saying that any further replies would result in the person and their supervisor having to explain themselves to him tge next week. Was dead quiet after that & gave the mail server admins time to fix the underlying issue.
Long ago, in a galaxy far, far away, my secretary did a broadcast email of 100MB in photos to my entire company. Our home-brew mail server practically exploded.
oh please, if i hadn't replied, the other 100,000 replies would have still happened. i knew they were already on their way. it's not like i could have stopped it.
i could live a little longer in this prison
We had something similar happen at a previous employer. They took exchange down for 2 or 3 minutes while they did some sort of cascade delete and all traces of the e-mail were gone from the entire company.
People are stupid... This happens at many employers. People still reply to all, it's really annoying. You simply cannot rely on people to not reply all, be smart. When sending out email to a large number of people you absolutely MUST USE the BCC line instead of TO. That way they can't reply all to everyone on distribution. Just leave that TO line blank...
I know brexit was supposed to brink UK back to stone age, but can't the NHS IT staff just filter an address server-side?
Why is this even a /. story?
Because it's precisely, exactly, within Slashdot's remit. You're a twat.
I do not want your cheap brainburning drugs. They are useless for work. And I am a working man today.
Another data point that shows the government attracts some of the worst employees.
I suspect you will rapidly forget any evidence to the contrary.
I do not want your cheap brainburning drugs. They are useless for work. And I am a working man today.
4) after it has all been resolved, and things are back to normal, those people who were on vacation return to work and start it all over again by doing a 'reply all' asking to be removed from the list.
My beliefs do not require that you agree with them.