Slashdot Mirror
The NHS's 1.2 Million Employees Are Trapped in a 'Reply-All' Email Thread (businessinsider.com)
An anonymous reader shares a Business Insider report:The NHS's 1.2 million employees are currently trapped in a "reply-all" email hell. A "test" email was accidentally sent to everyone who works at the UK health service - prompting a series of reply-all responses from annoyed recipients going out to all 1-million-plus employees of the organisations. An NHS employee told Business Insider that there have been at least 120 replies so far -- meaning that more than 140 million needless emails have been sent across the NHS's network today. As a result, they said, its email systems are running "very slow today." The NHS Pensions department is currently warning people on Twitter that "if contacting us by email please be aware that there may be delays in responding due to an issue currently affecting all NHS mail."
Please remove me from this distribution, I don't know how I got on it.
Mail everyone@nhs.co.uk for extra fun!
Gee, why don't they just disable the alias until the issue sorts itself out?
I survived Bedlam DL3
The story of when Microsoft themselves fell victim to the same issue, and how it was resolved.
This sort of thing happens all the time. You can mediate it a bit by setting the permissions on some of the global type DLs so that only certain people can use them. But people are stupid. Heck, all the time you get people putting a conference call on hold (causing music to everyone on the call). People that stupid always do a reply all.
...email was God's gift to business. Transformative, empowering, a paradigm-shift.
It's Satan that added Reply-all, and then BCC just to continue the general fuckery.
-Styopa
Most MUAs won't accept To/CC/BCC of unlimited length.
They should send out an email to everyone advising them not to reply.... Oh and don't reply to this one either or we might be forced to send another in a similar vain.
With a set of mail servers woth its salt (and a group of sysadmins worth their salt) this would be easy to sort out. I bet they are stuck with Exchange and MCSEs. They ask for it, they get it. And when those despicable Microsoft sales reps come in next time and say "mail is dead, everyone's using Tah Cloud (mumble mumble 365) these days", upper corporate goons wil listen. Seen that at $my_company, which I'll not name, for obvious reasons.
That said, having enough BOFH sysadmins to swarm out and smack each user having clicked "reply to all" would be some sweet revenge...
Obligatory Dilbert:
http://dilbert.com/strip/2003-04-06
I've worked at a couple of companies that have sent out mass emails like this. My guess is that 1% of employees are stupid enough to hit "reply all" when requesting being removed from the email thread.
I seriously doubt that only 120 people in the NHS have hit reply all. My guess is that there will eventually be a few thousand who do this. That's assuming the NHS has above average intelligence employees.
Why is this even a /. story? I'm not sure how this is news for nerds(we all know that lusers are stupid) or how this matters?(I don't have a /. story for every stupid Corp IT crisis)
This place has really gone downhill.
I don't work for a company with over a million employees but I see this happen frequently. People reply all out of rote habit, not even consciously. It's so annoying. I know one company who customized their distribution of Outlook to not have a reply all button. Short of that, my recommendation is to either protect the large distribution group so that only a select few can email it and/or making use of BCC for that group when original emails are sent so that even if people do reply to all there is no further waste of time. I tried individually shaming people when they did it for a while, sending them dumbass instructions on the location of the Reply button versus Reply to All, but that had a limited impact other than for more people to know I'm an asshole. Oh well...
The simple solution is to lock down the "all" distribution list so only certain (very few) senders can send mail to it. I am not sure why this would not have been done... I am actually surprised that this hasn't happened before if everyone has send access to this list....
If course, if the distribution list was enumerated on the client prior to sending, that is a different story. But I can't imagine any client that would work reasonably with a million individual recipients.
My eyes reflect the stars and a smile lights up my face.
I have seen it few times in big corporations I worked in. Somebody sends email to wrong group by accident and then we have 3 waves of attack:
1) Clueless people hitting 'reply all' asking for removal from mailing group
2) Even more clueless people hitting 'reply all' asking people to not 'reply all'
3) "Champions" trying to save a day by putting all in BCC and telling people to not reply all, unless you put it in BCC [1]
And then, few hours later, next timezone wakes up and things start again.
Why is it useful? After it is obvious what is happening, you create folder called 'idiots' and redirect all these emails into that group by outlook/whatever rule. After that, if you need to deal with somebody in your organization, first check if he/she is in idiots folder and approach accordingly.
BTW, 120 replies seems very low. I have seen mailstorms with group of 10k recipients (it was not 'all' group, just some subset of company) generate over 600 replies total in these 3 waves. 120 replies from 1.2 million looks to be technical limitation (or, maybe, there was some hero in IT department who pulled the plug fast enough...)
[1] - My favorite is self correcting champion, which first sends 'reply all' and then does reply to that with everybody in BCC saying he should have put everybody in BCC in first place...
Comment removed based on user account deletion
A bean counter at a local hospital sent out an email to everyone that explained the cost in lost productivity for each "reply all" email was $0.08 per person. Not surprisingly, someone hit the "reply all" button to respond that the bean counter's email cost the hospital $800 in lost productivity. It went downhill from there. Not sure if the tab was $80,000 or $800,000 in lost productivity when everyone stopped hitting "reply all" button. Executive management wasn't amused.
And this highlights one of the many uses of BCC
i worked at NG for 7 years, and there were several instances (the last one being in 2010) where email system company-wide was crippled or knocked offline by an email that was sent to the wrong mailing list. apparently, there was one available that included literally every single person in the system (probably about 100,000 people).
i remember one morning in about 2008 or so, getting an email addressed to some team i wasn't a part of, seeing the "CC" list was several miles long, and i knew instantly what was going to happen. i guess the "first post" instinct in me acted up for the first time ever. i knew we were all already doomed, so i hit reply-all and simply posted: "oh no, not again." i did manage to be first, but before i could blink, i had over a thousand new emails all saying some variation of "WHAT IS THIS?" "REMOVE ME FROM THIS LIST" and "STOP REPLYING FOR GODS SAKE". my new emails hit 30,000 in a few minutes.
the entire NG email system was down for more than a day.
two days later i got called into my boss's office and he explained that top-tier management at NG had demanded that i be fired. my "oh no not again" was the last email most people saw before the system exploded. a very heated conversation between my supervisors and NG executives apparently just barely saved my job, but my supervisors were not pleased either and mentioned this would go on my permanent record (i thought that was just a high school thing). it didn't matter that i didn't actually do anything to cause the crash. i had merely made myself visible at the wrong time, and NG wanted someone's head.
so glad i don't work there now.
i could live a little longer in this prison
A typo in a e-mail address which autocorrected to the group e-mail address which included every single employee of the company all over the world; 10s of thousands of people. What's worse is to the average person it simply appeared the e-mail was sent to them alone by mistake rather than including everyone so people simply replied stating it was sent the wrong person.
I don't remember the exact number of replies that we all got but it was easily a good few hundred a day for 2-3 days, all while making the mail server crawl as it was one server for the whole company.
In a horribly boring desk job it was the highlight of our year.
And only a small division with a thousand or so employees.
There had to be threats of discipline from upper management to get it to stop.
I'm a good cook. I'm a fantastic eater. - Steven Brust
Alexa, release that Super Bowl commercial featuring Missy Elliott.
Probably because the CTO knew that it is his head that should roll since the email system that allowed such fuckery happened on his watch.
Got to find a low level scapegoat.
I'm a good cook. I'm a fantastic eater. - Steven Brust
I remember this in the late late 90s and at a shit Fortune 100 bank in the early 2000s. The stupidity was especially nauseating.
I am shocked it still happens in 2016. This is MS Exchange, I presume? It's completely unacceptable. It's been a solved problem for over a decade.
Another data point that shows the government attracts some of the worst employees.
I get an email for a division-wide thing.
I get a copy from my VP's admin, specifically targeting my team in case we weren't on the original distribution list.
I get a copy of this from my manager, since he doesn't want me to miss this.
I also get a copy of the original from a corporate level special interest group I'm part of.
Then I get a copy from a former team member. Just in case I was left of their distribution, since they left our team but believe they may be getting team emails that current team members are not.
And a copy from their manager, with a note to be sure the distribution list I cannot administer is properly updated to get these mails from the list they should not be and indeed are not part of. Just in case.
Then I get a copy from an interested team leader who wants to make sure we are in the loop.
And another from their #1 team member, who looks out for us.
And finally my cubicle mate leans over and tell me 'hey, did you get the email from......'
And so I have a 14GB .OST that i cannot backup locally due to GPO. and i get warnings occasionally that my file will be groomed back to an unspecified maximum size. Some day, real soon now. Right after they encrypt my files for no apparent reason, in accordance with some policy I cannot get a copy of.
I'm not bitter, really. I feel for the corporate security and cloud services guys. They can't fix stupid.
deleting the extra space after periods so i can stay relevant, yeah.
Back in I think 1996, when I was in the Air Force, a contractor sent to everybody on base the dancing baby animation. I think it was several megabytes in size. Nobody even had to reply to it for the mail system to crash.
They would have saved money (read: time) at this point if they just distributed memoranda on paper.
I'm curious what email client they use at the DHS. I feel like Gmail or Outlook would totally explode if you tried cc-ing 1.2M people.
Fast Federal Court and I.T.C. updates
this is why i simply ignore these emails
a lot simpler than getting outraged and emailing everyone to be taken off
yeah, when i was at Vicenza, Italy in the US Army around this time we did the same thing. only we did it on purpose to try to crash the email system
Dear sender,
I'm out of the office from 13 November until the 26th December,
Feel free to contact my colleagues for further information
Sincerely,
Paul
from the administration dept.
(ps, if it's about the form 426a, you should have filed the administration forms 293b and 732a , we apologize for the inconvienance )
There is no earthly reason why everyone should be allowed to send emails with 1.2 million recipients. Any email with more than some reasonable number (say maybe 100ish? Certainly not more that 1000 even with that big if an org) should not be allowed to be sent except from a very small list of authorized users.
Unless of course management caught a bad case of the dumbass and forced the admins to leave that setting on. In which case the managers should be fired.
I blame the email provider (Microsoft, I assume in this case.) The "Reply all" button should have never been included without some sort of throttle/warning mechanism. Something as simple as a pop-up saying "You are about to Reply all to x people. Are you sure you want to do this?" with the message and threshold number configurable by an administrator. Really hard core admins could set the message to something like "Management has determined that all useless emails will be counted, and the offender terminated when the count reaches xxx. Have a nice day."
This is the second-most retarded thing going on in the world today.
Not sure, but it's definitely what happens when you get on a Democratic Party mailing list.
Fucking annoying. Had to change my email address because of those whiny shits.
If /. was a British site I would assume the UK Govt health service, but its American, and the first thing that came up when I googled was National Honor Society
Theres also quite a few schools using that TLA and the National Highway System
Queue Alt-Right diatribe against socialized medicine in 3...2...1
Having used the NHS in the UK, and the US medical system, I can unequivocally say the NHS so, so much better. Better healthcare, lower wait times, and way better costs: about 1/3 of what it costs in the US per capita for healthcare, and it wasn't billed directly to me. My NHS contribution from my paycheck is far less than copays and insurance premiums I paid in the US to boot. But those facts will no doubt not prevent the Republicans screaming and insisting on the opposite, and dismissing facts they don't like as "anectdotal" (and while my personal experience, and those of thousands of others, may be "anectdotal," the reduced cost and better healthcare, as measured by any number of metrics, are not).
Use something like this or similar in your MTA.
define(`confMAX_RCPTS_PER_MESSAGE', x)
Done.
get the TPS report done correctly?
a few years ago. It was great fun.
"Having used the NHS in the UK, and the US medical system, I can unequivocally say ..."
Who cares, anecdote dude.
Fucking annoying. Had to change my email address because of those whiny shits.
You could have unsubscribed. No need to play the victim game.
If you allow your large distribution lists to be used by all, you are handing your own users a loaded weapon with a round chambered. You'd think professional email sysadmins would know this, sigh
Comment removed based on user account deletion
There is no way this is a reply-all problem, because I refuse to believe the From field had 1 million addresses. People just hit "Reply", and it went to the list. So, this problem could have been avoided if the idiots had used a "Reply-to" field with any emails sent through that list.
it didn't matter that i didn't actually do anything to cause the crash.
Actually, you did so something to cause the crash: you hit the reply all button on a message that had a CC list several miles long. Granted, you weren't the only one to do so, but you were definitely contributing to the crash.
Comment removed based on user account deletion
Nothing is simple when you have a million users. Sure, it's quite easy to do a quick search and delete on a small number of mailboxes but for a million? It's a shit ton of servers and mailboxes to search through. At least it works better than it used to... Still, i'm surprised that they didn't restrict who could send to the distribution group (I"m pretty sure they are using MS Exchange.) There are lots of examples for scripts to do the search & delete but it probably has to be broken up into lots of smaller batches to avoid killing servers or hitting memory limits. Get-Mailbox -OrganizationalUnit "domain.local/Users" -resultsize unlimited | Search-Mailbox -SearchQuery 'Subject:"PDF version of memo" ' , 'From:"email@domain.com” ',”Sent:03/02/2015" –DeleteContent -TargetMailbox “target@domain.com” -TargetFolder “Delete” -loglevel full
https://leanpub.com/email-antipatterns
Never reply to all unless you mean it.
1.
Dear all,
due to the current economic situation we must reduce everyone's salaries and will begin enacting our early retirement program.
Sincerely, Your CEO.
2.
Dear finance department members,
you will get your bonuses as planned, and do not worry: the retirement program that has been just announced will not affect our department.
Sincerely, Your CFO.
3.
Dear IT department members,
we have decided to outsource the finance department, and do not worry: you are getting their bonuses.
Sincerely, Your CTO.
P.S. Oops, did I just send that to all?
Comment removed based on user account deletion
If you've successfully unsubscribed, you need to share your method with the world.
I donated a couple bucks in 2008, an I've lost count of the number of times I've "unsubscribed" and received a confirmation email of unsubscribing.
I still get emails.
It was a joke on The Daily Show in 2012, and it's probably the real reason Trump won the election: Millions of people who are pissed off at the Democrats for not unsubscribing them from their spam lists.
If you've successfully unsubscribed, you need to share your method with the world.
Scroll down to the bottom of the email, click on the unsubscribe link, and follow the directions on the webpage. That simple.
I was there for that. I thought it was you MJ but since you still work there, you must have been someone else. There was one back in the earlier 2000's which ended with one my wise ass co-workers posting a very large joke about engineers in a bar. I think by the time everyone started reading it, it gave them enough time to rethink about a reply and no further mails were sent out.
is there any way that this can be blamed on Atos?
We are truly in the darkest timeline.
of the receivers used reply all. That's not bad.
Back in the 90s I was in college and the CS department implemented some mailing-lists per course and one for the whole department (this was on a Unix server). Someone was smart enough to subscribe the mailing-list to itself (and they didn't saw it coming). He successfully mail bombed all the accounts in the department and crashed the server :). Evidently the CS department was not amused, proclaiming they would certainly find the prankster but back in those days you could just probably have done this by telnet-ting the smtp server so nope ...
:-)
Another classic in big companies (and has happened several times at one I worked for) is people trying to send an e-mail to a department in the company like HR, finding a list named something like "HR_Dallas" in the global address listing not knowing this is not the HR Dallas department but rather the list that sends an e-mail to all Dallas employees. Yep, you just send your private confidential mail to all the employees
I would have replied with a 15MB embedded picture.
Testing, testing, testing
https://youtu.be/_Cmv58lTWgU
I worked at a place where someone went on maternity leave and had an Out of Office set up that somehow ended up causing an infinite loop as soon as someone sent her an email. I forget how it happened, but it was fun watching hundreds and thousands of emails appearing in everyone's inbox for a few minutes until they pulled the plug on it.
Our study abroad administrator didn't understand how email worked, didn't know how email *lists* worked, and didn't know you could suppress the email field via BCC.
She hand-typed the email address of every single student into a standard CC email field at a time when we only had something like 300KB of space for our *entire* email. The header alone was larger than that, given that we had over 2000 students. And *that* was before the "Reply-Alls" started rolling in. You could still send mail with your email storage full, it just wouldn't save the outgoing message, so the entire server filled up in minutes. Response time went through the floor. It took IT all afternoon to sort the whole thing out.
And then, two days later, she did it again.
No single raindrop believes it is to blame for the flood. poster
Ask someone that sits near you about email rules and spam filters. Someone else already mentioned UNSUBSCRIBE, so ask about that too. If the person is too busy with Excel, ask someone more computer oriented.
The underlying premise is the Government knows how to orchestrate all this health activity better than ordinary people.
You see the government can spot the wastefulness of giving cataract medication to someone before they go blind in one eye. (A true NHS policy, btw).
And the government understands how politicians should get large lump sums of money for the more people get put on the kill lists. (Part of the VA/Walter Reed thing).
And if everyone was enlightened as the Government, they would email bomb each other in waves of sophisticated confusion everyday, just like the NHS here.
But ordinary people are not as enlightened as the Government.
its email systems are running "very slow today."
Actually I think their email systems were running faster than they ever had before. Well to be annoying, one would hope they run full-speed at all times, so faster and slower are the improper terms to use.
it would be 1 message, per reply, and a bunch of extra pointers to it. This it wouldn't be "140 million needless emails"
Also, it's a piece of cake to go in and script removal of the message(s). It's pretty easy to clear them from regular pop/imap too.
It's obviously impossible to specify 1M+ recipients on a To: or Cc: header, so there must be an alias or list expansion that goes to all. Simply disable that alias until things calm down. It's not rocket science.
I used to get in a conversation with someone on them. I knew what I was doing, but I was having fun holding a convo.
"Lets play a quiet game. Everyone has to not say anything."
"okay!"
"No, you lose."
"Why?"
"Because you said something."
One smartass is bad enough, but if you get two, its all over.
Sometime in the 2000's an email made the rounds with a large power point slide show of pictures depicting a foam fire suppressant system test that went better than expected. I want to say that the attachment was 35 megs but that seems a bit over the top. Anyways the unit from whence the photos originated was the subject of a lot of mockery among airmen everywhere though there wasn't any re-alls going on with it that I saw. Within a day or two though an engineer from that group sent out an email seemingly across the entire Air Force insisting the test went perfectly as planned and that everyone should stop laughing about it and wasting Air Force resources forwarding the email.
The NHS and the WHO collaborate, but they are different organizations. Both have doctors, Dr NHS and Dr WHO is what they are called.
__
Men with no respect for life must never be allowed to control the ultimate instruments of death.
GW Bu
I perticularly liked the help desk emails going out to the same distribution list saying 'Your incident has been raised with the Help Desk. Your incident number is 5235" ..
The alt-right is a national socialist movement. They are explicitly for things like the NHS. They argue that it's much easier to have things like the NHS when you have an ethnically homogeneous state.
I can imagine a globalist libertarian like MILO would be against the NHS. That's one reason why he's not alt-right... the other reasons are because he's a kike, a faggot, and a ni99er lover.
Anyone else want to check my math, but assuming the average email address is 20 characters long (that's on the low side from what I can find), the list of recipients on each email should be about 24 Megabytes? That's a lot of bandwidth even without these ridiculous quantities. Or is it just a mailing list name on the "TO" line and the mail-server looks up all 1.2 million addresses from its own table?
Comment removed based on user account deletion
The 4 star was on the distro and replied all saying that any further replies would result in the person and their supervisor having to explain themselves to him tge next week. Was dead quiet after that & gave the mail server admins time to fix the underlying issue.
Long ago, in a galaxy far, far away, my secretary did a broadcast email of 100MB in photos to my entire company. Our home-brew mail server practically exploded.
my company has the list address show up as the sender, so hitting normal reply will send your reply to the whole list. Same problem, and all the out of office messages also go back to the list, again and again. Nothing better than 10k emails waiting in every inbox.
people argue that they didn't hit reply-all, so it isn't their fault. Why this still happens cracks me up. It has been a known problem for over a decade. IT administrators think they're worth 100k + / year.
oh please, if i hadn't replied, the other 100,000 replies would have still happened. i knew they were already on their way. it's not like i could have stopped it.
i could live a little longer in this prison
We had something similar happen at a previous employer. They took exchange down for 2 or 3 minutes while they did some sort of cascade delete and all traces of the e-mail were gone from the entire company.
People are stupid... This happens at many employers. People still reply to all, it's really annoying. You simply cannot rely on people to not reply all, be smart. When sending out email to a large number of people you absolutely MUST USE the BCC line instead of TO. That way they can't reply all to everyone on distribution. Just leave that TO line blank...
Internal Denial of Service? Hmmm. 2 Days you say? A pair of out of office replies ....
I know brexit was supposed to brink UK back to stone age, but can't the NHS IT staff just filter an address server-side?
4) after it has all been resolved, and things are back to normal, those people who were on vacation return to work and start it all over again by doing a 'reply all' asking to be removed from the list.
My beliefs do not require that you agree with them.