Slashdot Mirror
State Dept E-mail Crash After "Reply-All" Storm
twistah writes "It seems that a recent 'reply-all storm' at the State Department caused the entire e-mail infrastructure to crash. A notice sent to all State Department employees warned of disciplinary actions which will be taken if users 'reply-all' to lists with a large amount of users. Apparently, the problem was compounded by not only angry replies asking to be taken off the errant list, but by the e-mail recall function, which generated further e-mail traffic. One has to wonder if capacity planning was performed correctly — should an e-mail system be able to handle this type of traffic, or is it an unreasonable task for even the best system?"
*Can* one adequately capacity plan for that hunk of crap?
Technology -- No Place For Wimps! Grateful Dead and Jerry Garcia Chatroom -- http://www.wemissjerry.org
http://msexchangeteam.com/archive/2004/04/08/109626.aspx
Again...
-Ghostis
Computer Science is all about trying to find the right wrench to bang in the right screw. -T.Cumbo?
Like rain on your wedding day.
What an irony that they decided to mass mail when they've warned their employees not to do so. What they should have done if they were concerned about their load [which evidently they should have] was to warn their employees in blocks, perhaps 10% at a time with space between to take care of the massive response... However, judging by the nature of their work [it is the state department after all] I don't believe it unreasonable that there could be events in their future requiring such mass mailings again and having the whole system crash under the load would be no doubt very bad in emergencies.
Sigs are too short to say anything truly profound so read the above post instead.
Why do they have people sending to a list that anyone can reply to in the To: or Cc: lines. It should work one of three ways:
1) it limits the number of receipients (after list expansion) in the To: or Cc: lines, so those mailing a large list must put it in Bcc:
2) it should only allow certain people to send to large lists (implemented as a whitelist)
3) it should massage things on the server so that a list called 'all-company-list' would show up in the To: or Cc: lines of receipients as 'all-company-list-reply' and the list admin and sender are the only ones who see the replies to all
Honestly, mailing lists are not new technology and this has been a solved problem for years. Because they are incompetent mail admins they are forced to threaten employees!
Being able to send email to everyone is stupid. Imagine being able to hack into *any* account on a system and then automate emails to everyone to create a Denial of Service attack.Emails should be from the big boss, then disseminated to the little bosses and onwards down to the lowly employee.
I worked at a college using the groupwise e-mail system and the same thing happened. Someone sent out an information email to all students and instead of BCCing the entire list of addresses, they were all plopped into the "To" field. It bounced around forever and everyone was completely confused.
Luckily it wasn't my department and I didn't have a student email account, so I was immune.
Long story short, the system did survive unscathed....
Of the reply all button. Please do not respond with the reply all button. What they need is a reply some button.
How unfortunate for the outgoing administration that the best fix for the Bedlam will be a complete server wipe.
I am becoming gerund, destroyer of verbs.
Whoever wrote the headline for this summary needs to have their slashdot editor privileges revoked.
TFA states "an e-mail storm nearly knocked out one of the State Department's main electronic communications systems", and "a major interruption in departmental e-mail". The problem is clearly spelled out as "e-mail queues, especially between posts, back up while processing the extra volume of e-mails".
This is simply the queues backing up, not the servers crashing. Nowhere does TFA state anything to suggest that there was a "State Dept E-mail Crash", which the summary's headline boasts. The proper headline should read "Large E-mail Queues at State Dept After Reply-All Storm".
No, I'm not new here. That's why I'm fed up with the sensationalist "journalism" that is getting worse and worse here.
I remember my first year of college when I wanted to send Xmas greetings to 'everyone'. I remember, the IT director of the college running from computer lab to computer lab looking for student number xxyz.
Fun times.
If, like approximately 1/2 of the American population, you currently had no health care at all, your attitude would probably be different.
And you might want to remember that the current financial and industrial collapse was given to us by the finest and most highly educated examples of stupid, greedy, incompetent, short sighted, overpaid, negligent, and possibly criminal management that private enterprise has been able to produce and promote.
a) Maintaining large list by copying all recipients into the hrader is a fucked up idea at best (because there is no way this list will be kept updated), and a informaiton leak at worst (because somebody eralier on a non-updated list may get information which he should not get - e.g. former employees). Why do governmental institutions still us it?
b) Why in the world do modern e-mail clients still allow reply all to hundreds of recipients without an additional safety question. I would expect my program would warn me before sending an emails to thousand people.
I was at a company that used 'exchange' for their mail (sigh). most users did windows but I run freebsd and used imap and local ascii client for my mail.
one day a marketing person sent out mail and sent the wrong thing. they then sent some kind of 'recall' message.
the thing is, my ELM user agent didn't listen and neither did my IMAP puller ;)
recalling an email. yeah, right. pretty laughable.
--
"It is now safe to switch off your computer."
Dear state department
I'm sorry to hear about your recent trouble
There is a brand new invention on the internet which have the ability to ease the strain on your mailservers. it is called maillist managers. one is called mailman and can be found at: http://www.gnu.org/software/mailman
There are several others, some free, and some non free, but they exist for most server platforms. If you don't have the expertice in house to set it up corrctly, you can get any number of consultancy companies to help you out.
Yours faithfull
Almost anonymous coward
The article mentions e-mail queues "becoming backed up" from the extra volume imposed the reply-all messages. I would think the mail administrators could have simply dumped these messages from the queue to get mail flowing again, then disabled the list temporarily to prevent further damage.
OpenNet, by a very quick look on google, seem to be their network name for the non-classified bits and pieces. Supposedly Microsoft + Cisco stuff.
Feel free to disagree, but please provide a URL reference to the OpenNet email server software vendor if doing so.. ;-)
To all the mods, please don't destroy all my Karma. I really do hate that Reply All button.
Good to know that rigorous competition in the marketplace has totally eliminated misuse of 'reply-all' in the private sector. I look forward to continuing to have a lower life expectancy and higher infant mortality than Canadians and Swedes.
And you might want to remember that the current financial and industrial collapse was given to us by the finest and most highly educated examples of stupid, greedy, incompetent, short sighted, overpaid, negligent, and possibly criminal congress.
Fixed that for you.
Do you even lift?
These aren't the 'roids you're looking for.
It is interesting to me how the computer is used less and less to enforce policies to help users that might make an occasional mistake, or discourage users that might want to commit intentional fraud, and more more to make jobs simply enough so that incompetent people can be hired to do a job.
The fact is that if a email comes from a send only address, then the server with that address should simply ignore all emails to that address. If a user is not supposed to be able to reply all, then the enterprise client software should not allow that functionality for those messages. Policy enforcement is nothing new. It is why enterprises pays for software. If employees are using reply all and crashing the system, that is management issue.
In any case, this is mostly a case of untrained managers who thinks everything is a nail so always uses a hammer. It reminds me when managers used to use spreadsheets to write letters. There are many ways to distribute information, and email is only on of the tools. Managers could, for instance, use instant messaging.
"She's a scientist and a lesbian. She's not going to let it slide." Orphan Black
Yes, tricks like this would work!
(But I am wondering, how many of their Admins have heard of Bcc: ? :( )
Paul B.
No email system should ever "crash" under any reasonable load. Back in the late 90's, I was involved in designing and implementing email systems for some of the largest (at the times) ISP's as a consultant for a company that an NDA forbids me to mention. One of the things we did was limit the number of simultaneous connections, such that a "reply storm" (or, more often, a DOS attack) would hit a speed bump fairly quickly. Sendmail has done this for 25 years, by cutting off acceptance of new messages when load average goes above a certain (configurable) limit.
The point is that the very fact that a simple "reply all" storm could take down a mail system is, itself, an indication that the mail system is poorly implemented. Anyone taking bets on the system in question?
"He who would learn astronomy, and other recondite arts, let him go elsewhere. " -- John Calvin, commenting on Genesis 1
Anyone here part of the great ccpeople fiasco just over a decade ago?
C'mon, represent!
When I worked at a small nonprofit, mail was handled via unix mboxes. For reasons unknown, the system completely ground to a halt every time an mbox got bigger than about 100MB. To avoid this, emails older than 2 months were automatically archived. Well, one day the executive director managed to get 100MB of email in two months... It took us a few hours to track down the problem. The solution? set the archive script to 1 month and run it on his mailbox. Problem solved!
I might be stupid, but that's a risk we're going to have to take.
The really good thing though is that you are not bitter at all.
And in even further news, corporations are not perfect.
I take it you're not familiar with how enterprises plan. They plan for regular load, not aberrant once-in-a-blue-moon load. This is bog standard behavior for a system responding to people doing stupid things. If you think this is restricted to the US government, you've never worked in corporate IT.
Those who can, do. Those who can't, sue.
yes its exchange internally
openNet is what they brand it as
feel free to correct me with evidance that it was not the case any more but I know 2 exchange servers there and this say's otherwise
exchange has the recall ability and so does lotus notes
most other servers do not have this feature for very good reasons l
regards
John Jones
www.johnjones.me.uk my blog about email and digital communication
Doesn't sound like they know what they are doing. Some business have a 24/7 exchange support group.
Is it me or the confirmation image to summit message is deranged. Keeps telling me it failed that I'm not human. Technology logic and common sense epic fail.
Comment removed based on user account deletion
"should an e-mail system be able to handle this type of traffic...?"
Any system should be designed in such a way that a mere clueless user should not be able to bring it down accidentally. If an e-mail system can't handle "reply-to-all" when used carelessly, then it shouldn't have that function.
http://alternatives.rzero.com/
1. In its effort to ensure that your taxpayer dollars are conserved, the government rarely wastes money on esoteric concepts like "capacity planning". 2. In the effort to avoid Microsoft technology, the State Department apparently used an email system that allowed reply-all to massive distribution lists. Exchange Server allows use of such lists to be restricted. 3. The BCC field should have been the obvious and correct first line of defense. The fact that the BCC is normally suppressed by default is probably a factor. State would not have wasted money training personnel to use the BCC field (see #1, above). 4. Having done #3, the "reply-to" field should have been redirected to a bit bucket. Same as #3 and #1. 5. Threatening employees with adverse action for something you should have trained them to make impossible: priceless.
I've seen it so many times over the years. I wonder why it's so hard to add an administrative setting that limits Reply All to a certain number of users? Set at 100, it would only send the first 100, then ask the user if they wanted to send the next 100. Or 300 or 400 or whatever.
I can't count the number of people sending a hasty and blistering reply to thousands of people. Not only committing public suicide but accounting for who knows how many unproductive man hours while the entire organization stopped to read their spew. It's just crazy.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
A modern email system really should be able to handle this. High performance messaging systems will store one copy of the message, with n number of pointers to it per back end store. Sending a message to 10k users results in one store insert event and a 9,999 cheap pointer operations. The MTA will have to perform directory look ups for the recipients, but should use LMTP to insert them into the store and prevent redundant directory queries, etc... Sun's big mail server will even "relink" duplicate messages in the store that arise from user migrations, and free up disk space.
~300 - message per second insert rates with 50 kbyte average sizes is possible on modern workgroup class servers and disk arrays.
I just received an amusing outside-contact reply-all storm from a private company (the company's customer/vendor contact list was accessible from outside), but it only made 200 messages. So I'd say that the government can make much better reply-all storms than private enterprise, based on a sample size of one.
The determined Real Programmer can write Fortran programs in any language.
http://www.hanselman.com/blog/HowToEasilyDisableReplyToAllAndForwardInOutlook.aspx
2 simple lines that you can include in your Outlook client to prevent this action internally on your exchange server.
Note this does not include any macros in the email.
This just in, President-Elect Obama Assassinated! Oh, don't take it so literally. I was just trying to capture your attention in a short amount of time. Obama wasn't killed, silly. There was just some CHARACTER assassination against him on a late night talk show.
Seeing that we've established that this was OpenNet which uses public-available systems and in this case that means Exchange, wouldn't it be reasonable to assume that as we're approaching the end of the error.. er.. era of the Bush admin there would be an uptick in "Goodbye, here's where to reach me" mails to entire address books? From there, it'd take no time to hit the hard limits in Exchange for file storage... talk about ungraceful failures that we've known about for years. (Wait, that's another Bush reference!)
I use the reply-all button frequently, for ad-hoc small group discussions. If I have a document I want two people to review, I send it to both of them, and they send their comments back to both me and the other person I sent it to with reply-all, so we're all on the same page.
If the same group of people is frequently collaborating you can set up a mailing list, but it's a real pain in the ass to set up a mailing list every time you want a group of 3 or 4 people to exchange 5-10 emails.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
I'm surprised noone else has mentioned BCC. BCC works like To and CC (AFAIK the only difference between THOSE is to indicate who the message is intended for) except recipients don't see the BCC list. So they only see themselves as the recipient, thus no reply all.
At least, that's my understanding of how it works. IANAEmailExpert.
(I'm the one who added the bcc tag fyi)
We used to use bcc for mass emails, with a note at the bottom to inform users that they had been put in bcc to avoid the problems associated with reply to all.
Pretty simple solution, worked really well, zero additional hardware/software etc required.
Only certain departments (secretary of department heads typically) had the ability to create mass emails, so training was easy.
The problem is the message replied to having - RTFA - several thousand addresses in the To: and CC: fields. This is what BCC is for . Allowing people to put several thousand addresses in to the headers will eventually result in a mail storm, whether someone hits Reply To All or not. The first time someone opens a virus laden attachment that goes through their (archived by law, this being a federal agency) emails, it will send itself out to thousands of equally clueless people. One of them will run the attachment, which will send another copy to several thousand people. And so on. This happened where I work once, by people who should have known better. Before it was done, I was getting two hundreds copies of the virus per day.
Whoever sent out the message replied to should be fired and criminally prosecuted for deliberately sabotaging the State Department's email system. But since the article doesn't mention this at all, I'm assuming it was some dumbass boss somewhree who is immune to any form of disclipline for anything, up to and including murder.
What about when the Communists butchers invaded Afghanistan?
THL phish sticks
Sendmail can slow to a crawl but it doesn't generally crash. Run sendmail and you can handle this sort of nonsense.
If only we could do CAL properly...
In this day and age there is no excuse for this. I work for a company that has over 300,000 Exchange 2003 mailboxes. We ran into this problem back in the Exchnge 5.5 days, but squashed it post E2K by setting a relatively low threshold for the maximum number of recipients allowed in a single email (I think we have it at 50). For legitimate mass mailings we use an internal isolated app that routes its email to the Exchange org via SMTP. Only a very few people have access to use it and all recipients are BCC'd.
Reply All should be used sparingly. Nothing annoys us IT folks more than Reply All for NFR. "So and So received a promotion. Congrats." and the rampant Reply All'ing ensues.
It's called Netiquette. Perhaps along with normal training (please PLEASE tell me they train their new hires), proper Netiquette should encompass a normal training routine.
Isn't this dumb? Why would it cause a crash? Is their email not single instanced, like using MS exchange would get you?
You're suggesting that Congress somehow forced private enterprise to behave stupidly, greedily, incompetently, negligently and possibly criminally? That private enterprise was unable to resist and followed each other, like lemmings, over the cliff?
Keep in mind the fact that a recurring theme on /. (plus in most conservative, liberal, centrist, and libertarian publications) is that government in the US has been reduced to not much more than the paid-for pawn of special interests with big money.
A huge problem is that the thing which surprises us most about politicians is not that they're whores, but that they're cheap whores. Many seem perfectly willing to sell out their constituents, the country, and the constitution for relatively small campaign donations. I might be able to understand an expensive whore of a politician accepting payment of hundreds of thousands or millions of dollars in exchange for extending some favor. But the politician who takes a few thousand dollars in exchange for letting make millions or even billions of dollars be made at the ultimate expense of the rest of us is a cheap whore.
If, like approximately 1/2 of the American population, you currently had no health care at all, your attitude would probably be different.
A) Nobody has "no health care". There's always the emergency room -- in the US they can't turn you away for lack of ability to pay. This isn't ideal but it disproves your statement of "no health care at all"
B) There's ~47,000,000 Americans without health insurance. Out of a population of ~300,000,000. That's 15.67%, not 50%
Neither of those are ideal but if you are going to post on a subject at least get your facts straight.....
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
"The term Deltic (meaning in the form of the Greek letter Delta) is used to refer to both the Deltic E.130 opposed-piston high-speed diesel engine designed and produced by D Napier & Son, and the locomotives produced by English Electric using these engines, including their demonstrator locomotive named DELTIC and the production version for British Railways, who designated these as (TOPS) Class 55."
For a train your English is quite good.
Not a Twitter sockpuppet... but I wish I was.
This is the government we're talking about. That would be above their pay grade.
The solution is much simpler... Limit access to who can send e-mail to the larger distribution lists.
All things are subject to interpretation, whichever interpretation prevails at a given time is a function of power and n
And that's why we don't have the Army, Navy, Air Force, or Marines any more. We just have private contractors, because they are far more patriotic, and loyal. Plus they don't TK as often. We should have gotten rid of those government socialist military types BEFORE we had to spit on them for fucking up Vietnam.
Now THAT is a troll.
Fascism trolls keeping me up every night. When I starts a preachin', he HITS ME WITH HIS REICH!
Parent is a typical Linux/FOSS supporter :-)
http://www.dilbert.com/strips/comic/2003-04-06/
Have you ever worked for even one day in IT? Do you seriously expect people to just retype the entire CC list every time? Good luck with that. Why not just get rid of email altogether? Please, do yourself a favor and get some real world experience before having this kind of mental bowel movement on everybody at /.
It's a very dark ride.
A Cyrus mail-store can handle this with aplomb. Single instance store means only one copy of the message body need be store per backend, and all recipients are linked to that copy.
The people subject to disciplinary actions should be the idiots who stuff a large number of recipients into the To: and cc: fields, instead of using bcc: .
If the list is needed repeatedly, it should be set up as a mailing list.
When you e-mail from some subscribed mailing list, you don't see the name of everyone on that list in your headers.
The contents of To: is a name representing the mailing list. The mailing list uses lcc: to target the recipients, which is similar to bcc:.
It's very easy to reply-all by accident. Typically, the two cases are not very different in the user interface. There is a different button or whatever to reply versus reply all, but otherwise the flow of replying is the same.
The guy who wants to quit but doesn't because he'll only get unemployment benefits if he's fired :)
Um...which goes to show how little you know about unemployment. At least in MA, you don't get shit if it is "termination with cause", ie fired. If you're laid off, great- but even then, your employer gets a phone call from the unemployment department asking whether you were fired or laid off. Nothing stops them from lying and saying you were fired with cause- and then you've got a legal battle on your hands, which you can't afford.
Other fun facts about unemployment in MA: you don't get paid for two full weeks after you FILED- not after you were laid off, but after you FILED. You get a pittance compared to your normal salary; you'd be lucky to make rent on a studio apartment in Boston based off an entire month's unemployment checks.
Any income is deducted from your UA check. Say for example you find a 2-3 hour consulting thing on CL and make $150 helping someone fix their computer. Guess what? Your unemployment check for that week will be $150 smaller. This basically means that you have no incentive to find any kind of income while you're on UA.
Last but certainly not least: you have to pay taxes, medicare, medicaid, etc on your unemployment benefits. It's not bad enough that you're basically on welfare- you have to fork over a portion of the money the government is giving you, BACK to the government. Cute, eh?
Please help metamoderate.
You're suggesting that Congress somehow forced private enterprise to behave stupidly, greedily, incompetently, negligently and possibly criminally? That private enterprise was unable to resist and followed each other, like lemmings, over the cliff?
Yes, see Housing and Community Development Act of 1977 - Title VIII (Community Reinvestment Act) and Federal Housing Enterprises Financial Safety and Soundness Act of 1992 which forced Fannie Mae and Freddie Mac to do what they did to those poor poor people.
A huge problem is that the thing which surprises us most about politicians is not that they're whores, but that they're cheap whores. Many seem perfectly willing to sell out their constituents, the country, and the constitution for relatively small campaign donations. I might be able to understand an expensive whore of a politician accepting payment of hundreds of thousands or millions of dollars in exchange for extending some favor. But the politician who takes a few thousand dollars in exchange for letting make millions or even billions of dollars be made at the ultimate expense of the rest of us is a cheap whore.
See Rod Blagojevich for what happens when they get greedy.
... that would make sense. We can't be having any of that.
Cheers,
>I take it you're not familiar with how enterprises plan.
>They plan for regular load, not aberrant once-in-a-blue-moon load
OK, so I can understand a small business not planning for e-mail storms. But the US State Department is a likely SPAM and cyberattack target.
27,000 e-mail recipients, even if 1,000 people use the 'Reply-All' feature, shouldn't be a big deal.
In fact, that's why distribution lists are there - so you can get a clue and implement something that DOESN'T make 27,000 copies of that message data and associated 27,000 copies of network traffic (at least, until the recipients actually READ the message)
I am on a list of bidders for potential contracts with the Washington Metropolitan Transit Authority, which operates the Metrobus and Metrorail for Washington, DC and the nearby suburbs in Maryland and Virginia. The annual budget for the Authority is in excess of a billion dollars; it's larger than the budget of the entire State of Montana, for example.
One time I got a message with more than 25 recipients on it regarding a change in the way they were operating their procurement website. Well, I suspected that it was some spammer pretending to be from the Authority, because one of the "red flag" signs of being spammed is more than 10 recipients on the same messsage. But I discovered that it really was from the Transit Authority, it was simply an ordinary announcement with no url links and nothing but the announcement. But instead of simply either making the recipients BCC recipients, and sending it to an internal transit authority e-mail address as To:, or sending individual messages to each potential supplier, the contracting agent had simply sent it out To: listing all persons who were registered as bidders with the authority.
My e-mail address was one of these potential suppliers along with a few other people.
1,627 other people to be precise. This was the longest To: list on an e-mail message I have ever seen on a piece of e-mail that wasn't spam; 1,628 contacts. No, I didn't reply all, but I couldn't think of a way to refer to this incident as a "Send All" message and tie into this story. The other half of this incident was that the procurement agent had also just given all potential suppliers to the Authority, every other supplier's e-mail address, too.
The lessons of history teach us - if they teach us anything - that nobody learns the lessons that history teaches us.
Brain fart here or something, but I read:
... which really got me scratching my head trying to figure what this new metaphor might be about. :)
Cheers,
It's more if you count "underinsured" people, those who have insurance but it won't actually be enough to cover anything really serious.
There's ~47,000,000 Americans without health insurance. Out of a population of ~300,000,000. That's 15.67%, not 50%
I don't know anything about those source numbers, so I'll just go ahead and believe them, but I've gotta call you on those sig figs there. 15.67? 4 sig figs? How about just 20%.
(I'm not sure whether to thank or to blame all of my physics teachers for drilling us in sig figs)
coding is life
The University of Washington uses Mailman extensively, and apparently someone set up a list for an entire school (thousands of students) in such a way that replies to the list would be sent to all members.
A few announcements went out, and people started hitting "reply" (not reply-all) asking to be removed. Apparently the "filter administrivia" feature was also turned off, because within days I had hundreds of "I hate this - get me off this list and everyone stop replying-all" messages from irate users who couldn't figure out how to contact the list's owner.
To make matters worse, the list was about social justice and diversity, so the first dozen or so people who unsubscribed got flamed for being racist or insensitive. After about three dozen messages, everyone started to realize that there were some structural problems.
So, three mistakes:
1. The list was improperly set up so replies went to everyone (and moderation and the administrivia filter were turned off).
2. No one explained how Mailman works to either the list owners or the members, so no one knew what was going on for several days.
3. It wasn't an opt-in list - everyone in the college was subscribed without having been asked.
Note that this all happened without anyone hitting "reply-all."
Fortunately, I knew about Mailman's -control addresses and just sent a message to listname-unsubscribe@myuniversity.edu and got off the list right away.
This happened a couple years ago at the University of Minnesota. Somebody sent an email to tons of people using the TO field instead of the BCC. Probably fifty people hit 'Reply All' saying 'take me off the list' and 'I want off, too'. They let anybody in college these days.
Please unsubscribe me from this list I do not want to receieve any more mails.
and of course, off-topic from TFA, I signed up with a Product Testing Place. They email me once every six months and see if I want to test some new gadget or something and I get paid $75.
I signed a confidentiality agreement with them.
I am not allowed to discuss ANYTHING about the product or reveal I am testing it or anything. I was never there, I am nobody.
Last year I got an email - From The President of The Testing Company - personally thanking me for all the help in the last year.
He also thanked everyone else who "helped" last year as well and I could see who they were because apparently the President (or the secretary) just put all our emails into the TO: field and let it fly.
Lots of Identifiable people on the list because they used their WORK email, like john.doe@largecorporation.com So it was easy to see who else was part of that big Butt Plug testing program.
I did a REPLY to ONLY the President and laid into him about the confidentiality agreement and told him if he didn't know how to use email to stay away from the computer.
Later that day we all got another email from the President, this time apologizing for revealing all our personal emails, never happen again etc etc. And apparently he figured out how to use BCC!
So yelling at someone does seem to work to change behaviour.
Also- this is a dupe comment, I posted this once before on Slashdot someplace, but since this is Slashdot I didn't think a dupe would be a problem.
I like microcars
The primary issue here is the ability for one person to send emails out to large distribtion lists. Exchange (and I'm sure others) have the ability to prevent average users from sending company wide or otherwise very large numbers of emails out. Let only certain people at certain roles in the company to send out company wide emails. Reply-All to a distribution list which you have no access won't get you very far...
Ok, you can tell SendMail to limit the number of recipients of a message. But.... why? What if you actually, legitimately, need to contact 2,000 people?
I remember reading an email sent by some clueless Gubbmint official, with THOUSANDS of people on the "To:" line. So just because I was feeling puckish, I hit "reply all" and lambasted the sender for not using BCC and exposing everybody on the list to any viruses and spam coming from any compromised system from ANY of its users.
Now, my mail server is a mere Pentium 3, running at 500 Mhz with no cache. Make no mistake, it's so slow (by today's standards) that it's just stupid silly. Yet, this craptastic mail server handled all these replies, and all their responses (at least 20 of the affected victims also chose to 'reply all") creating a nice, hilarious email stormm that lasted for the better part of 2 weeks.
Worth it? OMFG YES!
I have no problem with your religion until you decide it's reason to deprive others of the truth.
everyone is saying to ban the reply all button but they arnt thinking
if it can crash the system it shouldn't be allowed to happen in the first place, so you have to ban it in the MTA, this is really easy to do
Posting to cancel moderation - please mod accordingly.
Emails are outdated. So much IT energy is spent administering it is a joke. Yet still at my work the email quotas are limiting, the backup/archiving a pain, and project/task messages have no business staying in a email database format. We need a smarter messaging system more like a blog/RSS or forum like this. It's not up to you to spam everyone if you want everyone to read a message. Right now our email system is as wasteful as snail mail - you seem to have no right to refuse junk mail. It just creates more garbage and wasted trees, and fighting it will take up all your time.
A) Nobody has "no health care". There's always the emergency room -- in the US they can't turn you away for lack of ability to pay....
They do. Ask any doctor on hospital staff about the "billfold biopsy", especially those working emergency services. People get turned away all the time.
I myself have been turned away from a major US hospital's emergency room, despite having insurance at the time. I was too sick/delirious with fever to dig out the insurance card from my wallet in response to their badgering about the name of the insurance company. By the time my ride had parked her car, I had been ejected for being uninsured. "If you don't have insurance, you can't be here." I had her drive me home. A few days later, I found the card still in my wallet, but by then I was busy juggling jobs and swapping shifts in return for those I had missed.
Anyway, the State Department's problem is only the tip of the iceberg. MS Exchange has been infecting US hospitals and that means downtime, lost messages and vastly decreased productivity. In healthcare, that means lost lives. In the State Department that means lost money and lost accountability. The latter is probably the main reason the outgoing administration chose to deploy it: no records == no trials.
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
However, here the problem is MS Exchange / MS Outlook combo. It just can't be set up right because the combo fails so spectacularly as a mail service. The dumbasses that chose MS Exchange should be fired and prosecuted for criminal negligence or mischief. The half-trained monkeys that carried out the deployment should be prosecuted for being accomplices and blacklisted from IT work.
nuf sed init.
The real problem there is that usually a mailing list does look like a single user, and then there's some list server which distributes it to everyone on his subscriber's list. There's not much a client can do to differentiate between an email sent to, say, "moraelin@company_name.com" versus "team42@company_name.com" versus "company_name_all@company_name.com". They all look like exactly one recipient to the client. (And at least the first two are very legitimate destinations too.)
Asking the client to know who it will be sent to, would require a way to ask a list server for the complete list of subscribers... and you can probably see how that would be a spammer gold mine. Not to mention a breach of privacy. (If I were on some bestiality mailing list, I wouldn't want everyone to be able to check if my email address is subscribed. And)
IMHO the right thing to do is to simply not have a mailing list with everyone in the company. It's a mail bomb waiting to happen.
There are extremely few things that absolutely need to be received by everyone, from CEO to janitor, across the globe. And you can go hierarchically if you ever actually have such an announcement to all. I.e., ask the department or team managers to forward that piece of information to their departments or teams. That's how the job pyramid was supposed to work in the first place.
A polar bear is a cartesian bear after a coordinate transform.
I admittedly dos not know exchange very well, but I do run mail servers that do extensive spam filtering,and a few things strike me as odd.
1. How come a large amount of extra mail crash and just not slow down a mail server?
2. In my experience, something in the neighbor hood of 5-10 % of mail is legit, relay doubling, tripling or even multiplying the amount of legit mail still counts for less mail that what I would expect during peek spam periods.....
www.aleo.no
at least with default installation procedures. each 'profile' directory, where stuff like that is held, is randomly named. you'd need to script something to read the profile.ini file and update each profile accordingly. Moving stuff to specifically named directories is impossible to do with (at least default installs of) mozilla stuff.
creation science book
Wait tell you see what Washington Politicians can do with your health care and your seized 401K retirement accounts. They did a great job with capacity planning with Social Security.
telnet 127.0.0.1 25 is the only way to send a "fuck you" message 8-)
I admittedly dos not know exchange very well, but I do run mail servers that do extensive spam filtering,and a few things strike me as odd. 1. How come a large amount of extra mail crash and just not slow down a mail server?
Thats a good question. Perhaps an exchange expert could answer.
2. In my experience, something in the neighbor hood of 5-10 % of mail is legit, relay doubling, tripling or even multiplying the amount of legit mail still counts for less mail that what I would expect during peek spam periods.....
Usually the spam filtering is done before the exchange servers by unix servers running the same software you use, or appliances. Therefore, the exchange servers only deal with the 10-15% of legitimate email.
--- Justin Dearing http://www.justaprogrammer.net/ We're just programmers.
They used exchange and got screwed, just like -everyone- who uses exchange.
This happens all the time just most companies cover up stuff like this because it's not good for the share price.
Am I the only one to think that it is quite peculiar that it is happening 9 days before the Government turns over? I mean, how much difficult could it be to say that some sensitive/embarrasing mails got lost during this crash? I think this should be looked into in more detail and make double sure that no mail was "lost".
I'm using a pretty recent download and install:
sdkfjlskfjl qwertyuiop Murcosoft bwbies.
Nope, no red lines, even though I've made sure that the spell-checker is set to to "check as I type" with a nice little tick in the appropriate box.
See, although technically the spell-checker is built-in, the dictionaries are addons, and the program doesn't warn you that although the spell-checker feature is enabled, it won't actually do anything if there isn't a suitable add-in downloaded and installed. It's not smart enough to "grey-out" the spell-checker enable/disable box when there are no dictionaries installed, to warn the user that there's an issue and that the spell-checker won't function until it's resolved.
I now download a dictionary, allow FF to reboot, and try again. Now it works. The downloaded UK FF dictionary still doesn't recognise "Firefox" as a valid word, though ...
Eric Baird
When sending a mail, before I send I read through the message to make sure it makes sense, and look for any spelling errors that the spell check didn't pick up (like a misspelling that is in fact a different valid word). I also double-check the recipients and make sure it's going to the right people. That's what we need to train people to do.
Sounds like a poor choice of mail systems together with poor controls. As an ISP, we handle thousands of mails per second. If I send a single mail to every customer in our system (thousands) the total time to have them delivered to the mail box (server) is less than 1.5 minutes. Rate controls should also be used to ensure no one mail can slam a system.: max messages per second max messages per user max connections per client etc.
Where I used to be a school sysadmin, the "everyone" group was accessible for everyone to use, since all employees could have occasional legitimate use for it--and I'd rather not spend all day fielding requests to e-mail "everyone". The only thing it blocked was e-mail from outside the forest to stop spamming--and to stop employees from e-mailing it from their personal e-mail accounts. The e-mail server was actually at the district office, so I just had group control via Active Directory.
So what happens one day? The drama teacher e-mails everyone a 15 MB TIF of the promo for some student drama production. Mailbox stores were limited to 40 MB, so I spent the next week going from one user to the next explaining archive folders.
Immediately after that, I limited messages sent to "everyone" to 750 K. This had the added benefit of keeping users from sending everyone stupid forwards or pictures of their children/pets. It did not stop, however, the latest stolen-UPS-uniforms e-mail scare from going around...and me getting sternly lectured for debunking such via Snopes.
Another problem I had was users trading multi-MB PDF files with each other via e-mail. So instead of a 10 MB PDF being in one shared folder, an entire department would have one in their personal home directory and one in their mailbox store. Did I have shared department folders set up? Yes. Did they use them? Of course not.
This is a partial dupe comment, someone posted this bit before on Slashdot someplace, but since this is Slashdot I didn't think a dupe would be a problem.
Wait.. So, a mass email was sent out warning people not to use the "Reply All" button. Then people used the "Reply All" button to respond, asking to be taken off the list.
/Happened on campus once. It was fun to watch.
Then there's the requisite conversation going on between list members trying to figure out 'wtf' is going on, and demanding people STOP emailing them, (using "Reply All", of course)
All of this along with the "Out of office replies" from a percentage of staff, for each one of these "Reply All"s.
How can we correct you without knowing what "evidance" is? Did you mean "evidence"?
Actually, I'll let you in on the Swedish secret.
Our infant mortality rate is so low because, well, we just don't have kids anymore.
The Swedish population grows a bit every year, but it's solely thanks to immigration.
As for our health care, the national insurance agency we all talk to when we get sick for more than two weeks is constantly claiming they have computer problems, so they never actually register people as sick (resulting, as you've probably guessed, in a very low number of people who are ill).
When a lot of users royally screw up like this, the problem is either poor training or the design of the tool they are using. This was not just one ignorant user. Based on the dozen or so email packages I've used, I'd hang this one on the designers.
No sane user wants to send a message to everyone on the network by mistake. A user interface that makes it easy is just badly designed. A good design by default:
I'm a Programmer. That's one level above Software Engineer and one level below Engineer.
Microsoft has described this in excruciating detail before, because at one point even they managed to crash their Exchange server - through mail list reply all spam.
http://msexchangeteam.com/archive/2004/04/08/109626.aspx
Sounds like the State Department might not have upgraded to Exchange 2003.
For a site about things like basic rights, Slashdot users sure do like to censor "dissent".
The very fact that there is a "recall" feature shows a lack of understanding of how email works.
What's the phrase? Trying to take something off the internet is like trying to remove piss from a pool.
One can argue that internally they should have complete control, but even then one is racing against time to delete the message *on the client* before the person reads it. What's the point?
It's better to vote for what you want and not get it than to vote for what you don't want and get it.
- E. Debs
When I was in college, I managed to send an 'hello everyone' email to the entire student body (>12,000 students) by using the same distribution address that the university used to send an official mail to everyone. I probably got a couple hundred responses or so--60% were friendly, 30% confused and the remaining 10% essentially 'f you'.
Needless to say, after that incident, the school abandoned the address in favor of one-off's that could not be re-used.
Remember that Exchange systems are mainly used for internal corporate mail, not just for mail to the Internet, so most of the users are under the control of the same system. Outlook clients typically get mail in one of two ways - reading it off a server, or downloading to the user's client and storing it there. If it's stored on the server, Recall can succeed; otherwise it's a bit of a race, but can often succeed, though they often fail or the recipient might read it before the recall gets there. My organization mostly uses them for correcting mail that was accidentally sent without an attachment or for meeting notices that had some detail wrong.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
I still have the message, there were 1255 recipients, me plus 1244 not me plus 1627.
The lessons of history teach us - if they teach us anything - that nobody learns the lessons that history teaches us.
I had a funny 'Reply All Story' some years ago. One of the managers in our company sent out an all staff to everyone for a staff member that was leaving, going along the lines of "thanks so and so for their hardwork and dedication to our business. You will be missed and all that".
Within minutes later of that going out, the managing director send back an email, obviously intended for the manager only. It went something along the lines "WTF is going on. we just put this guy on training and now he's leaving. this has got to stop. we have to start getting people to sign training contracts or such. any way, good riddance to him".
Needless to say, a call from the MD's PA arrived at IT shortly there after to either stop the email (too late) or shut down the entire email system for over 800 staff and go and remove the email from everyone's mail box...
some discussions later between the IT Director and MD on the impracticle nature of the work and how important email is (and that everyone has seen it now).. the MD had to eat humble pie and pretty much refused to use his email for about 2 weeks. We're still working on a project to "prevent this from ever happening again".
...is to have a listserv that uses the same address for Sender and Reply-To fields. Then some bright bulb goes on vacation and assigns his email to send an "I'm on vacation..." reply to all emails. You can imagine that creating an endless loop... The listserv issue got fixed in a hurry. The guy who administers it is pretty savvy, but he is busy making a living, so it hadn't hit him yet how bad that could be.
Every problem has a solution that is simple, easy and wrong. Selling our Liberty for a little Security is a much too de
MS Exchange == loss of mail; and in a hospital lost mail or even delayed mail means damaged or lost lives.
So yeah, I don't doubt that the losers who get paid to babysit Exchange aren't going to fess up to any wrong doing. Nor would the uber-losers who made the choice fess up and risk going to club Fed. In *every* case I've encountered, they all sing the same tune: oh, praise be to Bill, it is perfect.
Leave the basement and the tune changes. The servers are unavailable frequently and mail is so frequently lost or delayed that it becomes expected. The only use case I can see for MS Exchange is when the top management is up to no good and actually wants the plausible deniability that it will grant when auditors come around or courts subpoena records. Over the last 10 years, I've never seen a working instance of MS Exchange -- unless one redefines "working".
I've seen >10% data loss in (no chronological order) units/businesses/institutions with users numbering in
dozens
hundreds
hundreds
dozens
hundreds
hundreds
hundreds
dozens
thousands
thousands
thousands
hundreds
thousands
So if you say you think MS Exchange works, either you have a funny definition of works, or are paid to lie on behalf of Bill, or haven't ever spoken with the users.
When mail goes missing, only a fool or a shill is content to shrug and bleat the M$ talking point "only old people use e-mail" When you track down the cause, it is the defective design M$ is famous for. I don't use or condone the use of MS Exchange. The last field test I supervised, which which was in Decemeber, MS Exchange lost 28% of the mail in Exchange-to-Exchange sendings on the same server.
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
If you don't know which one this is, check out this link:
http://www.f-secure.com/hoaxes/msemtrk.shtml
The page doesn't document every variation, but it has quite a few. The variant I generally encounter says it's TRUE, was on Good Morning America, and has been verified by a lawyer zOMG! This particular hoax e-mail is amongst the older ones, has bitten me twice, and I fear it's going to repeat.
The first indecent was at the last company I worked for, which employed 100,000+ employees globally and 75% of that were in the US. Some moron forwarded it from his Hotmail account to his work e-mail. From there he opened the GAL, started at the top, and selected the max number of entries (256 IIRC) that would fit in the TO line, then proceeded to do the same for the CC line. A mail storm ensued as about 20 other idiots did a Reply All and replaced some of the entries with addresses of their friends. From there a few more goofballs did Reply Alls saying to stop or to take them off the list. This caused e-mail processing in a 20+ server Exchange cluster to come to a screaming halt forcing the Exchange Admin team, that thankfully I wasn't a part of, to shut it all down. The CEO or a VP sent out a mass mail telling everyone not to pull those shenanigans again.
At my current employer, one of the sales reps almost did the same thing exactly. We didn't have a mail storm thankfully since the company only employs about 120 people and there aren't a whole lot of DLs and mail-enabled PFs. I believe one grunt did a Reply All calling BS and a couple of others sending a WTF via Reply All. With the company being so small, we were able to stop the idiocy quickly.
I usually try to bottom post on emails, but I think something should be recognized, bottom posting is a new approach, and top posting is the old tried and true approach (at least generally).
Millions (??) of business writers have been trained to encapsulate the most important part of a business letter in the first paragraph. Most important things (and a summary) at the top, less important things (and details) later.
Similarly, tens of thousands (??) of journalists have been trained in a similar fashion--most important stuff in the first paragraph, less important stuff and details later in the article (and, in fact, ranked (based on somebody's judgment) of order of importance. Among other things, so that if an editor has to shorten an article, he just cuts stuff off the end. (Likewise, a reader can read just the beginning of an article to get the basic gist.)
Also, similarly, for a business reader, she can get the gist from the first paragraph.
So now email comes along and (nominally) espouses a different approach (yes, I know email can be structured with bottom posting to cover the most important points first, but I'm not sure how much thought is given to that)--what do you (the collective you) expect to happen?
With respect to email, I find my own habits changing and am starting to do more top posting. I have reordered quoted parts of an email I'm to which I'm responding in order to have a sequence more conducive to the point(s) I want to make.
Since it's not implemented in my client, the "recall" notifications tell me which messages I should look for the "good stuff" in.
Combine that with a filter that sends all "important" and "urgent" flagged messages to the trash, and it's perfect....
Tell them what you're going to tell them... Tell them what your telling them... and tell them what you told them. Right?
All the hallmarks of a shill.
I don't expect anyone who defends or supports MS Exchange to fess up to wrongdoing. Just like I don't expect it to work: Every last MS Exchange site I have investigated has been plagued by mail lost and down time. The only question is whether the users find that acceptable.
It'd be too easy then to force them to pay for the damage they cause if MS Exchange boosters admitted they knew they were f*king their employers to further the M$ agenda. As far as suckage of MS products go, brand recognition does cut both ways. If a company does great work for many years, they get a good reputation. If they make crap for years, they get a bad one. M$ appears to work very hard to cultivate the worst possible reputation. Don't like it? Then improve the technology, start with standards support. Or behave better in the market,start by unbundling the formats and protocols.
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.