Slashdot Mirror
Email Offline At the Home of Sendmail
BobJacobsen writes "The UC Berkeley email system has been either offline, or only providing limited access, for more than a week. How can the place where sendmail originated fall so far? The campus CIO gave an internal seminar (video, slides) where he discussed the incident, the response, and some of the history. Briefly, the growth of email clients was going to overwhelm the system eventually, but the crisis was advanced when a disk failure required a restart after some time offline. Not discussed is the long series of failures to identify and implement the replacement system (1, 2, 3, 4). Like the New York City Dept. of Education problem discussed yesterday, this is a failure of planning and management being discussed as a problem with (inflexible) technology. How can IT people solve things like this?"
It's the backend. When you have too many connections on too few servers, with not enough storage
you usually see this kinda issue.
It's an economic one. It needs an economic solution.
e.g.
Have people buy a $10 ticket to get an account on the email server.
Deleted
When I started college in 1991 I was amazed by the telnet access I had to the email account given to me by the University. I hadn't had an email address prior to that. Now I have an email addresses through hotmail, gmail and yahoo that I use for different things and facebook also gives me an email address. So, I doubt students really need email addresses provided by the university anymore. As for the NYC Dept of Ed example, I think it just shows that trying to build IT competence into a government agency basically a waste of money because the institutional culture of government. In short, all of these kinds of organizations could just offer email through gmail/google business or any number of other providers that will scale up almost infinitely.
if your life is such a big joke then why should I care?
Briefly, the growth of email clients was going to overwhelm the system eventually, but the crisis was advanced when a disk failure required a restart after some time offline.
Capacity planning is supposed to account for reduced capacity due to component failures, system outages, and temporary demand spikes due to restart events.
Wow, Squirrelmail. So at least they managed to migrate from pine at some point.
IT goes to management and says "based on current usage/loadings etc the system will fail in 6 months to prevent it we need to do this....." Management says "Really, that's not what the sales man told me and its his equipment so he should know".
Undetectable Steganography? Yep, there's an app fo
Now I have an email addresses through hotmail, gmail and yahoo that I use for different things and facebook also gives me an email address. So, I doubt students really need email addresses provided by the university anymore.
You are quite wrong. Email addresses - especially .edu addresses - are still quite valuable. At lot of academic resources that take registration via email won't allow registration to go to a throwaway account (a la hotmail, gmail, yahoo, etc). Many organizations that are interested in real information on users insist that users use an actual unique account and not a freebie. And when you're in college and making very little money a lot of those things can be important.
I think it just shows that trying to build IT competence into a government agency basically a waste of money because the institutional culture of government
You're not very accurate on that, either. Government organizations need to be able to keep track of their email - especially internal communications - which they would not be able to do if they outsourced email and other telecom.
In short, all of these kinds of organizations could just offer email through gmail/google business or any number of other providers that will scale up almost infinitely.
With the various privacy breeches that have occurred, that would be a terrible idea. And on top of that, IT is a lot more than just email. Do you want the government to turn to comcast for networking support while their at it? What if the IRS web servers go down on tax day? Do you want them to have to lean on an outside company to get it back up?
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
Maybe it has something to with the fact that the state of california has cannibalized the funding for my beloved alma mater.
They wouldn't have to if they didn't have too many colleges (they do), and try to send too many kids to college (they do), many of whom may have no business being in college (they don't). Tax revenue is not an infinite resource. But California seems to have a community college on every two dirt roads, and several 4 year (or higher) colleges in a similar area.
Life is hard, and the world is cruel
Since both my alma mater and my current institution have migrated to Google, and both are covered by FERPA and other privacy laws, I am inclined to say that that argument is bogus. However, I have a separate issue with outsourcing student email: third parties get to set the rules for student conduct without any action by the university itself.
Typically universities have acceptable computer policies and at those institutions that run their own mail servers, such policies usually govern email. Students and faculty can demand changes to university policy if the policy does not properly align with the academic mission of the institution. Students and faculty have essentially no power over the terms of use that Google or Microsoft or any other third party email service imposes on them. It is easy to say, "Well, it is not like Google is going to demand something outrageous!" but there is really nothing preventing Google from doing so (if you do not think they have done so already). Google does not have the best interests of academia in mind when it sets its policies, nor is there any reason for Google to care about academic needs.
Palm trees and 8
In the video, they don't even mention sendmail at all. Are they using it?
Also, they mention that the cost of the system is something like $1.30 per account per month. I don't know much about IT budgeting, but that seems like a really low number for something as critical as messaging and calendaring. I have to imagine that they spend more money per user just cutting the grass around the campus.
There aint no pancake so thin it doesn't have two sides.
I agree that the overall system is probably too large, but we are talking specifically about the flagship university of the UC system. arguably the best public university in the world, and it is getting hurt just as bad as UC Riverside. that's absurd and embarrassing.
Beware the Jubjub bird, and shun the frumious Bandersnatch.
Tons of schools use Google as their email provider. Here's a quote from a Time article from 2009:
Google now manages e-mail for more than 2,000 colleges and universities, enabling students to transform accounts capped at 100 mb into Google-managed inboxes that allow for 70 times as much mail. Microsoft also provides free Web-based mail for thousands of schools, including colleges in 86 countries.
Here's the article: http://www.time.com/time/business/article/0,8599,1915112,00.html. Now, a specific school? Sure, my daughter and I just toured California State Sonoma and they use Google services.
I've only heard from people on one side of this but the story that I hear is that in the past, many departments had their own IT, mail servers, web, etc. When the campus built its centralized computing services facility, there was great pressure on departments to move to the central system. There was some griping about the costs for central services often exceeding the internal costs the departments formerly had but there was, I'm told, much need to justify the expense of and to pay for the new center. I've heard that some departments have been able to resurrect their internal systems to get through the outage.
Perhaps someone with more inside knowledge than I have can fill in and/or correct information from both sides of the story.
That slideshow is pure management-spin right from the opening "look how complicated and difficult this is..." I love how the "solution" to a system that is soon to outstrip its capacity is to stop expanding (and, it appears, properly maintaining) said system and hope it doesn't implode before you can toss the potato to an external party (who can then take the blame). Guess I was never learned at that school of capacity "planning".
~~~~~~~
"You are not remembered for doing what is expected of you." - Atul Chitnis
The press pretty much reads like this to me:
1) We didn't size the system large enough to handle the possible outages.
2) The outage we didn't size for happened, basically taking everything down.
3) My team is now working on a band-aid solution, which basically involves hobbling the application.
4) Since we're incompetent, we're going to outsource this next year.
I mean, if I was the CIO's boss I would have fired him on the spot. Maybe outsourcing is a better answer than putting in place a proper system and looking at that analysis could be interesting. I see no indication any of that was done here, basically the CIO gave the Barbie response, "Mail is hard, let's go shopping." If he doesn't understand how to do it in house, he won't understand how to arrive at a good outsourcing agreement.
Which means this pretty much sums up everything that is wrong with large org IT today.
Outsourcing would work, because when there is another failure they will have another party to blame instead of pointing fingers to a decision made in Spring 2011 (even as a total stranger I could feel the bitterness under that bullet point in the slides).
lucm, indeed.
This is really quite common. It happened at my alma mater as well. Servers could not handle the POP requests, so they started blacklisting students that checked their mail more than four times an hour. A month later a RAID drive failed and email for 17k people (including a hospital) was completely offline for 3 days. It is sad that seemingly anyone can be a high paid "IT Professional" these days, but without a clue about HA.
HTFU
One can have all the clue in the world, yet be powerless to prevent failures if not funded to purchase the appropriate equipment.
Do not fold, spindle or mutilate.
I hate it when people try to act as if IT isn't subject to budget constraints and having to prioritize spending like any other department of a large organization. Sure the money comes out of the "client" departments, but it's an issue that IT does have to plan for and deal with.
The summary asks "How can IT people solve things like this?"
Forward the emails and responses to the demands for planned capacity growth to the public.
Oh, you didn't keep the email from your manager refusing to pay for a needed capacity upgrade? I guess you haven't been in IT long enough to learn to cover your own butt.
I do not fail; I succeed at finding out what does not work.
HA isn't an IT problem. It's a business issue. I've never seen a business who put a dollar amount on their downtime. I know they are out there, but every "real" place I've worked has never quantified their costs. How do you justify having a cold spare if it's a "waste" of money planning for an outage that is "free" (after all, if downtime were a problem, then someone would calculate the cost of downtime). IT did what they should, gave the users the best they could within their budget. That the budget was too small is partly the fault of the IT manager, but only partly.
Learn to love Alaska
History.
Computers originally came into companies to do accounting and related work.
The preferred solution is to not have a problem.
Believe it or not, maintaining a mail host for a larger, geographically diverse
If it were easy, there'd be no push to outsource it to "the Cloud" (or anywhere else), and countless organizations wouldn't be moving from the "burden" of administering something like Exchange (ie, a trivial amount of knowledge is required compared to any other MTA) to Office 365 or Google.
It's not just as simple as setting the mx to point to a 'working host', especially not in academia (though many try). Do you have to deal with this kind of thing?
As someone who has to deal with this stuff on a daily basis - I had dealings regarding CalMail last week on a similar mail related problem of their's - and with academic mail systems in general, let me clue you in:
* This is not your business mail system, where everyone has a uniformly specified mailbox.
* It is not dictated from the top down how mail is run. In a corporation, there is standardization. CalMail is the exception in academia, as far as I can tell, in that it's run somewhat like the business model. However, there is still somewhat of the "Greek" (vs. "Roman") model of management involved, and this does tend to lead to problems. (This is much more true with other academic mail systems, from what I can tell.)
* Unlike in the work place, there is very little systems experience where it is needed (ie in the actual administration). Even with dedicated IT, very few people are actually good with the mail system due to how broad and complicated mail management can be.
* Running a mail server effectively is now quite difficult. Not only do you have to "just make it work" - ie, dealing with all the misbehaving mail systems out there from other academic institutions and verifying the VIP email makes it through (regardless of how much spam that means letting through - but never let any spam through!) - but it's got to run like a top.
* Often, you're dealing with decades of systemic dependencies. Mail was the first connected application, after all, and nobody's had it as long as Berkeley. Based on my own experience with networks which grew around their mail system, small changes can compound any sort of change or update. Suddenly, there's something everywhere that needs a specific mail system functionality which can't simply be copied over during a move to replicate it.
* An organizational system like this is big, it's not garden variety email. Hell, i guarantee you they don't have as many IT people maintaining accounts as they have admissions people, probably not even a 10th. Yet the IT people have to actually make sure those records get to the right places all while assuring the admissions people that the information transits securely.
* There is undoubtedly a faculty member with his pet requirements for email. He probably has things which will not migrate properly.
* There will undoubtedly be the people using their mail account for file storage.
* Believe it or not, it's actually fairly difficult to migrate mail from, say, Cyrus IMAP to anything else. It takes time (and anything at all with Cyrus, which I'd not be surprised if they were using, takes a lot of time). Sieve scripts, procmail, IMAP states, et al. It's a pain in the ass, and takes a loooong time to do seamlessly. Doing it under duress of hardware failure is something else entirely.
From my reading of the events (and seeing some other things not mentioned in OP or linked article) there were a number of things which caused this prolonged outage. First and foremost, the system was not designed to be resilient so much as it was designed to scale up (or proper failure condition testing was not performed beforehand). Second, they either don't have the necessary (knowledgeable) human resources, or enough time allocated to those resources, to effectively manage this system. (You would not believe how difficult it is to find a "mail administrator". Everyone's done it, but nobody seems to like it or is all that good at it. If they are, they want a LOT in compensation.) Third, they may
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
As a mail administrator for a big system, I completely agree with you.
The biggest problem was that they had everything on a single SAN, so when they ran out of IOPs, there was no spare capacity anywhere, and nowhere to mitigate it to. I've had people try to sell me on putting all our systems on a SAN too "it's so simple to administrate. It has plenty of IOPs, see, look at these shiny numbers". Fine when it's empty and you're only hitting the battery backed cache.
Which is why we have hundreds of separate little disk sets managed with templated configurations rather than any single points of failure. I'm really glad to be there!