Apple, Android Devices Swamp NYC Schools' ActiveSync Server
longacre writes "Just a few months after the New York City Dept. of Education shelled out over $1 million on iPads for teachers, the agency has stopped accepting new users on its Microsoft Exchange ActiveSync server as it is 'operating near its resource limits' due to an influx of iOS and Android devices. A memo from the deputy CTO warned, 'Our Exchange system is currently operating near its resource limits and in order to prevent Exchange from exceeding these limits, we need to take action to prevent any more of these devices from being configured to receive email. As of Thursday, November 10th no additional users will be allowed to receive email via NYCDOE's Exchange ActiveSync.' Existing setups will continue to operate, and students will not be affected."
Imagine what they could have done with the $700k they would have saved by choosing a tablet other than an iPad.
Issues like this are the reason you need to fully flesh out costs before flipping the switch on a large organization like this. almost every teacher I know has a smartphone of some kind and a lot of them are starting to get tablets. Why offer the service when you cannot fully offer it?
Given the resources, is there any reason they couldn't scale this right? I only pretend to know anything about Exchange, but this seems kind of strange.
I'm sure that resource limitations -- server CPU, disk, etc -- are the source of this, but you'd think a high profile customer like this would be able to get MS involved before the story becomes "iPads crash Exchange" or "consumer tablet bests high dollar PC server."
This is what you get with golf course deals people out side of IT makes deals like this and tell IT to make it work with out giving them the funds to make it work.
This why IT needs unions so they can stand up and say NO! we can't do it with the funds that we have. I hope that they don't place the blame on IT for something that is not there fault.
I see budget priority may be a little skewed. dun-dun-dun. weighing options... ipads? Server? Ipads? Server? Ipads? Server? screw it lets splurge on some ipads!
Thus solving the problem forever.
This is a pretty standard situation in New York City: lots and lots of money is spent, with poor planning, sweetheart deals with incompetent firms, and then a bunch of fallout.
Palm trees and 8
$1m spent on iPads only comes to ~2,000 iPads at most (assuming the cheapest model at around $500 each). According to Microsoft's handy little Capacity Planner (Exch 2010), it shouldn't take but perhaps (very rough calc here) 5 or 10 servers at most to handle that, unless they're also allowing every school employee to latch on their personal gear as well.
I'm guessing that something's missing from the story here...
Quo usque tandem abutere, Nimbus, patientia nostra?
Lack of resource forecasting/planning will get you every time. Its not like they didn't know how many would be deployed and on what schedule.. geesh
---- Booth was a patriot ----
Just that their IT staff is incompetent..
---- Booth was a patriot ----
BAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHAHA!
dumbasses.
"Bill, what do you think we should do? Should we buy $1 million worth of iPads or update our mail server and then use the remaining for iPads?"
"Ummmmmmm...F*#! THE MAIL SERVER! I WANT A MOTHERF#!?IN' IPAD AND SO DO YOU!"
"Yeah, but we could just make sure we are the first in the group that recieves them?"
"DAMNIT MAN I SAID BUY THOSE F#$*IN' IPADS!"
Right?
Vote monkeys into Congress. They are cheaper and more trustworthy.
If I were a teacher in NYC, I would just go rogue and use my gmail address. It is very accessible from my new iPad and I can avoid Exchange hell from my school district's IT staff.
I do not think that privacy is the chief concern when it comes to personal devices on school networks. More likely there is a support contract getting in the way; my high school (in NYC) had a bunch of desktops that could not be connected to the school's network because of a support contract stipulation. Internal emails are probably easy to forward or otherwise export from the schools' computers, and the security is probably very poor (when I was in school, the only think separating the teachers' network from the students' was the IP address assigned to the computer -- and anyone could manually set the IP address, which is how we defeated the censorship firewall).
Palm trees and 8
WE CAN'T HAVE OFFICIAL SCHOOL EMAIL ON OUR IPADS!!!!!! ANARCHY!!!!
fuck you slashdot! I am yelling! stop filtering my text! I swear I will fucking end you! damnit. okay, that was an empty threat.
It's great when a prediction is both public and quantifiable.
Why not go Google Apps? The enterprise version is free for Education and non-profits and has almost all of the features Exchange does (and likely all the features needed by Education). Easy to support and for the most part, hands off. Huge mailboxes, decent sites and docs and a UI that many find better than Outlook.
Disclaimer- I don't work for Google, but I do work for a media company where we rolled out a 5,000 seat Google Apps estate.
First, Exchange is not an email server. It is a groupware server. If they are using it as the MTA then they should be shot on sight. That being said, if one machine can not handle 2000 accounts there is something critically wrong with the software or the choice of hardware running it. If they didn't anticipate adding 2000 accounts to an existing system see: shot on sight.
Having to work for a living is the root of all evil.
This is the same IT group that closes its employee payroll information site on nights and weekends. Yup, you read that right -- the NYC DoE "Payroll Portal" where 80,000+ employees check their pay stubs is only open during business hours. It's never been clear why that is -- they couldn't possibly have people pulling the data manually for each request, could they? So you teach all day, go home, apply for an apartment, and can't get your pay stubs at 8 p.m. from a system that is touted as convenient and accessible over the public internet.
Point being, this is a function that is probably short on resources, but also fails to make the most of the funding and systems they do have.
Deploy another host, deploy another template VM to distribute load? Surely it's a plug-in, click/drag fix once they add a new host, right? +1 Scalability.
I'm not saying IT workers aren't social, or don't tolerate bureaucracy, but....
Just wait until the union you join enforces workrules which limit your productivity or options.
Some years back we serviced a rack-mount system we had delivered at Lockheed, which was a union shop. Only the 'equipment tech', could remove the device from the rack. Only a soldering tech could solder. Our engineer wasn't even allowed to type on the keyboard, some other specialist was required.
When the job was done and it was time to put the equipment back in the rack, the equipment tech was nowhere in sight. When our engineer 'just did it', it created a cross-company fiasco requiring senior level apologies and a personal apology from our engineer.
So, IT workers, good luck with the union. You're gonna LOVE it.
Rather than simply saying use a different mail-server, does anyone know whether limiting access to the Exchange server via IMAP would provide less impact?
Jumpstart the tartan drive.
$1MM of iPads represents about 2500-3000 users depending on the discount they received. First, I'm presuming that these users already had mailboxes and it's just the additional load of ActiveSync that is causing the trouble. If that's the case, with the types of discounts that government and education receive from microsoft and hardware vendors this is like a $15,000 problem at best. In the scope of a million-dollar project a 1.5% budget problem represents poor planning, but I've seen much much worse.
What's happening in NYC is what comes of choosing a product with cleverly engineered rarity built-in. How much time has been spent on ensuring Exchange is -not- available? Perhaps that time could be better spent on making a product that works better, as opposed to intentionally failing.
One of my complaints about Exchange (and indeed, Microsoft's products in general) is that they're full of bad interactions like that.
(My personal favorite is that installing Outlook (the Exchange client) on the same box as Exchange server causes the server to stop working. (For 2000 and 2003. Not sure on 2007+.) Not that I plan on reading email on the server, but for trouble-shooting it would be useful.)
You're pretty much forced to keep everything on separate servers if you want everything to work as designed.
Sure, in a good sized organization you'd be doing that anyway for performance, but in smaller orgs it's a real pain.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
iPads are extremely easy for an enterprise to manage, because they integrate nicely into Exchange (e.g. you can define mail policies on your Exchange server, and iPads do what they're told - encrypt, require password lock, etc.).
We're not finding iPad/iPhone easy to manage at our business. The available management policies are very meager compared to BlackBerry handhelds. Too many things require iTunes, and iTunes is a bear to deploy, update, and manage. When the iDevice malfunctions, diagnostics and repair attempts are very limited. And if we need to do a service/warranty exchange, pain results. They won't ship an FRU; you have to go to a store. And apparently Apple's corporate policy forbids stores from telling customers if they have stock of FRUs, so the only way to find out is to drive to each store and try.
This is not saying that Android or Playbook tablets are any better (we haven't even tried those yet), but iDevices aren't all lollypops and rainbows either.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
I worked first tier tech support for the NYC DOE for 2 years. The amount of space the regular employees get for email is 25MB. Asst. Principals get 30, and Principals get 50.
I bet cloud services are starting to look REALLY tempting to them right now. No worrying about overwhelmed servers, it Just Works.
Google is probably salivating while hearing about this.
That is the dumbest way of troubleshooting I have ever heard. You're troubleshooting a client issue on a server os, use a VM of windows 7, vista, xp whatever the client is running...
Not everything divides cleanly into "client" and "server", even in the best designs. Sometimes it's a network or transport issue. Being able to run the client on the server would make it easier to determine where things are going wrong. This is especially the case with MAPI, since before 2007 it's basically just a set of RPC calls into the Information Store structure.
I never understood the desire to remove tools from one's arsenal.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
Switch to the Google Apps GovCloud or Microsoft Live 365. Oh, it isn't secure!!! Do you think the current admins have Exchange set up securely? You also get spam blocking and Postini with Google.
A NYC lawyer blogs. http://www.chuangblog.com/
What if they just escaped MS Exchange and switched to Zimbra? Isn't Zimbra a lot more scalable, and requires only installing a Zimbra MAPI connector at the client to switch to the configured Zimbra server?
They'd have to rewrite in all their legacy emails SMTP addresses instead of the Exchange X.500 addresses, or replying to them will cause the new recipient addresses to be blank. Isn't there a way to do that inside Zimbra after loading it with the old messages, rather than running something inside the overwhelmed old Exchange server?
--
make install -not war
So you are saying that's not a problem for open source software packages that make you pay per user?
I did not believe, until today, that someone could come up with a klein-bottle-shaped thought. I bow down to you, sir.
-- Terry
It sounds like they failed to buy some new servers (and related infrastructure) when they purchased new clients. In a networked environment, server-side costs must be considered when new clients are purchased.
Exchange sucks. It's incredibly expensive, incredibly inefficient, and incredibly prone to problems. Every company where I've used it I've had issues. It seems the only reason IT people recommend it is to keep the helpdesk employed.
NY could have saved a million dollars by using google apps for education, which has the benefit of actually being able to perform a service to the people who are supposed to be able to use it and had the benefit of being free.
When the options are hassle and cost free vs. pile of exchange, I don't understand why anyone chooses exchange.
or else!
I could see how a policy like that would be logical (albeit paranoid) for internal communications. But what about emails to the outside? Assuming teachers use email to communicate with their students and parents, there will be plenty of "business records in the cloud" whether teachers use gmail or the school's Exchange system.
They should follow the state of Kentucky's lead and migrate everyone over to Live@edu, Microsoft's free cloud-based email, file storage, messenger and web app, system for schools. Kentucky was able to migrate 700,000 accounts over to Live@edu in the course of a weekend. (Ref: http://msftedublogger.wordpress.com/2010/06/03/state-of-kentucky-rolls-out-liveedu-to-700000-users/)
My school has been using Live@edu since the beginning of the year and it has been an unqualified success. Everthing is web-based (although you can use Outlook or another mail client if you prefer), which means it it also cross-platform. Mail is synced between all of a user's devices. It's pretty slick. The SkyDrive storage offers 25 GB of storage and is HTML5 based (but uses Silverlight on Windows).
Did I mention it's free?
An invasion of armies can be resisted, but not an idea whose time has come. - Victor Hugo
Not mentioned: they are running the ActiveSync server on a single 386 machine with 5M of RAM....
I'll repeat for your benefit:
This is not saying that Android or Playbook tablets are any better (we haven't even tried those yet), but iDevices aren't all lollypops and rainbows either.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
They were likely at or near capacity on their existing Exchange / ActiveSync Server when someone else outside of the Exchange group made the decision to get a bunch of iPads. I work with almost a thousand iPads and yes, you need to plan for capacity on the Exchange server. That being said, you had better also plan on WiFi capacity because iPads are bandwidth hogs (not really). They are so very useful that the users actually use them and in doing so pound your network quite a bit more than even laptop users.
I worked a large remote hotel gig where we deployed 200 iPads and it literally killed the hotels network about 6 times over 2 days! We had to shutoff the WiFi service in the convention rooms during a video conference for fear the iPads would crash the network again. We offered to bring our own WiFi network and work with Verizon to do it (big bucks on our part) but the hotels outsourced IT company refused to let us bring our own network. So we used what they provided and pretty much frequently killed network service for all other hotel guests during our stay. The iPads overwhelmed their wimpy routers. The hotel network was designed for guests to check email and web surf a little. It didn't like 200 iPads running Cisco WebEx. I doubt they used advanced networking with bandwidth shaping, etc.