Slashdot Mirror
Apple, Android Devices Swamp NYC Schools' ActiveSync Server
longacre writes "Just a few months after the New York City Dept. of Education shelled out over $1 million on iPads for teachers, the agency has stopped accepting new users on its Microsoft Exchange ActiveSync server as it is 'operating near its resource limits' due to an influx of iOS and Android devices. A memo from the deputy CTO warned, 'Our Exchange system is currently operating near its resource limits and in order to prevent Exchange from exceeding these limits, we need to take action to prevent any more of these devices from being configured to receive email. As of Thursday, November 10th no additional users will be allowed to receive email via NYCDOE's Exchange ActiveSync.' Existing setups will continue to operate, and students will not be affected."
Imagine what they could have done with the $700k they would have saved by choosing a tablet other than an iPad.
Issues like this are the reason you need to fully flesh out costs before flipping the switch on a large organization like this. almost every teacher I know has a smartphone of some kind and a lot of them are starting to get tablets. Why offer the service when you cannot fully offer it?
Given the resources, is there any reason they couldn't scale this right? I only pretend to know anything about Exchange, but this seems kind of strange.
I'm sure that resource limitations -- server CPU, disk, etc -- are the source of this, but you'd think a high profile customer like this would be able to get MS involved before the story becomes "iPads crash Exchange" or "consumer tablet bests high dollar PC server."
This is what you get with golf course deals people out side of IT makes deals like this and tell IT to make it work with out giving them the funds to make it work.
This why IT needs unions so they can stand up and say NO! we can't do it with the funds that we have. I hope that they don't place the blame on IT for something that is not there fault.
This is a pretty standard situation in New York City: lots and lots of money is spent, with poor planning, sweetheart deals with incompetent firms, and then a bunch of fallout.
Palm trees and 8
$1m spent on iPads only comes to ~2,000 iPads at most (assuming the cheapest model at around $500 each). According to Microsoft's handy little Capacity Planner (Exch 2010), it shouldn't take but perhaps (very rough calc here) 5 or 10 servers at most to handle that, unless they're also allowing every school employee to latch on their personal gear as well.
I'm guessing that something's missing from the story here...
Quo usque tandem abutere, Nimbus, patientia nostra?
Lack of resource forecasting/planning will get you every time. Its not like they didn't know how many would be deployed and on what schedule.. geesh
---- Booth was a patriot ----
Just that their IT staff is incompetent..
---- Booth was a patriot ----
Right?
Vote monkeys into Congress. They are cheaper and more trustworthy.
I do not think that privacy is the chief concern when it comes to personal devices on school networks. More likely there is a support contract getting in the way; my high school (in NYC) had a bunch of desktops that could not be connected to the school's network because of a support contract stipulation. Internal emails are probably easy to forward or otherwise export from the schools' computers, and the security is probably very poor (when I was in school, the only think separating the teachers' network from the students' was the IP address assigned to the computer -- and anyone could manually set the IP address, which is how we defeated the censorship firewall).
Palm trees and 8
It's great when a prediction is both public and quantifiable.
This is the same IT group that closes its employee payroll information site on nights and weekends. Yup, you read that right -- the NYC DoE "Payroll Portal" where 80,000+ employees check their pay stubs is only open during business hours. It's never been clear why that is -- they couldn't possibly have people pulling the data manually for each request, could they? So you teach all day, go home, apply for an apartment, and can't get your pay stubs at 8 p.m. from a system that is touted as convenient and accessible over the public internet.
Point being, this is a function that is probably short on resources, but also fails to make the most of the funding and systems they do have.
Not everybody likes having their data owned by Google.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
Deploy another host, deploy another template VM to distribute load? Surely it's a plug-in, click/drag fix once they add a new host, right? +1 Scalability.
I'm not saying IT workers aren't social, or don't tolerate bureaucracy, but....
Just wait until the union you join enforces workrules which limit your productivity or options.
Some years back we serviced a rack-mount system we had delivered at Lockheed, which was a union shop. Only the 'equipment tech', could remove the device from the rack. Only a soldering tech could solder. Our engineer wasn't even allowed to type on the keyboard, some other specialist was required.
When the job was done and it was time to put the equipment back in the rack, the equipment tech was nowhere in sight. When our engineer 'just did it', it created a cross-company fiasco requiring senior level apologies and a personal apology from our engineer.
So, IT workers, good luck with the union. You're gonna LOVE it.
Spoken like someone who knows nothing about email systems.
I am guessing that there are strict restrictions on using external email to relay school information.
After all do you want your information on your childs health, disciplinary issues, grades, concerns over abuse, etc etc. to be stored on googles mail server? I sure as hell dont.
yea for sladot. A few corrections
1) Exchange 2010 has a perfectly good MTA. I would argue that MTA's are the least of what a modern email system does.
2) Activesync utilizes 80/443 for connections, not port 25.
3) They are not adding 2000 new accounts, they are adding 2000 new devices to connect to the accounts.
4) In all likelyhood this is a simple issue of the CAS (Client Access Server) not being size right, or not being sized to include the increase in traffic which would occur (mind you the IT dept might not even have known about this deal when it was made. Hard to size something out when you dont know about it.
5) In all likelyhood this is just a matter of throwing hardware at the issue. If the CAS is running on an old server, or is very undersized, you just add a couple of servers into the array to handle the load.
It is not unlike a website getting overloaded and needing more nodes to handle the traffic. Im assuming they are already load balancing of course, but even if they were not it is not a huge deal. I put in an Exchange 2010 2 server CAS Array in 3 hours. configuration took a couple of days.
Depending on the rules of a particular organization however, you could get fired for that. At my currently employer, you cannot use a personal email address for business purposes. Doing so leaves business records in the cloud and that's a no no. Since this is a public school, I'm guessing that it would also be a no no there too.
3) They are not adding 2000 new accounts, they are adding 2000 new devices to connect to the accounts.
If the 2000 accounts existed and different devices are now connecting to them then there is something fundamentally wrong with the software.
Having to work for a living is the root of all evil.
Spoken like someone who knows nothing about email systems.
I am guessing that there are strict restrictions on using external email to relay school information.
After all do you want your information on your childs health, disciplinary issues, grades, concerns over abuse, etc etc. to be stored on googles mail server? I sure as hell dont.
I trust my anonymity with Google more than with a B-grade IT worker at a school district. Imagine 2 possible scenarios:
1. Google does something with my email data i don't like.
2. A disgruntled IT worker at the school district sells my email data for drug money.
#2 is far more likely.
Rather than simply saying use a different mail-server, does anyone know whether limiting access to the Exchange server via IMAP would provide less impact?
Jumpstart the tartan drive.
Are Americans really that uptight, or are you just assuming that school divisions are as paranoid as corporations?
$1MM of iPads represents about 2500-3000 users depending on the discount they received. First, I'm presuming that these users already had mailboxes and it's just the additional load of ActiveSync that is causing the trouble. If that's the case, with the types of discounts that government and education receive from microsoft and hardware vendors this is like a $15,000 problem at best. In the scope of a million-dollar project a 1.5% budget problem represents poor planning, but I've seen much much worse.
One of my complaints about Exchange (and indeed, Microsoft's products in general) is that they're full of bad interactions like that.
(My personal favorite is that installing Outlook (the Exchange client) on the same box as Exchange server causes the server to stop working. (For 2000 and 2003. Not sure on 2007+.) Not that I plan on reading email on the server, but for trouble-shooting it would be useful.)
You're pretty much forced to keep everything on separate servers if you want everything to work as designed.
Sure, in a good sized organization you'd be doing that anyway for performance, but in smaller orgs it's a real pain.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
iPads are extremely easy for an enterprise to manage, because they integrate nicely into Exchange (e.g. you can define mail policies on your Exchange server, and iPads do what they're told - encrypt, require password lock, etc.).
We're not finding iPad/iPhone easy to manage at our business. The available management policies are very meager compared to BlackBerry handhelds. Too many things require iTunes, and iTunes is a bear to deploy, update, and manage. When the iDevice malfunctions, diagnostics and repair attempts are very limited. And if we need to do a service/warranty exchange, pain results. They won't ship an FRU; you have to go to a store. And apparently Apple's corporate policy forbids stores from telling customers if they have stock of FRUs, so the only way to find out is to drive to each store and try.
This is not saying that Android or Playbook tablets are any better (we haven't even tried those yet), but iDevices aren't all lollypops and rainbows either.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
I worked first tier tech support for the NYC DOE for 2 years. The amount of space the regular employees get for email is 25MB. Asst. Principals get 30, and Principals get 50.
I bet cloud services are starting to look REALLY tempting to them right now. No worrying about overwhelmed servers, it Just Works.
Google is probably salivating while hearing about this.
That is the dumbest way of troubleshooting I have ever heard. You're troubleshooting a client issue on a server os, use a VM of windows 7, vista, xp whatever the client is running...
Not everything divides cleanly into "client" and "server", even in the best designs. Sometimes it's a network or transport issue. Being able to run the client on the server would make it easier to determine where things are going wrong. This is especially the case with MAPI, since before 2007 it's basically just a set of RPC calls into the Information Store structure.
I never understood the desire to remove tools from one's arsenal.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
Switch to the Google Apps GovCloud or Microsoft Live 365. Oh, it isn't secure!!! Do you think the current admins have Exchange set up securely? You also get spam blocking and Postini with Google.
A NYC lawyer blogs. http://www.chuangblog.com/
What if they just escaped MS Exchange and switched to Zimbra? Isn't Zimbra a lot more scalable, and requires only installing a Zimbra MAPI connector at the client to switch to the configured Zimbra server?
They'd have to rewrite in all their legacy emails SMTP addresses instead of the Exchange X.500 addresses, or replying to them will cause the new recipient addresses to be blank. Isn't there a way to do that inside Zimbra after loading it with the old messages, rather than running something inside the overwhelmed old Exchange server?
--
make install -not war
Are they public universities and schools? Are they the NYC Department of Education, the largest in the nation, with many hundreds of thousands of teachers, students and staff, and an IT budget of about $2 BILLION?
I didn't think so.
The NYC DOE doesn't have to use Microsoft. But it does have a lot more serious confidentiality and accessibility policies than most schools - or most organizations, even most governments - of any kind.
--
make install -not war
That doesn't make any sense at all. I read it, re-read it and I still don't understand what you wrote. How could there be something wrong with the software just because 2000 new devices are connecting to 2000 existing accounts?
Take a typical situation in a corporation: An employee gets a new laptop. Congratulations! That's one new device connecting to one existing account. Now, scale that up through a computer refresh cycle at a large corporation... say 200 new computers each quarter. Hey that's 800 new devices per year connecting to existing accounts!
Then add in smartphones, and, as in this case, iPads.
I'm not sure what point you were trying to make?
Regards,
dj
So you are saying that's not a problem for open source software packages that make you pay per user?
I did not believe, until today, that someone could come up with a klein-bottle-shaped thought. I bow down to you, sir.
-- Terry
Exchange sucks. It's incredibly expensive, incredibly inefficient, and incredibly prone to problems. Every company where I've used it I've had issues. It seems the only reason IT people recommend it is to keep the helpdesk employed.
NY could have saved a million dollars by using google apps for education, which has the benefit of actually being able to perform a service to the people who are supposed to be able to use it and had the benefit of being free.
When the options are hassle and cost free vs. pile of exchange, I don't understand why anyone chooses exchange.
or else!
They should follow the state of Kentucky's lead and migrate everyone over to Live@edu, Microsoft's free cloud-based email, file storage, messenger and web app, system for schools. Kentucky was able to migrate 700,000 accounts over to Live@edu in the course of a weekend. (Ref: http://msftedublogger.wordpress.com/2010/06/03/state-of-kentucky-rolls-out-liveedu-to-700000-users/)
My school has been using Live@edu since the beginning of the year and it has been an unqualified success. Everthing is web-based (although you can use Outlook or another mail client if you prefer), which means it it also cross-platform. Mail is synced between all of a user's devices. It's pretty slick. The SkyDrive storage offers 25 GB of storage and is HTML5 based (but uses Silverlight on Windows).
Did I mention it's free?
An invasion of armies can be resisted, but not an idea whose time has come. - Victor Hugo
No, they use Google Apps for Education, which is different. Google Apps for Education comes with an SLA, Privacy Agreement (i.e. Google does not, or claims not to, data-mine the information), and is set up by replacing (or supplementing) your existing infrastructure. It is, unlike Gmail, actually certified for the purpose and unlike Gmail it is legal to be used for official purposes. Just switching to your personal Gmail account for official school business violates everything from confidentiality laws to transparency (official records) laws, and everything in between.
Also, Exchange servers cannot be cracked by "a 13 year old with halfway decent hacking skills". You're just showing your FUD-spewing Linux Zealot tendencies there.
For a site about things like basic rights, Slashdot users sure do like to censor "dissent".
Oh I see, your saying they connect in addition to the original devices. I didn't know the kids synced their phones at the same time they synced their laptops, and desktops, and iPads. That sure seems to be a lot of devices. I've heard of schools going to a 1 to 1 program but a 3 to 1 program is pretty original.
Having to work for a living is the root of all evil.
It is probably staff rather than students, but if I see an email on my phone that requires a lengthy reply, I go to my desktop/laptop, fire up Oulook, and type the reply there.
"If the 2000 accounts existed and different devices are now connecting to them then there is something fundamentally wrong with the software."
Keep in mind that the 2,000 new mobile devices are being used IN ADDITION TO however those people read email. If they stopped using their desktops and just used iPads, the load on the system would be unchanged. Adding 2,000 new clients to any mail system will consume the system's capacity, and if the system doesn't have that capacity it'll be in trouble. There's no magic capacity fairy.
Enable 3D printed prosthetics!
I'll repeat for your benefit:
This is not saying that Android or Playbook tablets are any better (we haven't even tried those yet), but iDevices aren't all lollypops and rainbows either.
dragonhawk@iname.microsoft.com
I do not like Microsoft. Remove them from my email address.
They were likely at or near capacity on their existing Exchange / ActiveSync Server when someone else outside of the Exchange group made the decision to get a bunch of iPads. I work with almost a thousand iPads and yes, you need to plan for capacity on the Exchange server. That being said, you had better also plan on WiFi capacity because iPads are bandwidth hogs (not really). They are so very useful that the users actually use them and in doing so pound your network quite a bit more than even laptop users.
I worked a large remote hotel gig where we deployed 200 iPads and it literally killed the hotels network about 6 times over 2 days! We had to shutoff the WiFi service in the convention rooms during a video conference for fear the iPads would crash the network again. We offered to bring our own WiFi network and work with Verizon to do it (big bucks on our part) but the hotels outsourced IT company refused to let us bring our own network. So we used what they provided and pretty much frequently killed network service for all other hotel guests during our stay. The iPads overwhelmed their wimpy routers. The hotel network was designed for guests to check email and web surf a little. It didn't like 200 iPads running Cisco WebEx. I doubt they used advanced networking with bandwidth shaping, etc.