Apple, Android Devices Swamp NYC Schools' ActiveSync Server

longacre writes "Just a few months after the New York City Dept. of Education shelled out over $1 million on iPads for teachers, the agency has stopped accepting new users on its Microsoft Exchange ActiveSync server as it is 'operating near its resource limits' due to an influx of iOS and Android devices. A memo from the deputy CTO warned, 'Our Exchange system is currently operating near its resource limits and in order to prevent Exchange from exceeding these limits, we need to take action to prevent any more of these devices from being configured to receive email. As of Thursday, November 10th no additional users will be allowed to receive email via NYCDOE's Exchange ActiveSync.' Existing setups will continue to operate, and students will not be affected."

Comments

Best use of money?

[Anonymous Coward]

Imagine what they could have done with the $700k they would have saved by choosing a tablet other than an iPad.

Re:Best use of money?

[ozmanjusri]

Bought a decent mail server?

Re:Best use of money?

[TWX]

I don't see what's specifically insulting Microsoft with, "Bought a decent mail server?" as a comment... It could be insulting the juggernaut from Redmond, or it could be insulting the hardware. It's a supposition on your part that the OS/daemon is being insulted.

Mind you, I would not be surprised if the software was the target, but that's mainly because I don't think that the service was originally designed for the kind of usage that it's now seeing. We also don't know from the summary what versions of things they're running, they could still be back on Server 2003 or the like...

Re:Best use of money?

[OS24Ever]

Then the headline would have read 'NYC Schools waste Millions on tablets no one uses'

Re:Best use of money?

[multimediavt]

Imagine what they could have done with the $700k they would have saved by choosing a tablet other than an iPad.

Bought a decent mail server?

My thoughts exactly! The devices aren't the problem, their proprietary commercial mail system that sucks is the problem. Nice to watch people eat crow when they tout the charms of commercial software and its scalability advantages and it epically fails and costs more money than a FOSS solution. Best quote I ever heard was from a guy talking about AD, "It's got to be complicated, it has to scale." Face-palm!!!

Re:Best use of money?

[smash]

Like which decent tablet is significantly cheaper than an iPad?

Re:Best use of money?

[laird]

"Imagine what they could have done with the $700k they would have saved by choosing a tablet other than an iPad."

The iPad is under $500, so it costs the same or less than any other decent tablet. Are you saying that there's a tablet that costs $150 that's comparable to the iPad? That is pretty hard to imagine. Don't forget to include the management costs - iPads are extremely easy for an enterprise to manage, because they integrate nicely into Exchange (e.g. you can define mail policies on your Exchange server, and iPads do what they're told - encrypt, require password lock, etc.). Android doesn't do this properly yet. That leaves the RIM Playbook, which aside from sucking has the same list price as the iPad. I guess you could save some money buying discontinued products that are being dumped, but that's not a great enterprise hardware strategy. :-)

If you want to complain about the project, complain that they didn't plan for adding one more ActiveSynch server so they had capacity to support their users. Given educational pricing, the software is nearly free, and even an overpriced server would have been a trivial percentage of the project budget.

Re:Best use of money?

[chewedtoothpick]

Currently trying to find any kind of Open Source collaboration Server - I can assure you that the software costs alone are within 5% on any of the decently known and supported alternatives such as Zimbra, Zafura, eGroupware, open-Xchange etc. Zafura is the closest in terms of quality, but based on my testing, I have noticed that with my 15-user test groups (all users using at least three devices to sync continuously) Zimbra is the closest to Exchange in terms of efficiency, and if you remove OS resource usage I have noticed that the Exchange daemon is the most efficient. I hadn't gone through any kind of upgrade testing to see how easy that is (which could alone still sway me away from my current direction of updating Exchange) but when compared to how much easier it is to tie Exchange into Active Directory and properly apply any domain-controlled policies to clients, Exchange wins hands-down in any system that isn't wholly unix from the ground-up.

In conclusion, you are misinformed if you think any kind of FOSS system can compete with Exchange. If you want any kind of collaboration utilities (Calendar or Contact sync and grouping, etc) then you discard the F part and usually the OS part too - and the supposed knights of FOSS are even more greedy than Moneyholesoft from Redmond. At least Redmond allowed us a 3-month trial with 60 users to test out compared to the others. VMWare let us try Zimbra with 30 users for 30 days before they wanted to charge us - nobody else would even let us trial their packages at all.

Re:Best use of money?

[NatasRevol]

The playbook would also require a blackberry phone for everyone to check their email via.

Re:Best use of money?

[Albanach]

their proprietary commercial mail system that sucks is the problem.

You really think so? Microsoft sell Exchange to some of the largest organizations on the planet. It might not be my choice of mail server, but I don't think blaming the software is the right think to do here. There's plenty of evidence that Exchange can scale - it might need powerful hardware, or specialized configuration but it's clearly possible and widely implemented.

The real trouble here was not not the choice of software. Rather it was a failure to anticipate the growth and react to it before it became an issue. That's a very basic SysAdmin issue for any software, proprietary or otherwise.

Re:Best use of money?

[mosb1000]

He was saying that if they chose tablets significantly cheaper than the iPad, as the parent suggested, no one would use them. It's probably true.

Re:Best use of money?

[HateBreeder]

What's more ridiculous is that you insist on using terns that contain the word "fuck" and then use the substitution "fsck". If "fuck" is so impolite for you to use, then why not express yourself differently?

Re:Best use of money?

[jbplou]

The problem could just be a lack of capacity planning. When management says we are going to add $1 million worth of iPads on to our mail system plus let users use iPhones and droids the mail admins should be evaluating their infrastructure.

Re:Best use of money?

[tqk]

What's more ridiculous is that you insist on using terns that contain the word "fuck" and then use the substitution "fsck". If "fuck" is so impolite for you to use, then why not express yourself differently?

I'm screwing with the Pakistani forbidden word IM filter. :-)

Re:Best use of money?

[DigiShaman]

Agreed. If they only used one Exchange server holding all the roles, that was the problem right there. You can do that with a Microsoft SBS box, but it doesn't scale and there is a user limitation defined for a reason. Exchange is designed to have its roles delegated to other servers for load balancing and scalability. Technically this problem is easily solvable. It's getting additional funding that will be the hard part. Namely, additional MX records created and additional Exchange servers rolled out. Oh, and they may need more ISP bandwidth.

Here a some examples in how to implement Exchange per MS.

http://technet.microsoft.com/en-us/library/dd979781(EXCHG.140).aspx

Re:Best use of money?

[ninetyninebottles]

their proprietary commercial mail system that sucks is the problem.

You really think so? Microsoft sell Exchange to some of the largest organizations on the planet. It might not be my choice of mail server, but I don't think blaming the software is the right think to do here. There's plenty of evidence that Exchange can scale - it might need powerful hardware, or specialized configuration but it's clearly possible and widely implemented.

I'm fairly sure Exchange could scale up to more users. The problem is most likely twofold, they don't have the hardware resources and they don't have enough client licenses. Both can be solved with money, but the latter is only a problem on a proprietary commercial platform that makes you pay per user.

Re:Best use of money?

[gad_zuki!]

Don't bother with this crowd. These guys clearly have no practical experience with Exchange and are the same people who have been yelling "ZOMG POSTFIX AND EVOLUTION/CHANDLER/THUNDERBIRD WILL KILL OUTLOOK" 10+ years ago.

As much as I dislike defending my vendors, I have to say the Exchange is surprisingly nimble and the number of devices I can support with a very modest server is pretty surprising. The idea that you're getting 10x the number of users on similiar hardware with a similiar featureset is the same bullshit these FOSS guys have been peddling for years. I just with the FOSS crew could write a usable, supported, efficient Exchange/Activesync replacement. That product doesn't exist and the current crop are all nightmares. Heh, there's a reason why they won't let you test this junk.

Re:Best use of money?

[NeilO]

Imagine what they could have done with a million dollars in books, art supplies, lab equipment, musical instruments, ...

Triple the load =/= triple the servers?

[SlashdotWanker]

Issues like this are the reason you need to fully flesh out costs before flipping the switch on a large organization like this. almost every teacher I know has a smartphone of some kind and a lot of them are starting to get tablets. Why offer the service when you cannot fully offer it?

No better CAS topology experts?

[swb]

Given the resources, is there any reason they couldn't scale this right? I only pretend to know anything about Exchange, but this seems kind of strange.

I'm sure that resource limitations -- server CPU, disk, etc -- are the source of this, but you'd think a high profile customer like this would be able to get MS involved before the story becomes "iPads crash Exchange" or "consumer tablet bests high dollar PC server."

Re:No better CAS topology experts?

[gbjbaanb]

part of the problem is that Exchange is not an email server (many people think that because they only use it for email). It's a "groupware" server that does email, calendars, notes, journals, todo lists, integrated MSN status, etc.

Now admittedly, all those things shouldn't be particularly resource intensive, but the Exchange systems that have been around for years always struggled to to simple things. I think that they made it better at resource usage, but then probably made it much worse by bundling in crap like MSN status updates and probably facebook integration by now too.

If they replaced Exchange with a straightforward mail server like Dovecot, they'd handle a hundred times those users with ease. Sure, they wouldn't have an integrated calendar... but which of those users uses the exchange calendar anyway, using some preferred iOS or Google calendar.

In short, Exchange should only ever be used if you're inside a corporate network and you're an all-Microsoft shop. And you're rich enough to buy the super servers and licences you'll need. For everyone else, stick with stuff that just does a couple of things very well.

Re:No better CAS topology experts?

[DarkOx]

Yes, provided we are talking about an Exchange 2007 or later environment you can scale as big as need, Provided:

1. You have adequate hardware resources
2. You have selected an appropriate deployment strategy for your organizations size, and anticipated growth. If you expect to grow big but currently are not you do need to make some architectural decisions which will raise upfront costs.

*Modern* Exchange is among Microsoft's products that really can be considered carrier grade. Like all MS products through there is plenty of rope for the improperly trained to hang themselves with. Its not hard for an rank amateur to get Exchange running well enough even for a fairly large organization for years, and suddenly find they have steered themselves down the wrong architectural path are about to hit a wall and have only very expensive, in both time and money terms ways out.

Re:No better CAS topology experts?

[Locutus]

Don't be so sure about that hardware being the "resource limit" although it could very well be the case. Microsoft requires you pay a per device/client access fee and it could be they are exceeding what they paid for. The number of Microsoft Client Access Licenses( CAL ) to connect to the Microsoft Exchange Server wasn't listed and they are being pretty vague as to what "resource" limits are being reached.

LoB

Re:No better CAS topology experts?

[DarkOx]

I am not a Microsoft fanboi by any stretch and those were all valid criticisms of Exchange 2003 and prior, however Exchange 2007 and later have a pretty clean architecture and good support for open standards. The only real argument against is that, it is very expensive and you might really only need mail in which case you can get carrier class mail handling with FOSS.

Re:No better CAS topology experts?

[adamstew]

Exchange CALs are licensed per user and not per device. Each user can have any number of devices hooked up. If they are already properly licensed for each of their users, then all devices that any of those users brings are already licensed.

Also, even if they had met their limit on CALs, they are so cheap for the education market, Microsoft practically gives them away.

This is what you get with golf course deals

[Joe_Dragon]

This is what you get with golf course deals people out side of IT makes deals like this and tell IT to make it work with out giving them the funds to make it work.

This why IT needs unions so they can stand up and say NO! we can't do it with the funds that we have. I hope that they don't place the blame on IT for something that is not there fault.

Re:This is what you get with golf course deals

[laird]

"Unions work best for the health and safety of their workers. Anything beyond that is mob rule."

Add in "and are properly equipped and trained and resourced to do their job successfully". For example, air traffic control unions negotiated limits on how many hours controllers could be forced to work, and when they unions were broken and controllers were forced to work so many hours, with no breaks for even going to the bathroom or eating meals, endangering passenger's lives. And when teachers' unions negotiate limits on the numbers of students in classes, so teachers can actually teach students effectively.

Or do you think that the MBA who runs a company knows how best to do people's jobs, not the people who actually do the jobs?

Re:This is what you get with golf course deals

[laird]

The point of unions isn't that they render everyone angels, it's that it creates an organization that can negotiate in favor of worker's interests to balance the organization that already exists to support management's interests. So an IT workers' union could impose checkpoints in a process such that the workers could make sure that adaquate resources, training, tools, etc., were provided to allow the workers to be successful without working insane hours compensating for poor planning or resourcing. Yes, a good management team ought to be thinking of such things, but the software industry's track record is poor enough (only 10% of IT projects deliver what's required on time and budget) that giving the IT workers more leverage doesn't seem like a bad idea.

Re:This is what you get with golf course deals

[swb]

Go ahead and form a union, lots of people will keep making it work and getting your management's money.

FWIW, I have Exchange 2010 running in VMware. The host is only a Q6600 quad 2.4Ghz with 8 gigs of RAM. The Exchange VM is 3 cores + 3 GB RAM. Tiny. But it seems to serve 4 iOS devices and 2-4 Outlook clients simultaneously without any issues. The host itself runs 4-5 other VMs.

It's admittedly dog-slow to startup and even console logins are slow to process and launching the Exchange GUI is agonizing, but it does work, and this is all in a memory footprint way smaller than recommended.

Re:This is what you get with golf course deals

[hey!]

So, what we're talking about here is the question of who can be trusted with power, workers or management?

The answer is simple: neither. All the horror stories you've heard about unions? True for some union somewhere. All the horror stories you've heard about management? Also true for some managers somewhere. It'd be a different world if we could just assume people would cooperate when it was in their obvious common interest, but we can't. Good faith is such a fragile thing.

If there were one quality which could fix everything that is wrong with this world, it would be integrity. But we can't be truthful with each other because we're not honest with themselves. We often act like our personal insecurities entitle us to be a**sholes. So the best we can manage in a world where integrity is so uncommon is to organize all the a**holes into competing teams then let them duke it out.

Get ready for the headlines

[betterunixthanunix]

Get ready for followup headlines a few months or years from now:

• NYC drops $600 million on new email system
• Consulting firm under investigation for defrauding NYC public school system in email debacle
• Should public schools have email systems?

This is a pretty standard situation in New York City: lots and lots of money is spent, with poor planning, sweetheart deals with incompetent firms, and then a bunch of fallout.

Go figure

[nurb432]

Lack of resource forecasting/planning will get you every time. Its not like they didn't know how many would be deployed and on what schedule.. geesh

Re:So wait a minute...

[betterunixthanunix]

I'm guessing that something's missing from the story here...

They were probably near capacity before the tablets were deployed. NYC has a lot of schools and a lot of teachers and administrators checking their email. The fact that tablets are involved is secondary; if 2000 additional desktops had been deployed, the systems would probably have been overwhelmed as well. My guess is that the email system was deployed years ago, possibly by a consulting firm that is now out of business, and that some poor IT guy has been trying to keep everything together on a shoestring budget all this time. The tablet deployment probably occurred without anyone actually consulting the IT staff to see if the system could handle the extra load, and probably by the same group of decision makers who ignored IT's requests for additional servers prior to the deployment.

Typical...

[Anonymous Coward]

This is the same IT group that closes its employee payroll information site on nights and weekends. Yup, you read that right -- the NYC DoE "Payroll Portal" where 80,000+ employees check their pay stubs is only open during business hours. It's never been clear why that is -- they couldn't possibly have people pulling the data manually for each request, could they? So you teach all day, go home, apply for an apartment, and can't get your pay stubs at 8 p.m. from a system that is touted as convenient and accessible over the public internet.

Point being, this is a function that is probably short on resources, but also fails to make the most of the funding and systems they do have.

Re:Typical...

[thegarbz]

Maybe they're....y'know.... doing a pay run. or maintenance....

Then I'd accept 15min - 1h daily in the middle of the night as reasonable downtime for that kind of system. God our system at work handles 30k clients and is down for 20min a week to perform this task. Extensive maintenance sometimes brings it down for an hour on the weekend. Usually after 7pm on a Saturday.

Re:Something's missing from the story

[smash]

Alternatively, PHB simply said "iPads for all!" without consulting the IT department or asking about the back end infrastructure required to support such a decision.

Re:Google Apps?

[smash]

Not everybody likes having their data owned by Google.

Virtualization, Anyone?

[arhhook]

Deploy another host, deploy another template VM to distribute load? Surely it's a plug-in, click/drag fix once they add a new host, right? +1 Scalability.

Re:Teachers should just switch to gmail

[Sprouticus]

Spoken like someone who knows nothing about email systems.

I am guessing that there are strict restrictions on using external email to relay school information.

After all do you want your information on your childs health, disciplinary issues, grades, concerns over abuse, etc etc. to be stored on googles mail server? I sure as hell dont.

Re:Teachers should just switch to gmail

[bytethese]

Depending on the rules of a particular organization however, you could get fired for that. At my currently employer, you cannot use a personal email address for business purposes. Doing so leaves business records in the cloud and that's a no no. Since this is a public school, I'm guessing that it would also be a no no there too.

Re:Teachers should just switch to gmail

[cryfreedomlove]

Spoken like someone who knows nothing about email systems.

I am guessing that there are strict restrictions on using external email to relay school information.

After all do you want your information on your childs health, disciplinary issues, grades, concerns over abuse, etc etc. to be stored on googles mail server? I sure as hell dont.

I trust my anonymity with Google more than with a B-grade IT worker at a school district. Imagine 2 possible scenarios:

1. Google does something with my email data i don't like.

2. A disgruntled IT worker at the school district sells my email data for drug money.

#2 is far more likely.

iDevice enterprise mgmt != easy

[DragonHawk]

iPads are extremely easy for an enterprise to manage, because they integrate nicely into Exchange (e.g. you can define mail policies on your Exchange server, and iPads do what they're told - encrypt, require password lock, etc.).

We're not finding iPad/iPhone easy to manage at our business. The available management policies are very meager compared to BlackBerry handhelds. Too many things require iTunes, and iTunes is a bear to deploy, update, and manage. When the iDevice malfunctions, diagnostics and repair attempts are very limited. And if we need to do a service/warranty exchange, pain results. They won't ship an FRU; you have to go to a store. And apparently Apple's corporate policy forbids stores from telling customers if they have stock of FRUs, so the only way to find out is to drive to each store and try.

This is not saying that Android or Playbook tablets are any better (we haven't even tried those yet), but iDevices aren't all lollypops and rainbows either.