How Overhauling IT Was a Life-Saver For the American Cancer Society

Lemeowski writes: American Cancer Society CIO Jay Ferro lets readers peek behind the curtain of the nonprofit's IT organization, saying that when he took on the role a little over three years ago, the nonprofit had 12 different divisions — each with its own independent IT set-up and more than 600 independent applications in its portfolio. In the past three years, Ferro has aligned the entire IT organization into one global entity, consolidating dozens of data centers into three; increasing spending on strategic projects from 5 percent to 40 percent, and reducing 600 core systems down to fewer than 200. His journey is a powerful reminder that while streamlining IT can often be painful upfront for IT managers, the payoff for sticking with it, especially for nonprofits, can feed into saving more lives.

Comments

First Post with good info

[randalware]

IT leadership is lacking in most companies.

The idea a good manager can manage anything is wrong.

After many years, I had a friend in manager ask me a question or two outside the office.
He had been in many management roles, and was working as a project manager for some major Oracle DB projects.

After explaining and doing a short lesson on all the major tech terms the DBs & system admin use.

I asked what does management understand.
On time (or not), Manpower Head count & budget.

They almost never have a clue about the interaction of the systems, quality, maint, network (or any infrastructure) & replacement (timing)
One manager I worked for needed a powerpoint to explain why upgrading the memory in the main unix server was needed,
after all we had just install 8gb in his PC.

Buy another tape cabinet with every project - BOFH (?)

Re:First Post with good info

[Guildor]

Yeah, similar to my experiences. As a software development contractor, I see a lot of management that care more for office politics, and who's toes they can stand on next, whilst playing the game of "hot-potatoe" when it comes to blame. Not my department! (points to next manager, as they're all stood in a circle).

Often they're looking for career progression, and not caring about the software development changes or support changes, but more about the "is it done yet?" - meaning, forget about it being right, refactored, consolidated, any kind of documentation, etc, and testing is assumed that the moment code is up and running, that it's fixed. So they ask you "how you're getting on" - and the only thing word they hear is "working", and they report that. Then when they find out it isn't, they immediately pile on the pressure. "you told me it was working!" - "you better get it done, quickly!, and get it in!" (meaning source control and released) so as to cover up their lack of understanding.

I've actually confronted a number of project managers, and their managers on such issues - even taking it as high as contacting board members through email (risking my contract!) to tell them they've got a communication problem. Once they hear this, I'm shuffled into an office for a long chat, where they say they're aware of, and working towards improvement, and it just so happens the software developers are always the last department to see these changes filtering in from "above".

Once I suggested they go and visit a place I previously contracted at, as they were actually a supplier - making some excuse about a project they could work on jointly, and learn the way they develop, thus, bringing their style of project management in house.
I ended up with a previous work colleague contacting me asking who the hell xxxx was, and so it seemed they may actually of listened to me. Whether that brought about positive change, I don't know. But the moral of the story is, managers love to talk, and pore over financial figures, but don't hear you when you talk about technical debt, need for refactoring, or consolidation, or offer new ways to do things that could prove beneficial. They only hear "risk". - and managing that doesn't seem appealing when they don't know that you're talking about.

Re:First Post with good info

[scamper_22]

Yep, I had a very interesting talk with my wife's friend.
She's an accountant and she was just complaining about IT.

So I talked to her about our side, and it's amazing the disconnect there.

We talked and I mentioned how every IT project needs a maintenance budget. You need knowledge retention in case the service needs to be updated in the future. You need someone to support it... You know, just like any other project. You wouldn't build a washroom and budget someone to clean it.

So she asked, well why don't you include that in your estimates? So we can have proper costing for the project.

Then I thought about it. There is a huge gap in accounting and IT. In IT, we often view accounting as something in the way. Stupid time tracking software. Bah, I'll just charge all my time to my assigned project. I have to provide an estimate. I'll just provide an estimate for the development. It's not my job to think about knowledge maintenance or operations...

Now of course the average IT person should not think about this, but the upper IT folks definitely should by setting up the right groups and proper leadership on how projects are structured. It's their job to get the funding so to speak.

It's not so much a problem that all the heads want to know is budget, on time...
It's largely that IT accounting is very poor to not account for all the costs. And we're so used to delivering something even if it does not account for maintenance that they get used to it.

Information-poor article

[Jesrad]

I read the linked article hoping for insight on how to identify redundant infrastructure, steer divergent IT departments towards common solutions, or at least practical examples of uniformisation / centralization of *something*, anything, but there were none. It's all just a tech-free summary of the guy's accomplishments, as you'd present in a management meeting to tout your success as an IT manager. That's good for him and for the ACS, I guess, but it's pretty pointless to post it here.

Re:Information-poor article

[antiperimetaparalogo]

It was an information-poor article ("/." summary has almost all the "info"!), but at least it was about IT...

Translation

[Rich0]

increasing spending on strategic projects from 5 percent to 40 percent

Rarely does this sort of change involve spending more money in absolute terms. Most likely people doing stuff like support were retasked, outsourced, or simply cut.

Most of the stuff in the summary sounds like a good move, but I've seen companies that were fairly short-sighted in making these kinds of moves. At work we've been focusing on "strategic projects" for a long time now in belt-tightening mode and it seems like the biggest result is that every department under the sun has sprouted its own mini IT department that does all the stuff that IT stopped doing, usually less efficiently than it would be done if centralized.

When IT cuts a service that really was necessary, the result usually involves a net loss of money.

So how did it save them?

[Anonymous Coward]

one global entity ... consolidating dozens of data centers into three ... increasing spending ... reducing 600 core systems down to fewer than 200

As far as I can tell he created a single point of failure, reduced diversity and thus resilience, and instead of getting a cost advantage, it cost more. Reminds me of a joke:

A shepherd was herding his flock in a remote pasture when suddenly a brand-new BMW advanced out of the dust cloud towards him. The driver, a young man in a Broni suit, Gucci shoes, Ray Ban sunglasses and YSL tie, leaned out the window and asked the shepherd... "If I tell you exactly how many sheep you have in your flock, will you give me one?" The shepherd looked at the man, obviously a yuppie, then looked at his peacefully grazing flock and calmly answered "sure".

The yuppie parked his car, whipped out his IBM ThinkPad and connected it to a cell phone, then he surfed to a NASA page on the internet where he called up a GPS satellite navigation system, scanned the area, and then opened up a database and an Excel spreadsheet with complex formulas. He sent an email on his Blackberry and, after a few minutes, received a response. Finally, he prints out a 130-page report on his miniaturized printer then turns to the shepherd and says, "You have exactly 1586 sheep. "That is correct; take one of the sheep." said the shepherd. He watches the young man select one of the animals and bundle it into his car.

Then the shepherd says: "If I can tell you exactly what your business is, will you give me back my animal?", "OK, why not." answered the young man. "Clearly, you are a consultant." said the shepherd. "That's correct." says the yuppie, "but how did you guess that?" "No guessing required." answers the shepherd. "You turned up here although nobody called you. You want to get paid for an answer I already knew, to a question I never asked, and you don't know crap about my business...... Now give me back my dog."

Re:So how did it save them?

[SABME]

There was a man flying in a balloon. It was a beautiful day, but the wind came up and blew the balloon off course, and soon the man was hopelessly lost. Eventually, he let some of the air out of the balloon, and drifted down until he was floating low over a field, where he saw another man walking his dog.

The man in the balloon called to the man in the field and said, "Hello! Can you tell me where I am?"

The man in the field said, "You're in a balloon, floating about 40 feet above a field."

"You must be a developer," said the man in the balloon.

"Yes, I am. How did you know?" said the man in the field.

"Because what you've told me is technically accurate, but it's of no use whatsoever," said the man in the balloon.

"You must be a manager," said the main in the field.

"Yes, I am. How did you know?" said the man in the balloon.

"Because you don't know where you are, you don't know how you got here, and you don't know how to get out of the situation you're in, but somehow it's my fault," said the man in the field.

Re:So how did it save them?

[Anonymous Coward]

As far as I can tell he created a single point of failure, reduced diversity and thus resilience, and instead of getting a cost advantage, it cost more.

And how did you determine that? 200 servers, 3 datacenters, if you can consolidate all your stuff onto 66 servers in each datacenter, say 22 clusters of 3 with global routing (3DNS or something similar) so every datacenter is "live" with a portion of the traffic, clustered DBs between datacenters, etc, you could have excellent redundancy. And if you replaced 600 older servers with new big honkin servers you could have even increased the capacity.

Seems reasonable...

[fuzzyfuzzyfungus]

One would hope that the American Cancer Society would, at least, be an organization that understands that uncontrolled proliferation can be seriously detrimental to an organization; and that sometimes substantial resection, however unpleasant and expensive, is the best available course of action.

It's a lucky coincidence that that applies to IT systems as well!

Re:Seems reasonable...

[DigiShaman]

And on to the next topic. You nailed it!

Manager - Systems Architect - Project Manager

[jellomizer]

The problem is there is a hierarchy structure while for IT projects the Manager, Systems Architect and Project Manager really need to be on the same level, and not try to take over each other's roles.

The Manager: On Time, Manpower, and Budget is a lot more work than it sounds, doing such work means you cannot really focus Technical Details, even if you take a good tech and give him a manager job, he will either make bad tech decisions, xor fail to manage effectively. The same with having the manager trying to be a PM or a SA and vice versa.
The PM is track of the state of the project, handling time lines, Insuring dependencies are met.
The SA deals with the technical decisions, how things should work, and if there are road blocks they come up with alternate solutions, or work with the PM to figure out new dependencies or adjust time lines.
The Manager, looks the the PM Time line and ensures resources are available to work on it, and that they are doing it the way the SA needs it to be done.