Ask Slashdot: Herding Cats, Aging Systems?

An anonymous reader writes: I've recently started a job at a medium-sized enterprise in the UK. They claimed to be an advocate of open-source. The job was advertised as a Linux sys-admin. I've been in the role a short while and the systems right across the business are end-of-life: lots of XP and 2003 servers, a handful of LAMP web servers, and a large IT department with almost no skills in the technologies on site. Most boxes have the default password still. As a senior techie, I've been tasked with helping bring the skillset of the rest of the staff up. Where would you start, given that most of the kit is EoL?

Comments

Don't train them in the current systems

[Chris Mattern]

That's the most obvious thing. Bring in supported systems and train them in those systems as you deploy them.

Re:Don't train them in the current systems

[Archangel Michael]

Before you bring in supported systems, you have to have a budget. Without a budget delineated, the rest of the decision making process is pure insanity.

My first response is, estimate what the "golden" cost will be, and quadruple it. They will cut it in half, and it will cost you twice what you think it will, and you'll end up with an excellent system that is designed well and built right.

If you need "enterprise" grade systems, make sure that you are identifying the vendors in the space and calculate budget accordingly. And remember, vendors lie.

Go Virtual

[BDMcGrew]

Well, your question leaves out a lot of details but from what you've said so far, look at getting some new hardware in there and start virtualizing some of the the EoL systems. This will provide you an upgrade path for existing systems and a snapshot'd point of restore in the event of a failure.

Re:Go Virtual

[ShanghaiBill]

Well, your question leaves out a lot of details

The most important left out details are about politics, not technology. Do you have the support of top management? How powerful are the people that are opposed to your project? There are people that will actively work to sabotage your efforts, and use you as a scapegoat for everything that goes wrong. How are you planning to deal with that?

Since you are the "new guy" trying to change things that you don't understand, you didn't even mention end-user applications, and you seem to be more interested in OSS-evangelism than supporting your users and helping them get their job done, my prediction is that you are going to be out of a job in less than six months.

Running?

[gstoddart]

As a senior techie, I've been tasked with helping bring the skillset of the rest of the staff up. Where would you start, given that most of the kit is EoL?

Well, you have 3 main choices:

1) Try to fix it and succeed
2) Try to fix it and fail
3) Run like hell

You won't be able to force the rest of the staff to bring up their skillset. Management has clearly left it to rot on the vine for a very long time. And, by the sounds of it, they don't know what they've even got.

A large IT department with no skills with the technologies on site? What exactly is that large IT department doing for this company? If you have a bunch of people with no skillsets with the technology they have ... then what skillsets do they have, and how is it helping you?

Without more detail, I'm hearing "Hi, I've just joined a company with a terrible IT department, how do I fix that?" Who let it get into such a bad state? Because if they're still around, no way in hell you'll ever fix it.

Re:Running?

[TWX]

Yep. If you're not in-charge and able to make the tough calls (ie, figuring out who's actually supporting important stuff, who's not, and making the decisions about who gets a chance to migrate to something new and who needs to take their skillset elsewhere) then you're probably not going to make the difference that you want to make or that your superiors somehow expect.

What I can say, from experience, is that you need to actually learn how things are working now before you start making changes. I've had bosses brought in from the outside that thought they were gods' gift to the IT world that decided to try to remake the organization in their own image, only be be fired less than a year later because they pissed off all of the existing IT staff such that the boss got no results, and pissed off the users by failing to maintain existing workflow such that the users' jobs became much harder or required lots of direct assistance.

Learn what's there, why it's there, and understand that most decisions were made as a reaction to something prompting it to be necessary. Change what can be changed in a sane way, but don't take personal offense to anything as it is now as there are probably good reasons why it is the way it is. If you come in with the attitude that you can rip out everything without a care, you'll find suddenly that no staff will bother to warn you of the pitfalls in front of you that they're all well aware of, and you, not them, will be the one with egg on your face when it breaks because it was your decision to change it.

Re:Running?

[TWX]

The article submitter made it clear that he's new. He very well may not understand the workflow and who actually knows how to take care of what. He needs to learn that before he can start making changes, or he, not the existing staff, will be the one blamed when everything goes wrong.

IT attracts a fair amount of introverts. It's likely that a lot of his staff are playing their cards close to their chest because that's what they're simply used to doing. It's also possible that they themselves wanted to make changes but were not given the budget needed to do so, so legacy systems continue to be used. It could also be that a few incompetent people in key positions have gummed-up the whole works.

Do you think that anyone wants to be stuck with ancient garbage if there's something newer that actually demonstrably works better? Most of the time the decisions that hold back the IT department are made either by IT management or by those outside of the IT department.

Re:Running?

[FranTaylor]

This sounds like a highly dysfunctional environment

Mechanical-type people are usually pretty horrified by the short lifespans of computers. They are used to dealing with things like turret lathes and drill presses that can handle 50 years of continuous use. It could be a perfectly natural reaction.

Clean it up

[Anonymous Coward]

Kill everyone. Set fire to the place. Plead insanity. When they see what you were supposed to work with, they'll believe you.

A plan and boss buy-in

[i.r.id10t]

Make a map of what you have, what the main issues are with each piece, and then a plan for replacement/updating/whatever. Try to include some rough (and higher than you really think it will be) cost estimates. Then present to a boss, and get buy-in. If you don't get buy-in, start updating your CV and look for another job.

Low Hanging Fruit

[AdelieMan]

I would audit everything, Make a matrix of things that need to be addressed easy to hard, least significant to most, and start chipping away at it. It will take time to turn that ship around, but it will be worth it, and you will keep your sanity.

Re:Low Hanging Fruit

[bluefoxlucid]

Hear hear. I would suggest not being shy of technology; I've been interested in Microsoft Project 365 integration with Sharepoint for a while, and you should definitely look at your options for project management whether they come from Microsoft, Oracle, or some no-name company that provides a fantastic and little-known product as an open-source support-contracted service. What you have there is a long program, and I suggest you get RMCProject's CAPM Exam Prep and the PMBOK if you haven't got project management skills, and spend the 3 months getting a basic grasp of all that right out of the gate.

The primary tools you're going to want are risk management and hierarchical decomposition; however, on the scale you're talking about, full project management knowledge is going to be an outright requirement if you want to do anything resembling a competent job. You *won't* want to use the full suite of project management practices--you never want to use the full set of tools outright, but rather the ones you want, for any purpose in any field--but if that place is as big a rat hole as you say, you're going to need some accounting of what's going on.

As the parent poster here says, you definitely need to start here:

Make a matrix of things that need to be addressed easy to hard, least significant to most, and start chipping away at it.

Get a list of discrete, finite, deliverable projects. Things you can put into boxes and say, "This is one thing I want to produce; it's of a nature that I can tell you what work is required, how much time it will take, and what it will cost." You'll start by examining the array of systems, breaking them down into departments and components (what do they support? What do they do for each department?), and deciding what you're replacing. Are you upgrading Windows XP with stitched-together software to Windows Server 2008, or are you transitioning to a new set of systems to solve the same problem in a different way? Get that list down.

Each thing you want to address will be something small, finite, limited, and understood. You're replacing the groupware services--Exchange, for example; the thing that provides e-mail, calendar, and such--with an upgraded, better-implemented, or new product (exchange to Zimbra, Zimbra to Exchange, migration to a SaaS such as Google for Business or Office 365, etc.). Some things break out into phases or multiple projects, e.g.: migrating Exchange to Office 365 may involve a phase 1 of upgrading Exchange to the latest version, a phase 2 of enabling some kind of synchronization and backup that you don't have now, and a phase 3 of migrating to service; while you may find that your Zimbra installation has no back-ups because you need an enterprise backup solution, and so you can't get back-ups in until you get Bacula set up.

Once you have your list, you can start breaking them out by hierarchical decomposition. You'll want to decompose the work: each deliverable (e.g. your project, Bacula backup infrastructure, delivers a working Bacula backup infrastructure as its product) breaks out into a complete set of deliverables (e.g. project management, support services, back-up strategy design, servers, client deployment with Puppet or SCCM or Ansible, etc.), which themselves each break down further. Once your work is broken down, you hit the bottom with sets of work packages--each a deliverable--that you can understand completely; you can turn those into lists of activities and tasks to produce the deliverable.

The same goes for risks. You want to identify everything your experience says can go wrong, and use your experience to do qualitative risk analysis--what risks are important? Then you use a procedure of assessing probability vs severity to do quantitative risk analysis. You work out how to avoid (100%), mitigate (any%), accept (0%), or transfer (buy insurance) the negative risks (threats), and how to exploit (100%), en

Re:Olut with the old, in with the new

[bobbied]

Buy a new system. Power down every system in turn and try to power it up again. If it will not start, replace it.

NEVER power down old hardware on purpose unless you have backup plan for the system... Old hardware has a habit of not coming back when you power off and if it dies, you created an emergency for yourself...

There are going to be enough unforced errors in the process, you needn't go out and look to create them.

Lead, Mentor, Grow

[mtippett]

You've been dropped in an environment that is legacy and probably has production problems. Use that to your advantage.

You've been also dropped in a leadership role (not management, leadership).

Your #1 target should be to make yourself redundant (which ironically is likely to get you promoted, it's called succession :).

So look at doing something like identifying #1 problem (Pareto charts help). Ask for volunteers (or volunteer some people), give them the problem to solve, use whiteboards, etc to help them discover the solution. You may facilitate and provide hints to get things done. Empower and guide the people you are helping.

Read up on https://en.wikipedia.org/wiki/..., you are likely in a #2 or #3 combination. You can help lead people to move to a #3 with leadership, with the idea to get to #1 over time (with their help).

Of course there might be some issues that you might need to solve like EOL systems and any budget that may be needed. If the OS is old, then probably the HW is old as well. Budget for that is probably going to be your biggest issue.

Wanted:

[Drewdad]

Wanted: IT Director
Pay-scale: Entry level.

start

[JohnVanVliet]

-- quote--
  Where would you start,....
----------

with the thermonuclear option !

with DEFAULT passwords of "password"
and using XP and MS 2003

the use of DBAN has been authorized

Cheapskates

[scsirob]

They are not open source advocates, they are cheapskates who like the prospect of 'free' anything. No supported equipment, no updates, no training for their staff, they simply don't appreciate the value of their IT.

Let me guess, no decent backups either? No DR plan? Nothing of the sort? If you want to stay there, demand a decent budget ( = commitment) and build greenfield. If you don't get a decent budget, run.

A fire?

[onkelonkel]

Seriously, "accidentally" toss a lighted cigarette into the paper recycling bin in the server room on your way out one night. You'll be able to start fresh with the insurance money.

Make the separate firewall works?

[Anonymous Coward]

Make sure that the separate firewall works, then go from there. Were your bosses thinking that a Linux admin was a Windows admin with extra skills, that the Windows skills came automatically with the Linux skills?

Don't beat up on the geezers there for having stale skills. They might actually be OK at keeping those obsolete systems running. Some of them might be OK at getting a new system running, unless they're stuck in their ways.