Ask Slashdot: Getting a Grip On an Inherited IT Mess?

First time accepted submitter bushx writes "A little over a month ago, I assumed the position of programmer and sole IT personnel at a thriving e-commerce company. All the documentation I have is of my own creation, as I've spent most of my time reverse-engineering the systems in place just so I can understand how everything works together. Since I've started, I've done everything from network and phone upgrades to database maintenance with Perl, and thus far it's been immensely rewarding. But as I dig deeper, I notice the alarming number of band-aids applied by my predecessor, and it seems like the entire company's infrastructure is just a few problems away from a total meltdown. The big question now is, how do I, as a single person, effectively audit the network, servers, databases, backups, and formulate a long-term plan that can be implemented by one person? Is it possible? Where do I begin?"

Comments

methodically and late into the night

[sentimental.bryan]

say goodbye to your life for the next year. hope you're getting paid to mislay it....

Re:methodically and late into the night

[DrgnDancer]

My guess is he's not... I'm immediately concerned by: "position of programmer and sole IT personnel" and: "thriving e-commerce company" together in the same sentence. The fact that this does not appear to be a small mom and pop with two or three servers making up the "e-presence" adds fuel to the fire. I'm getting the image of a fairly large company that relies heavily on it's web and e-commerce presence. And has one guy to take care of that. What happens when he's on vacation or sick and a server dies? What happens when the website has an issue and then *anything* else goes wrong?

There's no bullpen here, if anything, anything at all, breaks there's only one guy to fix it. Day or night. If two things break you're already triaging. Surely a "thriving" company can afford a backup to what is pretty clearly a business critical unit?

Re:methodically and late into the night

[Lumpy]

"if they were half-way intelligent, you'll find an external email address in some source somewhere. Ping them and ask what really happened."

I have done this before.. the response was...

"Find another job and run, run as fast as you can. Oh and trust no one."

Re:methodically and late into the night

[gmack]

That's assuming the predecessor wasn't the problem. I have learned over the years that there are far too many tech types to prefer to be the only one that does a particular task and will make any excuse to management to make sure things stay that way. When these lone wolf types happen to not be as competent as they pretend to be they tend to themselves into too deep a hole so they either get fired or quit in frustration but when you talk to them it will always be some other person's fault.

I'm not saying management isn't at fault, they very well could be but don't assume that right off. The first step is to try and get a read on how good the predecessor was at their job otherwise he can get very misleading info.

Re:methodically and late into the night

[xdroop]

What happens when he's on vacation or sick and a server dies? What happens when the website has an issue and then *anything* else goes wrong?

Oh, that's easy:

• He gets called in from being on vacation or sick;
• he gets to work uncompensated time to fix the problem;
• if he fails to either respond to the call OR fails to fix the problem, he gets fired;
• if he succeeds in fixing the problem, he gets threatened with termination should something else fail while he's "unavailable".

In fact, I'd lay odds that's how the vacancy occurred.

Re:methodically and late into the night

[Anonymous Coward]

Small operations like his are common. I'd guess he is a reasonably capable person. Where his world and yours differ, is that he's a jack of all trades (master of none)... because that's what that kind of business requires.

Yes, in a larger company, you'd hire an Exchange pro, an AD pro, a networking pro, a programmer or two and a couple techs that are slightly more generalized guys to manage backups, the server room and help desk. The unfortunate truth is that specialized individuals are rarely any good outside their specialty... which is unhelpful to a small business that can't afford a stable full of tech talent.

I know, as I've been this guy. It's brutal work but can be pretty satisfying. Every day your work is different. But you're never an expert at one particular thing and you're never paid like someone who specialized early on.

Escalate

[sociocapitalist]

Brief your management on the situation. Explain what condition things are in and what is needed to get them into a manageable state. Give them a list of projects / tasks that you have to deal with and get them to prioritize.

Get management buy in...

[Lumpy]

You need to document it and get management to approve spending money.

I'll bet you $100.00 the band-aids are there because management refuses to spend money on Infrastructure and its' why it is a mess and the guy there beforehand has left.

99% of the time a hosed IT infrastructure is because management refused to spend any money so it had to be half assed.

this is a majorly funny story

[roman_mir]

Facts:

1. The job has lasted for 1 month so far.
2. The e-commerce company is 'thriving' apparently'.
3. All of the systems have been "reverse engineered" in that 1 month.
4. All of the documents are written in that 1 month.
5. In 1 months there have been: network and phone upgrades and database maintenance with Perl and it all has been 'immensely rewarding'.
6. The entire infrastructure is 'a few problems away from a total meltdown'.
7. Single person IT operation to do everything.

Question: is this for real? What's the size of the company and what's the budget?

Re:this is a majorly funny story

[1u3hr]

Question: is this for real?

It's an "Ask Slashdot". They're as real as "Letters to Penthouse". Both carefully crafted to create a fantasy situation to excite readers. Read them if the subject is something you're interested in, but don't waste your time giving advice..

Re:Quit

[Anonymous Coward]

No!

This is actually the kind of career building stuff one should leap at. What would you rather say in an interview for your next job:
- I took this system that was falling apart and made it run like clockwork.. downtime and issue frequency went from "it's down again" to "been up all year" ..
- Yeah it was pretty good when I got there, and I maintained the status quo

My thoughts on original question:

First step is comprehension. You can’t fix what you don’t know you have/need. Identify the key components of your system. Then for each key component, break it down to it’s parts and dependencies. Then break each one of those out, and so on, until you have a pretty damn good idea of what you have.

Next part is assessment. For each component you’ve identified, what is its current state.

And then it’s time to do triage. Prioritize stuff by largest potential impact.

And finally carry out your well thought out pla.. ok, can't say that one with a straight face. Basically try to fix stuff when you can, between putting out the daily fires.

Start over... slowly

[rabenja]

I was in much the same position 12 years ago at this company. I am now CIO with 7 people on my team with several business partners to help manage the infrastructure. My advice for what it is worth:

• - Take time every day to assess and analyze the bigger picture before allowing yourself to get drawn into the details.
• - Look at the entire system from a risk mitigation perspective. What areas are most likely to cause "meltdown". Spend the most effort there.
• - What are incremental changes that can be made that improve the overall risk picture? Focus on the biggest bang for the buck.
• - Defer anything that works well enough for the time being.
• - Avoid big bang solutions unless they can be contained and tested well, with the capability of rolling back.
• - Get help where necessary.

Re:Quit

[Anonymous Coward]

I worked in this environment for one year as to not tarnish my resume. I toughed out the last 4 months absolutely burned out and bitter. You cannot communicate to management that outages and issues aren't your fault; they're adopted. When you fix things, you'll inevidably miss something (I did because of the pace, not dictated by me). Get out. It's not worth the challenge to get proper budgeting to get the right tools in place or the organization as a whole wouldn't let things get how they are in the first place. The business model I came from is failing. If you're good, there are better paying, better rewarding, less "heart and soul" companies out there. You're doing basically startup work for at will employment pay.

Re:Quit

[The Moof]

I'd amend that to a big "maybe" for sticking around.

All of what you said (and the initial reaction to quit in the GP) all hinges on the root cause of the mess. If it's a result of the predecessor not doing things correctly and flying by the seat of his pants, you're correct at jumping at the opportunity. However, if it's caused by management screwing IT every chance they get with poor timelines, lack of funding, no foresight, and so on, run like hell.

Re:Quit

[Archangel Michael]

It is probably a combination of the two. Because MGMT always assumes IT can do something with very little, and often the Impossible with Nothing.

We are skilled (most of us anyway) problem solvers, and they rely upon that to function. I hate to say it, but to the original question should be answered this way: HIRE outside consultants to evaluate your system(s), and give you a hard copy report on their findings that you can present to MGMT.

If the situation is as I believe, it is worse than he even suspects. He needs more help than he can do by himself, to get ahead of the curve.