Ask Slashdot: Getting a Grip On an Inherited IT Mess?

First time accepted submitter bushx writes "A little over a month ago, I assumed the position of programmer and sole IT personnel at a thriving e-commerce company. All the documentation I have is of my own creation, as I've spent most of my time reverse-engineering the systems in place just so I can understand how everything works together. Since I've started, I've done everything from network and phone upgrades to database maintenance with Perl, and thus far it's been immensely rewarding. But as I dig deeper, I notice the alarming number of band-aids applied by my predecessor, and it seems like the entire company's infrastructure is just a few problems away from a total meltdown. The big question now is, how do I, as a single person, effectively audit the network, servers, databases, backups, and formulate a long-term plan that can be implemented by one person? Is it possible? Where do I begin?"

Comments

Me too

[weave]

I walked into a similar nightmare two years ago. Before I even took the job I assessed the situation and gave them a proposal for what needed to be done and a price estimate for the software and hardware. I told them I would not take the job unless they committed funds to support the function. I also warned them that there were numerous ticking time bombs and I'll defuse them as fast as possible but there was no magic fix and it would take some time and they could have a disaster still

I then convinced them to only hire me part-time and to also hire a part-time desktop support person for a few reasons including they don't want to pay me to do that and having two IT people at least gives you some continuity. Even if the desktop support guy doesn't know the high-end stuff, if I leave the desktop person can still guide the new person and save them a lot of time I never got.

My line of attack was:

1. Back up data. Wasn't easy. They had old cart tape drive units that were problematic. I ended up getting cheap TB externals to at least make mirrored copies of things. But at least if there was a disaster, I'd have their data safe somewhere -- even if it took me weeks to reconstruct systems to use it.
2. Secure data. Everything was wide open. All domain users WERE DOMAIN ADMINISTRATORS. Locking that down was a pain. An understanding of what would be impacted ahead of time would have taken months, so I didn't tell anyone what access they had, then started removing people from domain admins a few at a time and waited to hear what broke, then fixed access issues. Not user friendly, but getting that under control fast was necessary.
3. Renovated room with servers in it (that were 5+ year old deskside servers) so as to accommodate a rack with proper A/C flow, electrical feed, and physical security.
4. Had them throw ~$50k into a virtual infrastructure and SAN, then virtualized all their old deskside servers until I could migrate apps on them to fresh OS installs. Used Vspehere's DRS product to back up the OS images and data to another system I had them buy for their other site (thankfully not too far away and connected by fiber)
5. Identifying all in-house written programs and finding turnkey solutions to them, preferably cloud-based to reduce their dependency on in-house IT staff in future.
6. Documenting everything as best I can as I go.

Getting back to original point, a one-person IT shop is suicide. Them having a two person part-time crew is better because if one leaves, at least the other can provide some sort of continuity -- and that happened already. The fairly young guy I hired for desktop support two years ago died last month :-(

In this situation currently.

[YojimboJango]

The number one best thing you can ever do in your situation is ask your bosses what they think the system should be doing.

Step 1: All the squirrelly business logic and the rationale behind each system you have to maintain should have a plain text description. You have to know the 'Why' before the mess of band aids that is the 'How' will ever make sense. Have your boss (or his secretary, or whoever) document it and get it to you. Do NOT do this step yourself. Repeat do NOT perform this step.

Step 2: Put out fires till someone not you finishes step 1. Start making backups of every last scrap of data you can get your grubby hands on.

Step 3: Once step 1 is done compare it to the mess. Note where the realities that are in your bosses head diverge from what is actually happening. Your job is to now create a detailed functional spec that takes what your boss says, and expand on it with what is really happening. Try to include worst case scenarios and document them as intended features.

Step 4: Have your boss and sales and marketing, and every other top level manager sign off on it. This will not happen. No two managers in your company will fully agree on what the current system is actually doing. Your goal is to figure out what sales and marketing are telling your users that your products do. Do not disregard this step or it will come back and bite you very hard.

Step 5: Once every department actually agrees on what your job really is, you will be well equipped to start the long process of fixing things. Again make lots and lots of backups. Management will sign off on step 4, then you'll fix a gaping security hole, and some customer somewhere will throw a raging fit because sales promised that they'd be able to get admin access to your databases or something ridiculous.

Step 6: Don't be an ass. When step 5 inevitably happens, explain the miss-step in communication graciously, and roll back. If you pulled not being an ass off properly, you now have a great platform to explain to management why X was a bad idea, and present an idea to fix it.

I'm a grizzled vet to your situation. If someone would've told me what I just told you when I started out, there would have been a lot less headache and stress. Hang in there, it can be an intensely rewarding experience.

This gig has writings on the wall all over

[luis_a_espinal]

My backup was my boss who was technically competent, so there was that, but it's not like I've never worked a job as a one man show. You buckle down and do what you have to and make it so things don't break just because you're not around (yes, this requires budget, but I've been fortunate enough that anyone willing to pay me what I'm worth is also willing to invest in a solid infrastructure).

This made you (and the situation you described) an outlier, one with a positive outcome. Your experiences cannot be applied in general. In general, this is rarely the case, for one-man-tech shops that is.

For the most part, conditions as described by the original submitter typically have "GTFO ASAP!" written all over it. I've done IT in companies, small and large, and I can attest that what you say is true: Yes, it is possible to being the one-guy-IT-slash-programmer-shop at a small e-commerce company. But the question is why? I wouldn't do it (again) unless a good compensation package came with it (which is typically never the case), or if I'm fresh out of school with nothing on my plate to take (in which case, it is ok.)

Good companies are never based on one-man-IT-slash-dev-shops, regardless of size (or at least they try not to.) I know, again, I've worked with companies big and small. Conditions like that are typically good proxies for more systemic problems, and at the end of the day (whenever possible), you want a paycheck, a rewarding job and good working conditions. Rarely you see that with one-man-IT-slash-dev-shop gigs, rarely if ever, regardless of the size of the company.

That's just my $0.02 input from what I've seen. YMMV so readers be warned and please take this anecdotal piece with a grain of salt.

Re:methodically and late into the night

[Jane Q. Public]

Absolutely. I once found myself working on a web project that had been through 3 previous developers -- and it wasn't even that big of a project -- but of course I did not know that when I took the job. If I had known the history of the project, I probably would not have taken it.

I ended up trying to reverse-engineer a huge mess, without really being given the time to do so. They kept me busy making stupid little changes to the graphics, when it really needed some serious underlying code work.

Then, out of the blue, they sprung a deadline on me of like 4 days, AND they wanted to release on a holiday. I said "No way. I would need at least another week to get this working properly." I did not get the week. PLUS they kept making changes up until literally the last hour, PLUS guess who got blamed when things -- inevitably -- did not work right?

I was glad to get the hell out of there. As -- it turned out later -- were the 3 developers before me.