Slashdot Mirror
Ask Slashdot: Herding Cats, Aging Systems?
An anonymous reader writes: I've recently started a job at a medium-sized enterprise in the UK. They claimed to be an advocate of open-source. The job was advertised as a Linux sys-admin. I've been in the role a short while and the systems right across the business are end-of-life: lots of XP and 2003 servers, a handful of LAMP web servers, and a large IT department with almost no skills in the technologies on site. Most boxes have the default password still. As a senior techie, I've been tasked with helping bring the skillset of the rest of the staff up. Where would you start, given that most of the kit is EoL?
That's the most obvious thing. Bring in supported systems and train them in those systems as you deploy them.
Well, your question leaves out a lot of details but from what you've said so far, look at getting some new hardware in there and start virtualizing some of the the EoL systems. This will provide you an upgrade path for existing systems and a snapshot'd point of restore in the event of a failure.
No guns, no knives... do you pussies still get rope or are you going to have to find a tall building to jump off instead?
Get the finance dept in your camp before doing anything.
Bribe them with candy. Twizzlers.
You need to start a training program. If the current workers feel uncomfortable in the new technologies, the will oppose you every step of the way (though they won't say why). If they feel comfortable in the new technologies, they will push you faster in the adoption.
If a team has been working on a lousy codebase, your first priority should be to teach them to do better. You can try cleaning things up, but they will make messes faster than you can clean. You need to find a way to teach them to at least not make things worse.
"First they came for the slanderers and i said nothing."
Honestly, just run for your life. Herding cats has never been an easy task, especially if they are sensing danger (EOL)
I am in this same boat right now. Looking for a shore I can jump to.
BAIL!!!!
I don't know your organization's level of risk tolerance, but getting them to pay for one of the following would be an eye-opener:
- A vulnerability assessment will show a sea of red for the unsupported platforms. Maybe that'll be sufficient to convince them that it's time to upgrade (and train up on new stuff).
- A penetration test will take those same vulnerabilities, and combine it with attempting to use those vulnerabilities to see what they could get. The difference is in trying to use those issues, and turn them into "oh SHIT" screen shots in the report. It's the difference between "someone could theoretically do X" and "someone just did X, and documented it all for your edification."
On the latter engagements, especially with the dreadfully old stuff, it is quite enlightening to include those screen shots that show how I've added new users, logged in with them, and used them to poke yet more systems I couldn't reach from the starting point. The under-educated staff would only help things if social engineering was in scope too.
Well, you have 3 main choices:
1) Try to fix it and succeed
2) Try to fix it and fail
3) Run like hell
You won't be able to force the rest of the staff to bring up their skillset. Management has clearly left it to rot on the vine for a very long time. And, by the sounds of it, they don't know what they've even got.
A large IT department with no skills with the technologies on site? What exactly is that large IT department doing for this company? If you have a bunch of people with no skillsets with the technology they have ... then what skillsets do they have, and how is it helping you?
Without more detail, I'm hearing "Hi, I've just joined a company with a terrible IT department, how do I fix that?" Who let it get into such a bad state? Because if they're still around, no way in hell you'll ever fix it.
Lost at C:>. Found at C.
Seriously. If this is how they're running their operation to this day, chances are its not just harmless, easily washed-away naivety that is keeping everything so poorly organized and insecure. Chances are you're going to find this out the hard way though, and they'll mysteriously get instantaneously far less apparently incompetent when it comes to finding ways to get you fired first.
Let Milton come in & have his way w/the building
I know it sounds obvious, but I would simply do what they are asking you to do - educate users, etc.
You could mention the fact that their equipment/software is obsolete. From my experience, I would probably mention it but not push it.
The reason is that most companies have their own way of doing things. Even if you have a better way of doing it (and you probably do!), there is nothing wrong with making suggestions. But pushing your suggestions onto management (or whoever) - even if you mean well - won't necessarily be well received.
Most people are afraid of change; and hounding the company to "get up to date" (and spending more money) might not be very well received - especially if it's from the new guy.
That's just my $.02
Political correctness is really just herd psychology pushed by insecure people who desperately seek social conformity.
Buy a new system. Power down every system in turn and try to power it up again. If it will not start, replace it.
Unless you are the IT director and can start firing people en masse, I'd simply start looking for a new job. Even if you are able to clean the digital Augean stables, your fellow IT co-workers are likely to resent you for making them look bad and begin plotting your demise.
Been there, done that.
a large IT department with almost no skills in the technologies on site
No skills? Then what is it that they do all day? I suspect that your IT department has been the dumping ground for employees that can't just be gotten rid of. In other words; politically connected. I've been there and tried to deal with that. And in my opinion, it can't be done by someone without serious seniority.
You could try introducing training programs for the target architecture. And some of the motivated staff will avail themselves of this. But be prepared to run across a few people who refuse and insist on hanging on to their legacy Windows stuff.
Have gnu, will travel.
I'm not sure it is end of life. There are arguments that XP systems will be around at least another decade. The $50billion/year semiconductor company that I used to work at has numerous production machines that are XP, isolated, and going to run forever. They are paying good money to make sure they have hardware and software to replace the system at a minutes notice.
Maybe end-of-life in a windows-10 world. Not actually end of life.
So, can you make the systems run another 10 years without requiring huge changes?
What is needed for a maintain - don't upgrade staff? Hot-swapping a drive on a system, and some good VMWARE instances might be a decent start.
Hmm, if there are legacies, i would advise the following.
1. virtualize whatever you can.
2. remove the oses which are no longer supported by the vendor MSFT...2003/xp..because of security issues.
3. get a list from business of all the critical system. There will need to create a roadmap at this stage for migration.
4. in the mean time. start working / training the team on open source, specifically for (1) as a virtual platform.
5. start planning (3), by training people in the direction to be...
Kill everyone. Set fire to the place. Plead insanity. When they see what you were supposed to work with, they'll believe you.
Make a map of what you have, what the main issues are with each piece, and then a plan for replacement/updating/whatever. Try to include some rough (and higher than you really think it will be) cost estimates. Then present to a boss, and get buy-in. If you don't get buy-in, start updating your CV and look for another job.
Don't blame me, I voted for Kodos
https://www.youtube.com/watch?v=Pk7yqlTMvp8
Lots of it!!!
It depends on how much actual authority you have, how conservative the corporate culture is, and whether there are any entrenched ways of doing things. This isn't a technical question but a political one. If you actually (as opposed to officially) have authority to tell them how to do things you need first find out how the system is working now. Maybe they didn't set up passwords because multiple departments need to connect to the same server and there's no secure password control in place. Maybe they're disorganized. Maybe they're inexperienced. These all require different activities to repair the problem.
You mentioned EOL hardware, but you didn't say whether a migration is planned or whether the money is available for one. Obviously new hardware is a great opportunity for user training, but again there are too many unknowns here. How much extra time do the engineers have to train? How much of the existing system setup is invisibly a part of how the users interact with it?
It sound to me like you're standing on a powder keg. The right way to deal with it is to gather information. Make benchmarks. Understand system inter-operations and use. Learn who is doing what and why. Only a fool would start declaring X and Y need to be done without taking a look around first.
If video games influenced behavior the Pac Man generation would be eating pills and running away from their problems.
You're going to be looking in a few months anyway.
I would audit everything, Make a matrix of things that need to be addressed easy to hard, least significant to most, and start chipping away at it. It will take time to turn that ship around, but it will be worth it, and you will keep your sanity.
I think that bot from a few articles down is trying to weigh-in...
Do not look into laser with remaining eye.
Although there's big money in cleaning up someone else's mess, you got to recognize a hopeless situation when you see one. Fire everyone and bring in a professional IT team to take over the operations. Or run like hell and hope that the next job isn't as bad or worse as this one.
You've been dropped in an environment that is legacy and probably has production problems. Use that to your advantage.
You've been also dropped in a leadership role (not management, leadership).
Your #1 target should be to make yourself redundant (which ironically is likely to get you promoted, it's called succession :).
So look at doing something like identifying #1 problem (Pareto charts help). Ask for volunteers (or volunteer some people), give them the problem to solve, use whiteboards, etc to help them discover the solution. You may facilitate and provide hints to get things done. Empower and guide the people you are helping.
Read up on https://en.wikipedia.org/wiki/..., you are likely in a #2 or #3 combination. You can help lead people to move to a #3 with leadership, with the idea to get to #1 over time (with their help).
Of course there might be some issues that you might need to solve like EOL systems and any budget that may be needed. If the OS is old, then probably the HW is old as well. Budget for that is probably going to be your biggest issue.
Wanted: IT Director
Pay-scale: Entry level.
1) Start with the printer by printing copies of YOUR resume
2) Distribute broadly
3) Get the hell out before their stench rubs off on you
Seriously mate, you CANNOT fix this. They will drag you down to their level.
If the powers that be allowed such a situation to exist, and didn't specifically hire you to change it, along with a guaranteed budget with which to accomplish the task, then it will be almost impossible for you to fix it. The fact that they essentially lied to you about your role does not bode well.
If this is a resume building job for you then dink around the edges on things that won't require much, or any, money or many changes to the status quo. Make big talk about how you are improving things. Take every opportunity to educate YOURSELF on things you can use later in your career. Put in your year, then move on. It doesn't matter what you ACTUALLY accomplish in an environment like that. Only what you can spin things in to for your resume. Cynical, I know, but you gotta think about yourself first. This is a dead end job.
And you had no idea at interview? Did you ask questions? If you did, and if they told the truth, what WAS your intended strategy?
Does anybody here know how to get shit for free?
Just tell this guy to find a torrent for one of those *IT For Dummies* books.
And he got hired as "chief"?? Oh Lordy! Chief patsy, maybe.. Better cover his ass real good.
Getting out quick is great advice. However, if that's not a realistic option, then level with management that it's a big job, and ask for a priority list from them.
Make a list of things to be done, along with the reason or consequences of not doing them, give your suggested priorities (A, B, C, D, etc.) for each task, and ask management to confirm or reassign priorities.
Often they won't want to assign priorities because that commits them to their decisions. Managers like wiggle room to blame others.
To get around that tendency, state something like, "I have assigned default priorities to the given tasks. I shall use these priorities as a my default working assumption unless and until explicitly given new priorities. I invite your feedback and would be happy to answer any questions about them."
They probably won't like that pressure and explicitness, but better to take your lumps up front rather than have an even bigger meltdown in the future.
And some may even appreciate your documenting of tasks. It might help them justify getting you some help. But that's a secondary reason for the list.
Table-ized A.I.
-- quote--
Where would you start,....
----------
with the thermonuclear option !
with DEFAULT passwords of "password"
and using XP and MS 2003
the use of DBAN has been authorized
"I don't pitch OpenSUSE Linux to my friends, i let Microsoft do it for me
EoL is false statement. If cpu is still working it is doing work.
The question is
how well is it doing work.
Is that work meaningful
Can it be done better
If it crashing daily it fails 1
If it not longer in step with company it fails 2
If it can be done better. Than look else were to fix first
>> large IT department with almost no skills in the technologies on site...As a senior techie, I've been tasked with helping bring the skillset of the rest of the staff up
Stop right there and understand that you've been sent on a wild goose chase. You're not really going to train your existing staff - ever. Instead, what you're doing is writing the job descriptions for the outsourced personnel that your management will hire to replace the deadwood in your "large IT department" (because it's no doubt costing them an arm and a leg, and personnel budget is sucking up the money that should be flowing to technology upgrades).
So, what should you do? I'd say complete the assignment with a smile on your face and a suggestion that some of the skills are already available elsewhere in the market. Hopefully management will pat you on the head and let you move on to the next level: interfacing with the outsourcing company that's waiting in the wings.
You were brought on as Linux SysAdmin; you now know that the job is nothing of the sort, and getting things up to speed will require massive investments in technology, personnel, and many sleepless nights on your part, should you choose to perform this task.
If you want to do this at all (and it sounds like you do), you need to demand a raise and quit if you don't get it.
They are not open source advocates, they are cheapskates who like the prospect of 'free' anything. No supported equipment, no updates, no training for their staff, they simply don't appreciate the value of their IT.
Let me guess, no decent backups either? No DR plan? Nothing of the sort? If you want to stay there, demand a decent budget ( = commitment) and build greenfield. If you don't get a decent budget, run.
To Terminate, or not to Terminate, that's the question - SCSIROB
End of Life? Dude, I'm running Cobol code that probably dates back to before you were born!
If it does the job that's one thing. If it's all patched and up to date security-wise, you are OK. If however, the systems are constantly running at 100%, and you've got latency and throughput issues, then that's a different story.
If nothing's getting backed up because the systems are under huge strain, then in that case you are welcome to start upgrading. In fact, you might be able to keep the same hardware and simply "upgrade" to Linux slowly, which is less resource-intensive than Windows.
And the way to do it is to build a new box that "mirrors" an existing one; once you've tested it to death, and you're confident nothing has been missed, you retire the original box, then use that box for the next build that mirrors another box and so on.
If telephones are outlawed, then only outlaws will have telephones.
Take the paychecks while you can, but you need to get the hell out of there.
The company obviously doesn't care about IT, and you almost certainly aren't going to change that. Let them rot.
Alternative viewpoint: Microsoft Windows XP "end of life": Conflict of interest
"April 8, 2014: Microsoft began charging millions for support of its Windows XP product. "
You've been hired to fix an IT department where:
-the technology is obsolete
-the workers don't know how to use it
-the people who caused this state of affairs are your superiors and coworkers
-they probably aren't going to fire themselves or let you fire them
You need to leave asap. These people obviously have no incentive to improve and they're only going to become hostile once you start trying to make changes.
Look at it from the outside.
Hardware => Old
Software => Old
Crew => Untrained
Backup the data, get insurance and burn the place down.
Fire the staff since they have no purpose now. Use the money from the insurance (a pittance probably) and the funds that would've went into rebuilding the whole system and start from scratch.
Get new hardware and up to date software, the backups, get a new crew (or the old crew, they should've had enough time to brush up their own skills by then) and start working.
PS there are tests that show how well you evaluate risks, I suggest you try one.
Seriously, "accidentally" toss a lighted cigarette into the paper recycling bin in the server room on your way out one night. You'll be able to start fresh with the insurance money.
None of them can see the clouds; The polished wings don't care.
That job sounds like a shitshow. Get your resume in order and move on.
ok, saying PM will fix it is cheap but that's really the core task of project management "to devise a plan"
But a specific plan with goals needs to be based on an analysis of the environment you are actually in and you did the first steps, you tried to describe the environment.
medium-sized enterprise, UK
Ok, you're in the UK and have access to a solid workers pool of skilled IT, also available by contract workers.
Think about getting an aide, nothing is worse than a project manager that looses oversight!
They claimed to be an advocate of open-source.
But everything contradicts that, don't cling onto open source, don't waste energy to educate people that don't understand. It's useless, and open source is only one solution, that can work but also can fail. Especially if you have people with no skill!
The job was advertised as a Linux sys-admin. I've been in the role a short while and the systems right across the business are end-of-life: lots of XP and 2003 servers, a handful of LAMP web servers,
Do you have an overview, are all the systems accounted and cataloged with "Machine Power"/"Electric Power"/IP/MAC/Remoteinstall/config etc.. possiblities.
Because this actually looks like those who hired you did not have such an oversight.
and a large IT department with almost no skills in the technologies on site.
First you don't need a training plan you need a Maintainance&FIXIT plan = nothing is worse than systems failling, when you grab resources for training. because you will be blamed for, your oppinion will lose worth you will lose resources.
Idea: Have a fix-it task force at hand. Only selected few who really can "fix it".
(Think about high skilled contract workers.)
Most boxes have the default password still.
That could be getting fixed by a task force.
As a senior techie, I've been tasked with helping bring the skillset of the rest of the staff up.
Good luck with that.
Where would you start, given that most of the kit is EoL?
Make a plan based on the current equipment(what can still run and what is really dead) and the demand of the market. ("priorities")
fellow had just been hired as the new CEO of a large high tech corporation. The CEO who was stepping down met with him privately and presented him with three numbered envelopes. "Open these if you run up against a problem you don't think you can solve," he said.
Well, things went along pretty smoothly, but six months later, sales took a downturn and he was really catching a lot of heat. About at his wit's end, he remembered the envelopes. He went to his drawer and took out the first envelope. The message read, "Blame your predecessor."
The new CEO called a press conference and tactfully laid the blame at the feet of the previous CEO. Satisfied with his comments, the press -- and Wall Street - responded positively, sales began to pick up and the problem was soon behind him.
About a year later, the company was again experiencing a slight dip in sales, combined with serious product problems. Having learned from his previous experience, the CEO quickly opened the second envelope. The message read, "Reorganize." This he did, and the company quickly rebounded.
After several consecutive profitable quarters, the company once again fell on difficult times. The CEO went to his office, closed the door and opened the third envelope.
The message said, "Prepare three envelopes."
You're in for a wild ride..
1. Meeting with senior management
2. powerpoint presentation on just how fucked they really are, complete with flowcharts and LOL gifs
3. preview development plan on getting everything current
4. write number on piece of paper of what it would take to get the job done
5. pass it around, watch them squirm
6. go-pro camera
7. post results
Joking aside, given the current team doesn't understand the current technology, you'll have a much easier time getting them used to the new tech going forward. Good luck.
This cannot be stressed enough. Hire a contractor much more expensive and experienced than you, and sell to your management you, a much less expensive option will keep up with him in the auditing and renovation project, and keep it up after he finishes. If all goes well, well you share the laurels, if he all goes bad, it is fault. Wise up.
XP, LAMP, 2003 servers, all of those things are spiffy new systems to us. Almost all of my job is trying to get old PDP, MODCOMP and DOS systems into the modern era of things like Windows NT or (Jobs forbid!) linux. Sites with truly aging systems are rarely willing to spend anything like what it would cost to really bring what they have up-to-date and they often have good reasons -- how many security issues do you hear about those aged systems vs [recently] modernized ones?
Of course, it also help to keep all user interfaces the same as much as possible instead of forcing people to learn something new (are you listening, Microsoft?) That kind of change for its own sake rarely adds value. I've seen really great looking Windows software used on the operators' console at nuclear power plants -- except -- it is only great looking from a couple feet away. If you get farther away the lines being graphed become invisible and the text is too small to read without 20/10 vision. This stuff probably only changed format because some programmer (and marketeer and purchasing agent) thought it looked pretty in demonstrations in a conference room.
Ahem. Sorry, poor human factors in software "upgrades" is a pet peeve of mine.
It sounds like it isn't, since you described the role as a linux sys-admin role.
If that's the case, it's also likely that at best, someone higher up wants you to be their prodigy and 'fix' the mess they likely either inherited, or worse created.
I would also say that if you don't have a budget (and most sys-admins won't) then you have no real power.
Run.
Posting anonymously to keep the mods.
Welcome to management, whether you are called a manager or not.
You were lured in by someone saying they are an OSS shop, maybe the person holding the pole didn't know what they were talking about or were lying. Make your choice on whether you want to stay or not.
You're new, so you get a pass for the first couple of weeks. I would spend those weeks listening without preconceived notions. Find out your processes, ask your clients what works and what doesn't. Where are the gaps in the staff skillset? What are the problems that you are being asked to solve?
After you know what is needed, define in your mind what the end goals are, and build a timeline for the necessary transitions.
Changing infrastructure requires capital and training time, right now you don't have either and you have to get executive support to make those changes because there is no show without the dough. For executive support you need to show your business case, why will this save me money (demonstrable improvements in staff efficiency, meaning lower operating costs).
If, per your note, you come in swinging blindly and saying everything is broken, you'll be leaving soon. However, the first thing you did was ask for advice, you have a good shot.
Good luck.
Hmmm, if the country weren't incorrect, I'd say you work where I do. :-) I'm in a similar boat -- lots of old stuff, lots of "don't fix what ain't broke," etc. Change does happen, but it's very slow compared to other places. I can give you some advice based on what I've been able to do:
First, for all the EoL hardware -- secure funding for an appropriately sized VMWare or similar cluster, and P2V everything that doesn't absolutely, specifically require physical hardware. (That list is getting shorter and shorter by the year, even major enterprisey software vendors like Oracle have dropped or relaxed the "must be physical kit for support" requirements.) This at least gets you on supportable hardware, and having the servers on VMs makes the next steps easier. The cost can be justified by some of the astronomical prices vendors charge for out of warranty parts and/or the cost of rolling in yet another physical box per application/function.
Next, once the hardware situation is stable, target OS upgrades. Build a test lab (realistically, just use some of the spare VMWare capacity) and work with each application owner to determine whether their app will run, won't run, or will run with tweaks on a more modern operating system. Again, OS upgrade costs can be justified by pointing out the potential cost of staying on an unpatched, unsupported OS.
Finally, once you're on stable infrastructure footing, _then_ you can look at consolidating applications, moving some work from Windows to Linux, etc. Like all the other posters mentioned, inertia will be your enemy, and especially if you come in with a "savior consultant" attitude, the entrenched IT team will never trust you or support anything you're doing. The key is to involve them, even if they're totally wrong, rather than issuing a blanket prescription for what's wrong with their stuff.
it's the only way to be sure...
Donald 'Duck' Dunn: We had a band powerful enough to turn goat piss into gasoline.
Nuke it from orbit. It is the only way to be safe.
* People who didn't maintain and update their skills need to be gone.
* virtualize everything, by default.
* All deployments need to be devops-based. Infrastructure as code works.
I'd be more worried about management who allowed this to happen. They are losers. If I were paid $150/hr, I'd work there for 6 months and do the best I could, then leave. For anything less - I'd leave now.
First thing you do is sit down and write two letters...
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
Not enough details here to make informed advice try the following for a start:
Questions. Is the company reliant on IT? What is it worth in £million? Do they care about IT enough for you to make a huge fuss and be listened to?
If you are just a Linux admin - and they've lots of Windows admins in a huge company - you are a small cog in a large, crushing machine. If you are supposed to be bringing up their Linux skills and nothing more, do that to the best of your ability and leave.
If your job is to transition some (more) of their estate to Linux AND you have the remit to do it:
Talk to the beancounters about risk management: costs of change vs. vulnerabilities vs costs of remediation
* Make friends with the Windows administrators. Get them to share their main pain points with you: work co-operatively to produce a hitlist of things they want to
see fixed as far as you can
* Bring what Linux machines you have up to date: get patching for these handled correctly.
* Work out where you can usefully expand the Linux estate to fix the Windows admins hitlist.
Then grow out gradually
If you get to talk security posture, hardening, firewalls
Talk to the beancounters about risk management: costs of damage from penetration/loss of data vs. vulnerabilities vs costs of remediation
Make sure that the separate firewall works, then go from there. Were your bosses thinking that a Linux admin was a Windows admin with extra skills, that the Windows skills came automatically with the Linux skills?
Don't beat up on the geezers there for having stale skills. They might actually be OK at keeping those obsolete systems running. Some of them might be OK at getting a new system running, unless they're stuck in their ways.
The bad thing is it is usually accompanied with job-security-loving incompetent IT staff. They will do nasty things and will perfectly manage the blame (they had pretty much time to get good at it). Your first friend is THE backups. Make sure you understand backup _and restore_ plans for every vital system and make sure it is managed not by the person "most familiar and comfortable with that system". Well, your second friend is Niccolo Machiavelli.
You have an impossible task. Rejuvenate your CV, and find your next job.
Seriously though, start with a budget. Until you can secure funds you cannot do anything and the budget will tend to direct what you can accomplish next. Once you have cash, find the oldest piece of hardware in operation and start with that one. You will have more failures based on hardware than you will based on unpatched OS's. Disks are your primary concern in this realm.
Second, after you've completed a few of the more horrendous back-end server migrations, the desktops are next. This is a political move. It will endear you to the user community and this will make additional funding possible. If you focus entirely on the back end, you will run out of support and therefore money long before you can complete the task. You may have to do this step by department, so make sure that your most supportive users get their upgrades first. As I said, this step is entirely political in nature. You will not be able to perform all the upgrades in this step, so be picky.
Third, address the network. Given the health of the server architecture you've described, I suspect that even gigabit-Ethernet is foreign to your environment. Make sure you can build in redundancy along the lines of 802.3ad (LACP) etherchannel connections for all things. Redundancy is your top priority in a network refresh. Basically there are two (2) of every component, each of which is connected to two (2) others.
Fourth, take the remaining servers in order of business impact, most first. This will give you the opportunity to introduce the user community to the concept of "maintenance windows". It will also allow you to engage top management in the upgrade process, which should allow you to re-negotiate the budget; which will be woefully inadequate at first.
Assuming you've made it this far (doubtful) go back and finish the user PC upgrades.
Then prepare to do this entire process again in about three (3) years. Perhaps five (5) if you are lucky enough to get the funds needed to buy things which have significant life. Leasing is also a good thing here because it forces the refresh once the lease terms are fulfilled.
An already-EOL (since last January) OS? Really? Why? That was, like Vista, deliberately designed to bog down or fail on XP-capable hardware.
No company could pay me enough money to willingly take on that headache. I'd likely end up in a mental institution ....
Default passwords are telling you to start by teaching why's and how's to care for any system at all. The age of the existing systems tells you to go OSS + SaaS/IaaS route (by the magic of cloud!!) after sorting out the human and organization issues at first.
"As a senior techie, I've been tasked with helping bring the skillset of the rest of the staff up. Where would you start, given that most of the kit is EoL?
Sounds like they expect you to function as an unpaid tutor as well as your duties as sys-admin. Find out why the last fella left and how long he was in the job. Start looking for another job.
Sounds like you are a systems admin, and you want to be an engineer. You need to talk to the CIO/CTO. This problem pre-dates your tenure.
It's not the system that determines the solution, it's the vendor. So whatever it is they do you are already off to a bad start.
What budget to they have to replace their solution? Where are they in the market? How can you monetize this solution if they haven't already budgeted for it...
In my not so humble opinion your question (and more importantly how you are asking it) would have barely worked in 2003 or before. You are trying to focus on the platform, not the solution. If you don't have 3-5 vendors for your industry you may need to craft something out of LAMP or Open Source but there are not simple open source solutions anymore.
If they are looking at Open Source they need to have a staff to support it, which I am highly doubtful they want to pay for.
Good luck dude!
"Don't fear death... fear not living..." -me
"Medium sized enterprise" - No idea how many seats this is. But it's doable. CHANGE THE PASSWORDS!
Once you've gotten through the "here's the nightmare" aspects:
- Windows XP/2003 will play "nice-ish" with Linux/SAMBA. Just put your Linux servers on a different network domain.
- Most Linux distributions will run on just about any machine that Windows XP can run on. 32 bit or 64 bit.
- Most 2003 server hardware will run Linux nicely.
- A large planned network is usually pretty homogenous. Usually your workstations are pretty much the same as a your servers. (Blah 780's and Hype G5's)
- A large unplanned network is usually a nightmare.
- Building a "parallel" network is usually the easiest way to upgrade... unless you have the budget to greenfield.
So, just for sanity:
0. Get a budget to do a complete asset inventory. Not just "there's a box here" inventory, but "there's a box here with 4 gigs of memory, a 200 gig hard drive, with this motherboard, this video card, and is supported by Linux" type inventory. Figure 30 minutes inside each box, and 30 minutes to find each box.
1. Get an asset inventory. Estimate what the life span of each box is running Linux. Anything with a LWL (Life With Linux) of 4 years needs to get donated. Go through and try to standardize everything (video/memory/network card/drive/cpu) as much as possible. Figure out which network hubs need to be replaced with switches. At a minimum, you want 100BaseT from your switches to your workstations, and 1GBaseT from your switches to your servers. Cat5e and Cat6 are your friends. Full duplex, no auto negotiate, Static-DHCP, etc.
1A. Estimate a minimum of $300 per seat and 1 hr for a "converting" useless hardware up. Any video card with 1GB of video memory should be replaced.
1B. Estimate a minimum of $800 per seat and 30 minutes for outright replacing a workstation. You want 1 cold spare for every 15 workstations.
1C. No budget = no go.
2. Convert 1, and only 1, workstation into an on-site mirror for whatever distribution you are going to use. I like Mageia and Kubuntu. IMHO, Ubuntu just looks ugly. Mirror all your software repositories. Develop your apt-get/whatever plans so that all your future workstations use your local repositories to look for updates. Configure the firewall on your DMZ as appropriate. Figure out how you want everything configured. Write it down/print it out. Network Install scripts are your friend for installing standardized environments. Don't give your users access to root-level stuff by default.
3. Do complete backups of all servers, changing passwords as you go. If they are RAID, a NAS box would be best. If they are BOD, just leave them alone for now. XP/2003 does not like having DC's disappear from the domain. Take one workstation and turn it into a "server" for the Linux box created in step 2.
4. Give your EBKACs some exposure to Linux. Convert best candidate's workstation to dual-boot in a non-root account. Have them record every reason they have for booting into XP.
5. Payroll has no reason to see Engineering projects.
6. Prepare for lots of hand-holding and stupid questions. I would suggest having a steady supply of aspirin, antacids, and decent scotch on hand. Para, Tums, and Glenrothes Special Reserve works for me.
Preparing for a network conversion will take you roughly 1 admin-day per server and 2-3 admin-hours per workstation. A repository clone will take about 60 GB. 3 hrs per workstation to install/configure. 3 hrs per server to install/configure.
Hope this helps. The hardware/software sounds like fun, but the training bit is going to be painful.
Now. Otherwise you'll suffer the death of a thousand cuts, or you'll just be fired for being a 'boat rocker.'
First and foremost - lose no data.
Confirm backup AND restores (virtual is handy here) for all the older systems, along with any support contacts / vendors.
Once that's done you can address the rest of it - investigate, scope, plan and fix.
Security (document the flaws, fix, or get waivers) - including the usual AV kinds of things.
Virtualize or replace to reduce old hardware exposure.
Consider cloudy options if appropriate.
Look for special hardware items (licence dongles, modems, test equipment etc that might have special needs).
Don't forget software licence compliance either. Could potentially be a very big legal/$$$ exposure if there are instances of un/under licenced software.
Start by doing your job instead of asking slashdot to do it for you. On no, you have to think? Well fuck you. That's what you are getting paid to do.
You need to hire a couple-three top people who can migrate the systems. Migration from EOL systems is a headache, requiring plenty of debugging. I've been in those situations, and you need people who can do that task.
Why is he in the new job?
Either:
A) the last guy was fired as he was the problem, ran everything solo and didn't share info/admin roles and wasn't helpful.
Or
B) the last guy left as he started the same task you've now been given and either fail to do it or figured out he was the new fall guy, which is now you.
Ask the very top management team what their business plan and supporting ICT 3+ year strategic plan is today, right now.
If they don't know where they're headed, business and ICT wise, after firing the last guy or two and after asking something big of you like this and they need time to think about it and draw something up.... Then they will never get there.
As others have said, it sounds like you need to chuck out a lot of the systems and buy new. But your absolute first job is to secure what you've been left - change all those default passwords, do what you can to secure the ancient systems. Then establish your budget with management, replace your systems and bring in user training on the new systems, not the ones that are destined for the bin.
Having been handed the same box of shit the last time I changed jobs I chose VMware. These days you might pick something cheaper but back then enterprise virtualisation was relatively new. Virtualise the lot onto a smaller hardware footprint, then create copies of the any interrelated production servers to test your upgrades and make sure there are no unexpected surprises. Shadowprotect is your friend.
You've described existing infrastructure, but the important thing for the business is applications. That's the thing they need, every day. I worked in an infrastructure group in a UK investment bank, the only time they notice what you do, is when it snarls up or fails.
For example, there was a recent thread discussing whether Access has an open-source equivalent, IMO it doesn't really. So, if they use a lot of Access that will constrain upgrade path UNLESS they're prepared to take some risks and spend to take it out of the equation. But, mainly, the list of what's delivered to the business via the servers and on the desktop is the thing. No-one cares about infrastructure [except us, boo-hoo] as long as the price is right [including manpower] and it works.
It sounds, to me, like this is Windows desktops and Linux servers [and therefore Samba, LAMP etc. for example] this is not a bad way to live and many companies do so. That would mean that the client upgrades and server upgrades would be reasonably orthogonal, but I don't know all the details, either. To be honest, I'd be inclined to ask this on Server Fault, but unless there were more details, it's likely to be closed as being 'open ended'. Good luck!
On y va, qui mal y pense!
To round out the OP a little.
The IT estate is 75% windows, desktops and servers, AD with no GPO's in place. Exchange with no policies in place.
No windows skills at all on-site, except myself. Few people with Linux skills, and 1st line that have been de-skilled from fixing things themselves to just administering the ticketing system. No change management, no risk management.
The applications are all running from old citrix metaframe.
All except the DB and sales front end, which is open.
There is some chat about a budget to bring things into the 21st century.
But its very vague.
Writing 2 letters sounds like the best advice currently.
I've seen very useful posts regarding how to start a project like this, what kind of financial and executive support you will need and how to work the politics game to actually make this work.
But I haven't really seen a clear definition of what he is supposed to achieve? This sounds like a job where you have to wave your IT wand at problems to make them go away technomagically. I've seen many technical solutions that were applied because they were needed from a technical viewpoint. Upgrades, safety measures, terminal->workstation->terminal, etc..
But what is the endgame of your client? What goal is to be reached? Companies that let their systems degrade like this tend to do so because they have no idea what they actually do. They just want all the problems with the systems to go away.
Get clearly defined boundaries to work within. Not just budget, but also functional specifications beyond "What it used to do, but now on windows 10". This will require a lot of time and investment from managers and operational managers. If those do not invest time and/or resources in this process, you cannot give them what they need.
And clearly differentiate between what you see/think what the client needs and make what the client wants. That means getting clear specifications and have management/stakeholders sign off on all changes that you want to implement. Clearly state what the impact of this will be. In writing.
The best change manager I ever saw had everything in writing for a system merger of 3 companies. When one of the three CEO's came in angry because his mail address was changed, the change manager handed the CEO a document where the CEO had signed for this change. And that's just an e-mail domain change. (And of course a forward from the old domain was setup after that point.)
TL:dr;
Get clear specifications. Companies in such mess get there by not defining what they want. Not defining will make any solution a mess. Demand time invested from key users, operational and strategic managers in the specification phase.
I do this for a living, but I will give you some free advice just this once.
Given your summary, the only thing relevant is probably this:
[...] a large IT department with almost no skills in the technologies on site. [...]
As an experienced career consultant with many years experience I can read between the lines and glean information that every other person who has replied to this thread seems to have missed. Essentially it boils down to this: every single employee in that large IT department started out in your job. Lots of people in your position didn't make it into the IT department because they didn't realise this.
Basically what you have to do is entrench yourself in the corporate culture (lots of parties, arrive at work drunk, do drugs, that sort of stuff) until you're accepted as a functional part of the IT department. Once you are, and at this point you're still in charge, place an advertisement for your replacement. Hire the second (never the first) person who applies for the job, making sure that you have a 35-year contract in place to ensure your continued employment as a support person with no performance reviews and a guaranteed 25% increase in salary per year. Make sure that the new person in the job signs this contract as well as all upper management and the board of directors. Once that is done, play Freecell and become the world champion.
Leave now, it sounds like a joke and you are being put into the role of training complete monkeys. I doubt management will ever fund or sanction the real solutions. They want you to be the sticky tape solution and hope you can get them out of a hole.
OK, so this isn't just an excuse to post "Beowulf Cluster!" on /. one last time, but it probably sounds like it... :-)
What you really want to do is start using cloud services like DigitalOcean or (if you must) AWS. In this day and age you can pull more computing power out of the damn _air_ than currently exists in your building. This isn't "going OSS", this is a space where open source tools like Linux, GIT, Ansible, cassandra and other are simple necessities because nothing else can do the job. Licences are impossible to keep track of in cluster environments, so the only choice is to go GNU.
But cluster computers are just 'virtualizations' of real hardware, and one of the fastest ways to understand something is to build one. A little one. So, you want a little pile of dozens of identical linux machines, cheap enough to write off on the stationary budget as "training manuals", and that can be repurposed as fast as sticking a cartridge in a nintendo. You want a Raspberry Pi, Banana Pi, pick your flavor. Everybody gets one (thanks, spidey!) and that creates common ground. If you're lucky, three months in, everyone will be swapping their favorite games on SD cards. That's how you know you've won.
You show how lots of little computers can function as one large computer using modern tools, and you show how half of it can be sitting on your desk, and the rest can be on another desk, or in the cloud. Hardware is disposable.. it's the software that's immortal.
Once people have some familiarity with a new tool, and been given an opportunity to use it stupidly in private, then the inevitable next stage is that people will want to use their new hammer to hit every nail-like thing in the vicinity, and that's when you need a little careful enthusiasm management. Everything that breaks down or goes dodgy will get replaced with the tools that everyone knows best, the one they've been most recently learning.
You're in the UK. Learn the history of the Acorn BBC Microcomputer... and how it educated a generation... you may have taken it for granted, but it's a great example of what you're trying to do.
Autonomy, mastery, and purpose. That's what people want. Provide those, and the tools to get the job done, and mostly it's a matter of keeping out of their way.
Jeremy Lee | Orinoco
From the post I see you're hired as a sysadmin and it sounds like the additional duty of training other IT staff. What you can do will depend on where you are in the chain, if there's an IT manager and/or C position above you then you can only go so far. (Unless they're asking you to do their job.) As a sysadmin you have to focus on the servers and network IMHO. And in that regard I agree with several of the posters about going virtual. Ideally you need to store the VM's on a SAN of some type and plan your load so if some of the physical servers fail you can move the VM's around. I read thru a bunch of comments but did anyone ask if there's a security person on the team? Will you have to plan physical and digital security for this upgrade? Because that's a whole other bag of cats. Because changing from default passwords is one thing but having a plan to update every 6 to 12 months, where will they be recorded, and all the fun of creating access groups. If you have to upgrade the desktop side I'll give you two scenarios from my last 2 jobs. Company A) Purchases 4 years complete care from Dell. Without knowing numbers I know this is expensive but if anything happens I know all my PC's will be back up the next day even if the end user dropped the PC down the stairs. This org. purchased enough PC's to replace 1/4 one year, the following they purchased 2/4 more PC's to replace all that were running out of warranty. Company B) They purchased 3 year standard warranty on all PC's. Now here's the interesting thing. We performed an annual refresh in which we replaced 25% of the employees PC's. The only downside to this plan is we had 4-5 models of PC's and our image ended up having drivers for 10 models when you count desktops, laptops and precision workstations. And if anyone broke their PC between years 3-4 we had to refresh out of band so we had to have PC's on hand in supply. The advantage though is obvious, you balance the cost of the refresh and it runs as a continuous part of the IT budget instead of running the worry of being pushed off because THIS year wasn't so good for business. As far as training goes, well, I'm the tech that has to know WHY something works the way it does. So I usually end up annoying the teacher because I inevitably end up asking questions they cannot answer and then I have to figure out via the google gods (why can't I hard set 'Verbatim'?) I think the biggest problem you'll have with training is if you have techs who like the way the company runs now and don't want the extra work because they're going to fight you every step and/or do stuff half-assed. The best you can do is DOCUMENT. Figure out how things need to be and write the process/details and any tech who cares and wants to be there will be able to follow you. That's my 2 warped cents.
PS I guess I need some training, or my browser's screwed up. I had that typed up with paragraphs and breaks...
They're going to LOVE going from XP to Ubuntu.
You need to have at least three hardware servers, all with lots of memory and disk, so you can have a primary and backup for production use and another for IT infrastructure development. If you're doing both VMware and OpenStack, which is not a bad position to take, you really need 5, two per hypervisor plus a spare.
Otherwise you're going to be spending your time keeping your Shiny New Virtualization Platform up to date, instead of spending it Virtualizing Old Non-Shiny Stuff, which is what you should be doing.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
I'd focus on the people side - figure out what who you're going to get rid of, who you can work with, and build good habits of working well with them while you hold down the fort - feel out your first few changes to see what kinds of resistance you get from humans (and from technology). How that goes will give you a feel for the possibilities of larger change.
For every problem, there is at least one solution that is simple, neat, and wrong.