Slashdot Mirror
Ask Slashdot: Herding Cats, Aging Systems?
An anonymous reader writes: I've recently started a job at a medium-sized enterprise in the UK. They claimed to be an advocate of open-source. The job was advertised as a Linux sys-admin. I've been in the role a short while and the systems right across the business are end-of-life: lots of XP and 2003 servers, a handful of LAMP web servers, and a large IT department with almost no skills in the technologies on site. Most boxes have the default password still. As a senior techie, I've been tasked with helping bring the skillset of the rest of the staff up. Where would you start, given that most of the kit is EoL?
That's the most obvious thing. Bring in supported systems and train them in those systems as you deploy them.
Well, your question leaves out a lot of details but from what you've said so far, look at getting some new hardware in there and start virtualizing some of the the EoL systems. This will provide you an upgrade path for existing systems and a snapshot'd point of restore in the event of a failure.
No guns, no knives... do you pussies still get rope or are you going to have to find a tall building to jump off instead?
You need to start a training program. If the current workers feel uncomfortable in the new technologies, the will oppose you every step of the way (though they won't say why). If they feel comfortable in the new technologies, they will push you faster in the adoption.
If a team has been working on a lousy codebase, your first priority should be to teach them to do better. You can try cleaning things up, but they will make messes faster than you can clean. You need to find a way to teach them to at least not make things worse.
"First they came for the slanderers and i said nothing."
I don't know your organization's level of risk tolerance, but getting them to pay for one of the following would be an eye-opener:
- A vulnerability assessment will show a sea of red for the unsupported platforms. Maybe that'll be sufficient to convince them that it's time to upgrade (and train up on new stuff).
- A penetration test will take those same vulnerabilities, and combine it with attempting to use those vulnerabilities to see what they could get. The difference is in trying to use those issues, and turn them into "oh SHIT" screen shots in the report. It's the difference between "someone could theoretically do X" and "someone just did X, and documented it all for your edification."
On the latter engagements, especially with the dreadfully old stuff, it is quite enlightening to include those screen shots that show how I've added new users, logged in with them, and used them to poke yet more systems I couldn't reach from the starting point. The under-educated staff would only help things if social engineering was in scope too.
Well, you have 3 main choices:
1) Try to fix it and succeed
2) Try to fix it and fail
3) Run like hell
You won't be able to force the rest of the staff to bring up their skillset. Management has clearly left it to rot on the vine for a very long time. And, by the sounds of it, they don't know what they've even got.
A large IT department with no skills with the technologies on site? What exactly is that large IT department doing for this company? If you have a bunch of people with no skillsets with the technology they have ... then what skillsets do they have, and how is it helping you?
Without more detail, I'm hearing "Hi, I've just joined a company with a terrible IT department, how do I fix that?" Who let it get into such a bad state? Because if they're still around, no way in hell you'll ever fix it.
Lost at C:>. Found at C.
Seriously. If this is how they're running their operation to this day, chances are its not just harmless, easily washed-away naivety that is keeping everything so poorly organized and insecure. Chances are you're going to find this out the hard way though, and they'll mysteriously get instantaneously far less apparently incompetent when it comes to finding ways to get you fired first.
I know it sounds obvious, but I would simply do what they are asking you to do - educate users, etc.
You could mention the fact that their equipment/software is obsolete. From my experience, I would probably mention it but not push it.
The reason is that most companies have their own way of doing things. Even if you have a better way of doing it (and you probably do!), there is nothing wrong with making suggestions. But pushing your suggestions onto management (or whoever) - even if you mean well - won't necessarily be well received.
Most people are afraid of change; and hounding the company to "get up to date" (and spending more money) might not be very well received - especially if it's from the new guy.
That's just my $.02
Political correctness is really just herd psychology pushed by insecure people who desperately seek social conformity.
a large IT department with almost no skills in the technologies on site
No skills? Then what is it that they do all day? I suspect that your IT department has been the dumping ground for employees that can't just be gotten rid of. In other words; politically connected. I've been there and tried to deal with that. And in my opinion, it can't be done by someone without serious seniority.
You could try introducing training programs for the target architecture. And some of the motivated staff will avail themselves of this. But be prepared to run across a few people who refuse and insist on hanging on to their legacy Windows stuff.
Have gnu, will travel.
Hmm, if there are legacies, i would advise the following.
1. virtualize whatever you can.
2. remove the oses which are no longer supported by the vendor MSFT...2003/xp..because of security issues.
3. get a list from business of all the critical system. There will need to create a roadmap at this stage for migration.
4. in the mean time. start working / training the team on open source, specifically for (1) as a virtual platform.
5. start planning (3), by training people in the direction to be...
Kill everyone. Set fire to the place. Plead insanity. When they see what you were supposed to work with, they'll believe you.
Make a map of what you have, what the main issues are with each piece, and then a plan for replacement/updating/whatever. Try to include some rough (and higher than you really think it will be) cost estimates. Then present to a boss, and get buy-in. If you don't get buy-in, start updating your CV and look for another job.
Don't blame me, I voted for Kodos
https://www.youtube.com/watch?v=Pk7yqlTMvp8
It depends on how much actual authority you have, how conservative the corporate culture is, and whether there are any entrenched ways of doing things. This isn't a technical question but a political one. If you actually (as opposed to officially) have authority to tell them how to do things you need first find out how the system is working now. Maybe they didn't set up passwords because multiple departments need to connect to the same server and there's no secure password control in place. Maybe they're disorganized. Maybe they're inexperienced. These all require different activities to repair the problem.
You mentioned EOL hardware, but you didn't say whether a migration is planned or whether the money is available for one. Obviously new hardware is a great opportunity for user training, but again there are too many unknowns here. How much extra time do the engineers have to train? How much of the existing system setup is invisibly a part of how the users interact with it?
It sound to me like you're standing on a powder keg. The right way to deal with it is to gather information. Make benchmarks. Understand system inter-operations and use. Learn who is doing what and why. Only a fool would start declaring X and Y need to be done without taking a look around first.
If video games influenced behavior the Pac Man generation would be eating pills and running away from their problems.
I would audit everything, Make a matrix of things that need to be addressed easy to hard, least significant to most, and start chipping away at it. It will take time to turn that ship around, but it will be worth it, and you will keep your sanity.
I think that bot from a few articles down is trying to weigh-in...
Do not look into laser with remaining eye.
Buy a new system. Power down every system in turn and try to power it up again. If it will not start, replace it.
NEVER power down old hardware on purpose unless you have backup plan for the system... Old hardware has a habit of not coming back when you power off and if it dies, you created an emergency for yourself...
There are going to be enough unforced errors in the process, you needn't go out and look to create them.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
Although there's big money in cleaning up someone else's mess, you got to recognize a hopeless situation when you see one. Fire everyone and bring in a professional IT team to take over the operations. Or run like hell and hope that the next job isn't as bad or worse as this one.
You've been dropped in an environment that is legacy and probably has production problems. Use that to your advantage.
You've been also dropped in a leadership role (not management, leadership).
Your #1 target should be to make yourself redundant (which ironically is likely to get you promoted, it's called succession :).
So look at doing something like identifying #1 problem (Pareto charts help). Ask for volunteers (or volunteer some people), give them the problem to solve, use whiteboards, etc to help them discover the solution. You may facilitate and provide hints to get things done. Empower and guide the people you are helping.
Read up on https://en.wikipedia.org/wiki/..., you are likely in a #2 or #3 combination. You can help lead people to move to a #3 with leadership, with the idea to get to #1 over time (with their help).
Of course there might be some issues that you might need to solve like EOL systems and any budget that may be needed. If the OS is old, then probably the HW is old as well. Budget for that is probably going to be your biggest issue.
Wanted: IT Director
Pay-scale: Entry level.
EJECT!! EJECT!!
You'll never ever overcome that much inertia and penny-pinching. Don't spend the next five years being frustrated before you figure this out.
1) Start with the printer by printing copies of YOUR resume
2) Distribute broadly
3) Get the hell out before their stench rubs off on you
Seriously mate, you CANNOT fix this. They will drag you down to their level.
If the powers that be allowed such a situation to exist, and didn't specifically hire you to change it, along with a guaranteed budget with which to accomplish the task, then it will be almost impossible for you to fix it. The fact that they essentially lied to you about your role does not bode well.
If this is a resume building job for you then dink around the edges on things that won't require much, or any, money or many changes to the status quo. Make big talk about how you are improving things. Take every opportunity to educate YOURSELF on things you can use later in your career. Put in your year, then move on. It doesn't matter what you ACTUALLY accomplish in an environment like that. Only what you can spin things in to for your resume. Cynical, I know, but you gotta think about yourself first. This is a dead end job.
And you had no idea at interview? Did you ask questions? If you did, and if they told the truth, what WAS your intended strategy?
Getting out quick is great advice. However, if that's not a realistic option, then level with management that it's a big job, and ask for a priority list from them.
Make a list of things to be done, along with the reason or consequences of not doing them, give your suggested priorities (A, B, C, D, etc.) for each task, and ask management to confirm or reassign priorities.
Often they won't want to assign priorities because that commits them to their decisions. Managers like wiggle room to blame others.
To get around that tendency, state something like, "I have assigned default priorities to the given tasks. I shall use these priorities as a my default working assumption unless and until explicitly given new priorities. I invite your feedback and would be happy to answer any questions about them."
They probably won't like that pressure and explicitness, but better to take your lumps up front rather than have an even bigger meltdown in the future.
And some may even appreciate your documenting of tasks. It might help them justify getting you some help. But that's a secondary reason for the list.
Table-ized A.I.
-- quote--
Where would you start,....
----------
with the thermonuclear option !
with DEFAULT passwords of "password"
and using XP and MS 2003
the use of DBAN has been authorized
"I don't pitch OpenSUSE Linux to my friends, i let Microsoft do it for me
>> large IT department with almost no skills in the technologies on site...As a senior techie, I've been tasked with helping bring the skillset of the rest of the staff up
Stop right there and understand that you've been sent on a wild goose chase. You're not really going to train your existing staff - ever. Instead, what you're doing is writing the job descriptions for the outsourced personnel that your management will hire to replace the deadwood in your "large IT department" (because it's no doubt costing them an arm and a leg, and personnel budget is sucking up the money that should be flowing to technology upgrades).
So, what should you do? I'd say complete the assignment with a smile on your face and a suggestion that some of the skills are already available elsewhere in the market. Hopefully management will pat you on the head and let you move on to the next level: interfacing with the outsourcing company that's waiting in the wings.
You were brought on as Linux SysAdmin; you now know that the job is nothing of the sort, and getting things up to speed will require massive investments in technology, personnel, and many sleepless nights on your part, should you choose to perform this task.
If you want to do this at all (and it sounds like you do), you need to demand a raise and quit if you don't get it.
They are not open source advocates, they are cheapskates who like the prospect of 'free' anything. No supported equipment, no updates, no training for their staff, they simply don't appreciate the value of their IT.
Let me guess, no decent backups either? No DR plan? Nothing of the sort? If you want to stay there, demand a decent budget ( = commitment) and build greenfield. If you don't get a decent budget, run.
To Terminate, or not to Terminate, that's the question - SCSIROB
End of Life? Dude, I'm running Cobol code that probably dates back to before you were born!
If it does the job that's one thing. If it's all patched and up to date security-wise, you are OK. If however, the systems are constantly running at 100%, and you've got latency and throughput issues, then that's a different story.
If nothing's getting backed up because the systems are under huge strain, then in that case you are welcome to start upgrading. In fact, you might be able to keep the same hardware and simply "upgrade" to Linux slowly, which is less resource-intensive than Windows.
And the way to do it is to build a new box that "mirrors" an existing one; once you've tested it to death, and you're confident nothing has been missed, you retire the original box, then use that box for the next build that mirrors another box and so on.
If telephones are outlawed, then only outlaws will have telephones.
Take the paychecks while you can, but you need to get the hell out of there.
The company obviously doesn't care about IT, and you almost certainly aren't going to change that. Let them rot.
Alternative viewpoint: Microsoft Windows XP "end of life": Conflict of interest
"April 8, 2014: Microsoft began charging millions for support of its Windows XP product. "
Seriously, "accidentally" toss a lighted cigarette into the paper recycling bin in the server room on your way out one night. You'll be able to start fresh with the insurance money.
None of them can see the clouds; The polished wings don't care.
ok, saying PM will fix it is cheap but that's really the core task of project management "to devise a plan"
But a specific plan with goals needs to be based on an analysis of the environment you are actually in and you did the first steps, you tried to describe the environment.
medium-sized enterprise, UK
Ok, you're in the UK and have access to a solid workers pool of skilled IT, also available by contract workers.
Think about getting an aide, nothing is worse than a project manager that looses oversight!
They claimed to be an advocate of open-source.
But everything contradicts that, don't cling onto open source, don't waste energy to educate people that don't understand. It's useless, and open source is only one solution, that can work but also can fail. Especially if you have people with no skill!
The job was advertised as a Linux sys-admin. I've been in the role a short while and the systems right across the business are end-of-life: lots of XP and 2003 servers, a handful of LAMP web servers,
Do you have an overview, are all the systems accounted and cataloged with "Machine Power"/"Electric Power"/IP/MAC/Remoteinstall/config etc.. possiblities.
Because this actually looks like those who hired you did not have such an oversight.
and a large IT department with almost no skills in the technologies on site.
First you don't need a training plan you need a Maintainance&FIXIT plan = nothing is worse than systems failling, when you grab resources for training. because you will be blamed for, your oppinion will lose worth you will lose resources.
Idea: Have a fix-it task force at hand. Only selected few who really can "fix it".
(Think about high skilled contract workers.)
Most boxes have the default password still.
That could be getting fixed by a task force.
As a senior techie, I've been tasked with helping bring the skillset of the rest of the staff up.
Good luck with that.
Where would you start, given that most of the kit is EoL?
Make a plan based on the current equipment(what can still run and what is really dead) and the demand of the market. ("priorities")
Replaced a system that was running on an old minicomputer that was no longer supported by the vendor and they were cannibalizing old systems to keep running. That is EOL.
love is just extroverted narcissism
You're in for a wild ride..
1. Meeting with senior management
2. powerpoint presentation on just how fucked they really are, complete with flowcharts and LOL gifs
3. preview development plan on getting everything current
4. write number on piece of paper of what it would take to get the job done
5. pass it around, watch them squirm
6. go-pro camera
7. post results
Joking aside, given the current team doesn't understand the current technology, you'll have a much easier time getting them used to the new tech going forward. Good luck.
XP, LAMP, 2003 servers, all of those things are spiffy new systems to us. Almost all of my job is trying to get old PDP, MODCOMP and DOS systems into the modern era of things like Windows NT or (Jobs forbid!) linux. Sites with truly aging systems are rarely willing to spend anything like what it would cost to really bring what they have up-to-date and they often have good reasons -- how many security issues do you hear about those aged systems vs [recently] modernized ones?
Of course, it also help to keep all user interfaces the same as much as possible instead of forcing people to learn something new (are you listening, Microsoft?) That kind of change for its own sake rarely adds value. I've seen really great looking Windows software used on the operators' console at nuclear power plants -- except -- it is only great looking from a couple feet away. If you get farther away the lines being graphed become invisible and the text is too small to read without 20/10 vision. This stuff probably only changed format because some programmer (and marketeer and purchasing agent) thought it looked pretty in demonstrations in a conference room.
Ahem. Sorry, poor human factors in software "upgrades" is a pet peeve of mine.
You've been hired to fix an IT department where: -the technology is obsolete -the workers don't know how to use it -the people who caused this state of affairs are your superiors and coworkers -they probably aren't going to fire themselves or let you fire them
You need to leave asap. These people obviously have no incentive to improve and they're only going to become hostile once you start trying to make changes.
idk man, I'd take this as an opportunity. Get them in your pocket, show them that YOU are the GOD that can bring them into the new age. Could be a hell of a resume fixture.
Hmmm, if the country weren't incorrect, I'd say you work where I do. :-) I'm in a similar boat -- lots of old stuff, lots of "don't fix what ain't broke," etc. Change does happen, but it's very slow compared to other places. I can give you some advice based on what I've been able to do:
First, for all the EoL hardware -- secure funding for an appropriately sized VMWare or similar cluster, and P2V everything that doesn't absolutely, specifically require physical hardware. (That list is getting shorter and shorter by the year, even major enterprisey software vendors like Oracle have dropped or relaxed the "must be physical kit for support" requirements.) This at least gets you on supportable hardware, and having the servers on VMs makes the next steps easier. The cost can be justified by some of the astronomical prices vendors charge for out of warranty parts and/or the cost of rolling in yet another physical box per application/function.
Next, once the hardware situation is stable, target OS upgrades. Build a test lab (realistically, just use some of the spare VMWare capacity) and work with each application owner to determine whether their app will run, won't run, or will run with tweaks on a more modern operating system. Again, OS upgrade costs can be justified by pointing out the potential cost of staying on an unpatched, unsupported OS.
Finally, once you're on stable infrastructure footing, _then_ you can look at consolidating applications, moving some work from Windows to Linux, etc. Like all the other posters mentioned, inertia will be your enemy, and especially if you come in with a "savior consultant" attitude, the entrenched IT team will never trust you or support anything you're doing. The key is to involve them, even if they're totally wrong, rather than issuing a blanket prescription for what's wrong with their stuff.
it's the only way to be sure...
Donald 'Duck' Dunn: We had a band powerful enough to turn goat piss into gasoline.
First thing you do is sit down and write two letters...
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
Not enough details here to make informed advice try the following for a start:
Questions. Is the company reliant on IT? What is it worth in £million? Do they care about IT enough for you to make a huge fuss and be listened to?
If you are just a Linux admin - and they've lots of Windows admins in a huge company - you are a small cog in a large, crushing machine. If you are supposed to be bringing up their Linux skills and nothing more, do that to the best of your ability and leave.
If your job is to transition some (more) of their estate to Linux AND you have the remit to do it:
Talk to the beancounters about risk management: costs of change vs. vulnerabilities vs costs of remediation
* Make friends with the Windows administrators. Get them to share their main pain points with you: work co-operatively to produce a hitlist of things they want to
see fixed as far as you can
* Bring what Linux machines you have up to date: get patching for these handled correctly.
* Work out where you can usefully expand the Linux estate to fix the Windows admins hitlist.
Then grow out gradually
If you get to talk security posture, hardening, firewalls
Talk to the beancounters about risk management: costs of damage from penetration/loss of data vs. vulnerabilities vs costs of remediation
Yes. For one thing show them how a clean Windows 7 32bit looks (Aero Basic or Windows Classic?) and runs on the crappy old desktops, and how to deploy it. Probably with some "magical" network booting set up. "See, the DHCP can be set up to beam an installer or diagnostic tool on the network, if you press F12 on boot up". Have the technicians make "Oooohs" and "Aaaahs" and make feel them empowered ; try to have them do them this task. Tom Sawyer was tasked to paint the wooden fence, and he tricked the other children into wanting to do it.
Give control of DHCPs on network segments that only have desktops and printers then they will like collecting the MACs and entering them etc. and some dumb things like pinging the server or the printer. Wow them with the local DNS : with a cname alias, "printer1" is the same as "HPLSJ5M_ZORGBL" is the same as "192.168.3.81".
Or so I imagine.
Nom you need the support of the guy who actually signs the checks. The finance department just does what they are told
Make sure that the separate firewall works, then go from there. Were your bosses thinking that a Linux admin was a Windows admin with extra skills, that the Windows skills came automatically with the Linux skills?
Don't beat up on the geezers there for having stale skills. They might actually be OK at keeping those obsolete systems running. Some of them might be OK at getting a new system running, unless they're stuck in their ways.
You have an impossible task. Rejuvenate your CV, and find your next job.
Seriously though, start with a budget. Until you can secure funds you cannot do anything and the budget will tend to direct what you can accomplish next. Once you have cash, find the oldest piece of hardware in operation and start with that one. You will have more failures based on hardware than you will based on unpatched OS's. Disks are your primary concern in this realm.
Second, after you've completed a few of the more horrendous back-end server migrations, the desktops are next. This is a political move. It will endear you to the user community and this will make additional funding possible. If you focus entirely on the back end, you will run out of support and therefore money long before you can complete the task. You may have to do this step by department, so make sure that your most supportive users get their upgrades first. As I said, this step is entirely political in nature. You will not be able to perform all the upgrades in this step, so be picky.
Third, address the network. Given the health of the server architecture you've described, I suspect that even gigabit-Ethernet is foreign to your environment. Make sure you can build in redundancy along the lines of 802.3ad (LACP) etherchannel connections for all things. Redundancy is your top priority in a network refresh. Basically there are two (2) of every component, each of which is connected to two (2) others.
Fourth, take the remaining servers in order of business impact, most first. This will give you the opportunity to introduce the user community to the concept of "maintenance windows". It will also allow you to engage top management in the upgrade process, which should allow you to re-negotiate the budget; which will be woefully inadequate at first.
Assuming you've made it this far (doubtful) go back and finish the user PC upgrades.
Then prepare to do this entire process again in about three (3) years. Perhaps five (5) if you are lucky enough to get the funds needed to buy things which have significant life. Leasing is also a good thing here because it forces the refresh once the lease terms are fulfilled.
No company could pay me enough money to willingly take on that headache. I'd likely end up in a mental institution ....
"As a senior techie, I've been tasked with helping bring the skillset of the rest of the staff up. Where would you start, given that most of the kit is EoL?
Sounds like they expect you to function as an unpaid tutor as well as your duties as sys-admin. Find out why the last fella left and how long he was in the job. Start looking for another job.
"I am in this same boat right now. Looking for a shore I can jump to."
.. er .. longevity. Never-ever do unpaid overtime.
Go into teaching, it's a lot less stressful and has a longer job
Sounds like you are a systems admin, and you want to be an engineer. You need to talk to the CIO/CTO. This problem pre-dates your tenure.
It's not the system that determines the solution, it's the vendor. So whatever it is they do you are already off to a bad start.
What budget to they have to replace their solution? Where are they in the market? How can you monetize this solution if they haven't already budgeted for it...
In my not so humble opinion your question (and more importantly how you are asking it) would have barely worked in 2003 or before. You are trying to focus on the platform, not the solution. If you don't have 3-5 vendors for your industry you may need to craft something out of LAMP or Open Source but there are not simple open source solutions anymore.
If they are looking at Open Source they need to have a staff to support it, which I am highly doubtful they want to pay for.
Good luck dude!
"Don't fear death... fear not living..." -me
Or take a shot, tackle it, and find yourself Director of IT in five years with a path to VP. There are some people that can thrive in this sort of environment and ride the train to wealthy.
It doesn't hurt to be nice.
You need to hire a couple-three top people who can migrate the systems. Migration from EOL systems is a headache, requiring plenty of debugging. I've been in those situations, and you need people who can do that task.
You've described existing infrastructure, but the important thing for the business is applications. That's the thing they need, every day. I worked in an infrastructure group in a UK investment bank, the only time they notice what you do, is when it snarls up or fails.
For example, there was a recent thread discussing whether Access has an open-source equivalent, IMO it doesn't really. So, if they use a lot of Access that will constrain upgrade path UNLESS they're prepared to take some risks and spend to take it out of the equation. But, mainly, the list of what's delivered to the business via the servers and on the desktop is the thing. No-one cares about infrastructure [except us, boo-hoo] as long as the price is right [including manpower] and it works.
It sounds, to me, like this is Windows desktops and Linux servers [and therefore Samba, LAMP etc. for example] this is not a bad way to live and many companies do so. That would mean that the client upgrades and server upgrades would be reasonably orthogonal, but I don't know all the details, either. To be honest, I'd be inclined to ask this on Server Fault, but unless there were more details, it's likely to be closed as being 'open ended'. Good luck!
On y va, qui mal y pense!
I do this for a living, but I will give you some free advice just this once.
Given your summary, the only thing relevant is probably this:
[...] a large IT department with almost no skills in the technologies on site. [...]
As an experienced career consultant with many years experience I can read between the lines and glean information that every other person who has replied to this thread seems to have missed. Essentially it boils down to this: every single employee in that large IT department started out in your job. Lots of people in your position didn't make it into the IT department because they didn't realise this.
Basically what you have to do is entrench yourself in the corporate culture (lots of parties, arrive at work drunk, do drugs, that sort of stuff) until you're accepted as a functional part of the IT department. Once you are, and at this point you're still in charge, place an advertisement for your replacement. Hire the second (never the first) person who applies for the job, making sure that you have a 35-year contract in place to ensure your continued employment as a support person with no performance reviews and a guaranteed 25% increase in salary per year. Make sure that the new person in the job signs this contract as well as all upper management and the board of directors. Once that is done, play Freecell and become the world champion.
OK, so this isn't just an excuse to post "Beowulf Cluster!" on /. one last time, but it probably sounds like it... :-)
What you really want to do is start using cloud services like DigitalOcean or (if you must) AWS. In this day and age you can pull more computing power out of the damn _air_ than currently exists in your building. This isn't "going OSS", this is a space where open source tools like Linux, GIT, Ansible, cassandra and other are simple necessities because nothing else can do the job. Licences are impossible to keep track of in cluster environments, so the only choice is to go GNU.
But cluster computers are just 'virtualizations' of real hardware, and one of the fastest ways to understand something is to build one. A little one. So, you want a little pile of dozens of identical linux machines, cheap enough to write off on the stationary budget as "training manuals", and that can be repurposed as fast as sticking a cartridge in a nintendo. You want a Raspberry Pi, Banana Pi, pick your flavor. Everybody gets one (thanks, spidey!) and that creates common ground. If you're lucky, three months in, everyone will be swapping their favorite games on SD cards. That's how you know you've won.
You show how lots of little computers can function as one large computer using modern tools, and you show how half of it can be sitting on your desk, and the rest can be on another desk, or in the cloud. Hardware is disposable.. it's the software that's immortal.
Once people have some familiarity with a new tool, and been given an opportunity to use it stupidly in private, then the inevitable next stage is that people will want to use their new hammer to hit every nail-like thing in the vicinity, and that's when you need a little careful enthusiasm management. Everything that breaks down or goes dodgy will get replaced with the tools that everyone knows best, the one they've been most recently learning.
You're in the UK. Learn the history of the Acorn BBC Microcomputer... and how it educated a generation... you may have taken it for granted, but it's a great example of what you're trying to do.
Autonomy, mastery, and purpose. That's what people want. Provide those, and the tools to get the job done, and mostly it's a matter of keeping out of their way.
Jeremy Lee | Orinoco
From the post I see you're hired as a sysadmin and it sounds like the additional duty of training other IT staff. What you can do will depend on where you are in the chain, if there's an IT manager and/or C position above you then you can only go so far. (Unless they're asking you to do their job.) As a sysadmin you have to focus on the servers and network IMHO. And in that regard I agree with several of the posters about going virtual. Ideally you need to store the VM's on a SAN of some type and plan your load so if some of the physical servers fail you can move the VM's around. I read thru a bunch of comments but did anyone ask if there's a security person on the team? Will you have to plan physical and digital security for this upgrade? Because that's a whole other bag of cats. Because changing from default passwords is one thing but having a plan to update every 6 to 12 months, where will they be recorded, and all the fun of creating access groups. If you have to upgrade the desktop side I'll give you two scenarios from my last 2 jobs. Company A) Purchases 4 years complete care from Dell. Without knowing numbers I know this is expensive but if anything happens I know all my PC's will be back up the next day even if the end user dropped the PC down the stairs. This org. purchased enough PC's to replace 1/4 one year, the following they purchased 2/4 more PC's to replace all that were running out of warranty. Company B) They purchased 3 year standard warranty on all PC's. Now here's the interesting thing. We performed an annual refresh in which we replaced 25% of the employees PC's. The only downside to this plan is we had 4-5 models of PC's and our image ended up having drivers for 10 models when you count desktops, laptops and precision workstations. And if anyone broke their PC between years 3-4 we had to refresh out of band so we had to have PC's on hand in supply. The advantage though is obvious, you balance the cost of the refresh and it runs as a continuous part of the IT budget instead of running the worry of being pushed off because THIS year wasn't so good for business. As far as training goes, well, I'm the tech that has to know WHY something works the way it does. So I usually end up annoying the teacher because I inevitably end up asking questions they cannot answer and then I have to figure out via the google gods (why can't I hard set 'Verbatim'?) I think the biggest problem you'll have with training is if you have techs who like the way the company runs now and don't want the extra work because they're going to fight you every step and/or do stuff half-assed. The best you can do is DOCUMENT. Figure out how things need to be and write the process/details and any tech who cares and wants to be there will be able to follow you. That's my 2 warped cents.
PS I guess I need some training, or my browser's screwed up. I had that typed up with paragraphs and breaks...
You need to have at least three hardware servers, all with lots of memory and disk, so you can have a primary and backup for production use and another for IT infrastructure development. If you're doing both VMware and OpenStack, which is not a bad position to take, you really need 5, two per hypervisor plus a spare.
Otherwise you're going to be spending your time keeping your Shiny New Virtualization Platform up to date, instead of spending it Virtualizing Old Non-Shiny Stuff, which is what you should be doing.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
I'd focus on the people side - figure out what who you're going to get rid of, who you can work with, and build good habits of working well with them while you hold down the fort - feel out your first few changes to see what kinds of resistance you get from humans (and from technology). How that goes will give you a feel for the possibilities of larger change.
For every problem, there is at least one solution that is simple, neat, and wrong.