Slashdot Mirror
Transitioning From Small Shop IT To Enterprise?
Imaginary Friendly asks: "I'm the 3rd guy in a three-person IT firm. We're good and we're expanding. Our clients range from three computers to 30, with our largest client having six servers. We can handle the work but, thanks to my efforts and love (or just luck), I may be signing up two new clients who have 200 networked computers each. We're spread thin as it is, and hiring competent IT staff has been difficult. We're now doing 60-hour work weeks, so re-education has remained passive. How do we transition from manual rebuilds and CD deployments, to full scale (proper) IT administration?"
Since 90+% of your clients will be MSFT shops, drink the koolaid. Get an MSDN subscription. Buy a big multicore machine, and you can all learn on VMs. Master all the MSFT (RIS, SMS, etc) and bolt-on tools (NetPro, GPA, etc). Learn to do everything the MSFT way, and you won't starve (at least until Linux takes over, if and when). In your spare time, look at alternative desktop environments for tip and tricks.
I want to delete my account but Slashdot doesn't allow it.
Install linux servers (Gento's dah bomb!) and desktops overnight everyone. Nothing evah goes wrong so you can fire all your support staff. The clients will be so grateful for cost savings and increased productivity you'll get lots of bonuses.
In fact you can replace all their servers with Pentium 200's - sell them on ebay for extra money.
outsource your work and call yourself the VP of marketing and operations.
First read before being smart-ass.
"..., and hiring competent IT staff has been difficult."
If you mod this up, your slashdot background will turn into a beautiful sunset!
I would seriously recommend turning away those bigger clients for now until you first get the staffing to handle it. You can try to pick up the clients later. Maybe they'll hire someone else, but there is a chance that they will be unhappy with that someone else and come to you. You don't want to take the clients on now and screw up and ruin your reputation. If at all possible, hire someone (or multiple people) who already have experience with larger networks and kill two birds with one stone. I don't think it is really worth it to give you advice here on how to manage larger networks. You've got staffing problems. You need to address that first.
-matthew
"THERE IS NO JUSTICE, THERE IS ONLY ME." -Death
With windows you can run a serious huge enterprise operation just like you run a home computer. with the sames security risks and catastrophies. gigabytes of mission critical and confidential information can float around on your executives laptops, to be stolen or lost and viewed by god knows whom. remember, if a box doesn't have a screen with a mouse and keyboard, it's not a real server. bonus points if a server needs a constant logged in gui screen to get important batch work done.
Everything that will be said here at /. will depend on what your clients are doing and running. I would tell you to setup a single system for each of your clients and make a flash archive of it, and then setup a Jumpstart server at/for each client and configure it with to load that flash image to the client systems for desktop support.
But if you havn't figured out already, this will only work in a Sun Solaris house. The same will be said of solutions to every other problem, they will be machine/OS/client specific. Without knowing the different client needs, there is no "Thou shalt not worship false Gods", commandments style fix to your question.
Also, exactly what are your requirements of what services you provide? Desktop, email, web access, web development, server development, OS services integration, Unix/Linux/Windows/OSX/IRIX/AIX OS installation/patches, database maintainance/optimization, database access, backups, etc., etc., etc...
What you should do will depend entirely on what you need to do.
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
On the non-technical side of things, formal Service Management is a must. If you haven't already, I would strongly recommend formalising the management of incidents, changes, requests etc. with something like ITIL. Without formal change management you'll get breakages caused by change and unhappy customers. Recording incidents (every incident) allows you to build up a picture of where your pain points are with each customer, makes it easy for billing, and if you get the same incident again, and you can look at the resolution of the previous incident for a head start in solving the current one.
You mentioned manual rebuilds etc. It would pay to automate this as much as possible (I'm sure you'll get some responses on this). Quality can often be equated with consistency. If you give your customers the same thing over and over they will know what to expect, even if it's only 80% of everything they need. They'll be much happier in the long run than if you give them brilliant service one day and crap service the next.
"And then I visited Wikipedia
I moved from a small pc shop to a larger company with about 50 people.. i am by my self.. but we are spread out alot.
... the simple stuff makes all the diffrence.
what you need to is to plann everything. train people that the locations to handel minor things and make them a fire fighting team.. no company is going to complain that you train their people to handel the minor issues so that they don't have to call you. try to make everything in rounds.. if problems can wait let them untill the guys schedualed to come by can get there and have his list and go about his job.
with a good work order system you can plan for the jobs and have job kits for your workers.. a check list
and if you can put this in place then hiring people to do the work is alot easier as they don't all have to be experts.
also set up remote admin and monitoring.. companies might fight back alittle but make it fit their policies.. because if you can see a problem and fix it before they notice that is a good thing. also if it is something that could be done remotely you don't have to send people out there..
and for the multi server people a single port KVM over IP connected to a normal KVM rocks.. they arn't cheap but if you are making money from them droping the 500$ for a single port KVM over IP isnt' that bad.. also you can get them with modems so you can dial into them.. makes remote admin easy.
make network maps and keep them where everyone can get to them so that you don't have jsut one guy that can work at a specific place because he is the only one that knows how it is done
just some ideas.. but always plan..
'...if only "Jumping to a Conclusion" was an event in the Olympics.'
Have to agree. CS degrees on resumes of people I've interviewed in the past year or so are badges of inadequacy in the same way a Network+ cert used to be.
Above all, get in early on quantifying the benefits of working with you. Track customer productivity/revenue before and after they get involved with your business - it'll make a world of difference when you try and approach the next customer.
Finally - best of luck! It is a tough old world out there and if you can't differentiate yourself from the squillions of other IT shops that exist then you're in the wrong game...
R
Check out http://unattended.sourceforge.net/
I have been using this at our companies for the past two years and has GREATLY simplified our redeployment strategy. If you have different clients who use different computer systems that all run Windows. Do yourself a favor and check it out.
Your main problem is staffing. I doubt the problem is actually in finding people, but rather that you're not offering enough to attract them.
Build a gold image build of your platform/s of choice and use Kickstart/Jumpstart to build systems based on spec files or flar images. This way all your builds are identical and reproduceable. Keep archives of your images as you change them in case you need to recover an older system build.
As for the applications beyond the OS, you need to learn packaging and package your own applications. Again so you can reproduce a given baselined system in 20mins should you need to rebuild it.
Classify your add-on applications into meta-packages so you can install a given role for a system via one package.
You simply must setup a network build system and a package repository before you take on the 200 server client or you will surely drown; if not from overwork, from the frustration of not having a consistent platform that you are guaranteed to know before you log in.
Look into unattended, (http://unattended.sourceforge.net/) an open source alternative to RIS. It makes it possible to use linux servers to push out installs via network. A good unattended install with post installation scripts should make life easier.
/i \\server\share\Deploy\Office2k3\PRO11.MSI TRANSFORMS="\\server\share\Deploy\Office2k3\OFFICE _2003.MST" /q
/x {xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx} /q
As far as application deployment goes, a combination of the psutil psexec.exe and msi installers can make life a breeze. For example, scripting an office 2003 install is as simple as remotely executing the following:
msiexec.exe
Since we're scripting, you could have a vbs script execute a number of tasks on machines by pulling a machine list from a csv sheet, and passing on variables like the machine name to bat files.
Application uninstalls are just as easy; look under hkey local machine/software/microsoft/windows/current ver/uninstall and find the msi installer ID.
msiexec
Hate microsoft all you want, but MSI's have made my life much easier.
Hope this helps...
Spoken like someone who doesn't have a CS degree. I prefer someone who has a degree (any BA/BS is fine, I knew excellent IT folks with English & History majors) over someone who doesn't, but its not an automatic exclusion. But my experience says on average they guy with the degree is a better employee than the guy without the degree. Self taught experts and paper MCSE's rarely have the depth of knowledge and ability to step outside the problem; its Linux r0x3rs!, replace all you Oracle servers w/ MySQL, and how do I mount an NFS share? I've never worked in a multi-user environment.
But I'm sure you will explain why Linux is Da Bomb, MySQL can do everything Oracle can, and how your limited experience outshines my 30 years of working with computers and 3 advanced degrees
I would look at something like Novell Zenworks. You can easily manage all of those servers including Windows and Linux right from a single server. Makes it easier to handle large scale patches and license management.
When all else fails, hire me!
First establish boundries. Our shop (large corporate) handles 40 unix or 80 windows servers per admin. We run about 200 desktops per field tech. I'm not claiming super-man status because we got to where we are by constantly moving to automation and standardization where possible. (Be sure not to standardize to the point where you choke creativity to death though.)
So on establishing boundries go to your management (as a sysadmin team) and say you can't handle the workload. No crime in that because if they don't recognize it you'll be gone in 4-6 months anyway. As a joint sysadmin/management exercise work out what you all agree are reasonable support levels. Get commitment that they'll hire to those levels. If they don't then accept the writing on the wall and leave. (Or don't leave and burn out after that 4-6 months anyway.)
Spend your 40 a week doing work. Take those extra 20 and drive through anything that will automate your shop and let you work with fewer people. If they say, "You have to do more with less", then leave. By definition you can't do more with less. What you can do is ladder-step your environment to where fewer people can do more than before (but note that's not more-with-less. That's more (work) with more (tooling).)
Good luck. And I don't think you appreciate how lucky you are. One day you'll be in a very successful automated shop and be pining for the good-old-days when sysadmins would crawl into a burning office and fix a crashed disk with their teeth and save the day because the sysadmins of today just don't know how easy they have it.
Cheers,
-CZ
Seriously, I did this. I work on a 2 person team that handled a 500 user account for a local IT services company. When I left they hired some kid out of HS with his A+ for 1/2 my salary. Nothing but troubles after that. If you pay them, they will come. Kill some of your margin by hiring good people and you will gain the respect of the CIOs you contracted to... the same people who go golfing with other CIOs and complain or praise the companies they hire.
Religion and politics, without the flame. godgab.org
Given that each new machine will take some worktime, say 10 minutes/machine/week if you fully automate and have good plenty of spare equipment on service contracts, 200 machines is 2000 minutes/week, or easily 33 hours/week. You obviously need new staff to handle this: the first hire you make should be a competent contractor who's done seriously automated network build systems, suitable for CGI or Beowulf clusters, to reduce the load of OS imaging new machines or rebuilding your existing hardware into some kind of standard.
/. could recommend someone in your area to help you get past the hurdle of that initial auto-install or auto-management setup.
Where are you, physically? Perhaps someone on
Can you hire the current IT staff at your new clients? Or are you taking them away from a competitor? Do you guys specialize (somebody's the network guru, somebody's the server wrangler, the third guy's the application specialist)? If so, then maybe each of you should hire an "apprentice" and train them in your specialty. Consider trolling around the local community college with a decent IT program (if you have one).
One thing to do is compile a list of best practices for your shop. This will help communication among the staff (by establishing both common procedures and a common vocabulary), which is very helpful when you're trying to bring a lot of new guys up to speed. Even if it's something as simple as installing an OS on a fresh machine, document it. If you've got any common tools you use or wrote for fault isolation or troubleshooting, document them. Plug it all into a wiki or similar so everyone can access them. Communication is the key for keeping everyone in sync when you're in a rapidly-expanding environment.
Just junk food for thought...
no text here
You can't just have good customer relations management anymore, you need to to be a good corporate citizen with a net societal benefit and cultural ans social corporate conscience. Additionally, you'll need to make sure that all of your customers are liazoned by a single point of contact within your business to maintain a positive balance of customer goodwill. You'll probably also want to leverage centralized customer portals to meet your event horizon and allow you to expand your vertical market segmentation for optimized recurring revenue opportunities.
I worked for a VoIP company that did way to well way too fast. I was the lead provisioner and built the process. Being a very competent well rounded Geek I was doing customer service, orientations and provisioning all at the same time but pushing 60 hour weeks with not great compensation. And In hindsight if the Management had hired up more people and made service and excellence the priority they would have done even better, and good people would have avoided burnt out.
- DenialX
Give Noc Monkey a try. I used it on a fairly large network that was all *BSD and various Linux distributions. Works great, just pxe boot a server and it'll grab it's configuration. Of course this all depends on how your network is designed and implemented.
<happiness>beer</happiness>
You have already recognized you have limited time and it is hard to find versatile self starting quality IT people. The next step is to prioritize your clients (people are going to disagree with me here), but the clients who pay the most and who pay on time (collecting money can be a huge time hog) are your top priority. You are switching client profiles and larger clients are more valuable. Trying to get the next big client will be easier if you can say you support 2 200 users bases instead of Mom and Pops Juice stand.
If you don't have time for some of the smaller clients, the ones who have random, but few requests, don't pay the bill on time, etc, dump them. Now you don't have to be mean about it, you can partner with another IT firm and move them to there (or outsource if your trust their quality) or even hire a "lesser" IT person to handle the small clients. I think it is best to disconnect yourself from the client and let them know you can no longer help them; you can help them find a different IT firm, hire someone, etc. This prevents you from possibly being associated with bad IT support and dealing with the associated headaches. Also on this note you need to be able to recognize which clients you cannot support adequately after you prioritize them, or else again you could be associated with bad IT support.
One last final hint, hire a competent receptionist/administrative assistant. Someone who can remind you of forgotten appointments, schedule people effectively, and calm an angry customer; this will be extremely valuable.
Oh and one more thing, keep that luck going, you need it in the IT business.
Hire someone who knows corporate IT.
:-)
Please.
I've dealt with too many "three man IT shops" who treat IT work like auto mechanics. "try it, tweak something, try it again, tweak something, try it again, tweak something, try it again, tweak something". All the while, the company is offline. Corporate IT is about establishing procedures BEFORE the issues happen and about having backup plans for WHEN they happen, all of which is designed to minimize downtime.
Working with an office of 2-3 people... if you're diddling with their router for 2 hours, your time is probably worth more than the time the company has lost. But if you diddle with 200 people's connection for 2 hours, you've just cost their company $20,000, possibly more. Imagine what sort of investments could have prevented that downtime, how much cheaper they are than that downtime and why you should have implimented them
FYI, Documentation is more important than you think.
Stew
There are 10 kinds of people in the world. Those who understand binary and those who don't.
I can heartily recommend:
Bootstrapping an Infrastructure.
It describes good design patterns for computer infrastructure design.
Develop a knowledge base of solutions so that your newer staff can learn from your experts. Do it in Sharepoint and make it available to your clients, so you can demonstrate your expertise, and possibly save you time in diagnosing issues. Develop a culture of learning and pay your staff to learn. Offer paid certifications to get motivated staff, but tie them to a 2 yr agreement to stay with you or they have to pay back the certification costs.
Give your staff every technical toy they want, starting with a GOOD laptop and smartphone. DONT go cheap, because you need these tools to last. Build an MSDN laboratory network with a separate domain strictly for the purpose of testing and learning solutions and troubleshooting issues. Anything your clients run (within reason)should be testing for compatibility on your test domain. If you solve problems before they become problems, you wont have to worry about clients.
Want to weed out bad staff? Buy some Transcender tests and make applicants take them. Their score will determine their place on your salary range if you hire them, and if they cant pass a practice test, tell them goodbye or to take a very low rate in exchange for training, but ONLY if their resume was honest about their skills.
Just remember, the more you document, the less you have to explain. Good luck.
Businesses love to complain about how hard it is to find employees when they're being cheap on labor, or how they can't retain good help.
There's no talent problem; there's a "how the IT industry treats workers" problem. Here's the current IT talent pool "problem", as I see it:
Is it any wonder that IT staff leave the industry in droves after just a few years?
Please help metamoderate.
I can make that all happen for you.
Hi, I am solely responsible for a 200 user network over 2 floor, and assist in the support of a 100 user network in another city. For the first year, I busted my ass to implement my 3 point program. Those points are:
1) Standardization
2) Automation
3) Redundancy
1) Standardization.
The more they are the same, the easier it is to administer. Try to have the same hardware and image for each department. Use ghost like its going out of style. Use standard naming conventions. The borg naming convention is NOT standard. Keep all images as close to the same as possible
2) Automation
Use all tools available to you. If its a microsoft network, use Group Policies, login scripts (vbscript, launched from the GPO), and roaming profiles. Make sure clients are auto configured when they login for the first time (use Office Deployment wizard when doing the ghost images). Make sure you lock them down as well, so they can't break anything. This should reduce the overall client administration to nil.
3) Redundancy.
Virtual Server. Download it, install it, embrace it. Setup a minimum of 2 virtual host servers, preferabbly 3. Make sure you have a large capacity nearsite backup and have the virtual file systems backed up nightly. That way, if one virtual host server has a hardware failure, you can bring the servers back up in a matter of minutes, not hours.
Thats a real simple summary of how I am able to single handedly support 200 users, both client and server. If you have any more questions, please contact me at earlaker at lukewarmmail.com (replace lukewarm with something warmer)
Feed the need: Digitaladdiction.net
As stated earlier, figure out the base system & apps, ie Win XP, Office 2003, Acrobat reader, ... set-up printers Then make a ghost image. If you have enough hdd per machine, split the hdd into 2 partitions and install the ghost image on the second image, you can use gdisk to hide the second partition from the general user. Make sure that the users map all home directory's to a server(roaming profiles in action), then if a machine fails/gets infected by a virus you can boot off a boot cd, unhide the partition, ghost the first partition, rehide the partition. That should give you the ability to restore a machine in 5 minutes.
If you spend some time, you can even automate the whole process. Inventory all of the errors and mistakes that occur on the sites, then create a standard solution for solving it. Once you have that, teach the user so they won't repeat the mistake.
http://www.infrastructures.org/
Some of these essays are a bit dated especially with regards to the tools they are suggesting but the mindset is still quite sound.
Work on getting a working relationship with vendors, contract out a desktop hardware support company to manage the hardware breaks, contract out a helpdesk function. So that should leave you and your very small but focused team on dealing with managing the vendor and processes, and all of the projects relating to this small company (hardware refresh, software deployments, etc).
:)
I say small because 200+ workstations is not the enterprise, the enterprise is something much much bigger (talkin multiple platforms, across thousands of desktops, across multiple timezones, and that's a small enterprise infrastructure). But, you can use automation tools from enterprise class platforms to help with software deployments (altiris), patch management, remote access(citrix/softricity), etc..
You get that all down pat you can have a small team to keep all the money!!
MrJynxx
i think you can easily handle the large networks, if you minimize the amount of work you do for smaller jobs.. cut out the time consuming jobs that don't make you much money.. if you stay on top of the upcoming technology, give solid timelines and quotes, you should be able to manage.. i hate to say it, but i've been Pro-Linux for years now.. and even the things i've done time and time again, sometimes takes reassurance and research.. if a company is willing to spend thousands for their network, you might as well set them up with a Windows server.. you didn't necessarily say how exactly you set these networks up and what software you use so there's only so much input the Slashdotters can give.. Windows ultimately saves you time and money because SBS and 2k3 work incredibly easy, and there's only so much documentation you need.. and the employees that utilize the network, enjoy it.. if they don't, then you didn't set it up right..
one more small thing to save time, is to do as much remote administration as possible.. being that you're only a 3-man operation, I personally don't see any problems with remote assistance.. some people might shun this, and say that no company can trust a computer guy to have remote assistance whenever he wishes.. but c'mon.. if you're only a small company with lots of work to do, are you really going to have time diddly daddling with their personal data? of course not, and why would you want to? most IT guys just want to get the work done so they can spend as little time with the customer as possible.. it avoids having to make any time consuming unnecessary trips..
I say keep it small, don't hire any more people unless you need them to help you pre-setup equipment at your shop.. stay focused on the larger networks, and stay organized..
*plays the Apogee theme song music*
Just move em all to an i5........1 server...multiple O/S...and the last time we booted ours was over a yr ago.....
and don't do anything you see there!
"and there's tons of qualified people out there. "
I think your definition of qualified might be different from mine, because in my experience there is definitely not a surplus of qualified people out there.
I don't always use unix-like operating systems; but when I do, I prefer FreeBSD.
200 pcs with 6 servers- 3 guys= easy
current helpdesk metrics indicate that 1 tech with a well run management system can handle 320 PC's.
sounds like you need network management tools.
Do NOT play LAB in a production environment.
..your customer's their data, ...your company revenue and ....you your job.
aka
Do not LEARN on the job. LEARN in class or in a proper lab... not during an outage or during an install.
That is proper IT, anything else is flying by the seat of your pants which will cost
Experience is a necessary but not sufficient condition to not making newbie mistakes, and by that I mean decisions that sound logical and intuitive/plausible until you actually try them in the real world. Experience is very important. Yes you can have a very bad mechanic with 17 years in the trade. But I guarantee you no one in their right mind is going to let someone work on a multi-million dollar race car with 3 months experience just because they're intelligent. You need both.
As for scientific discoveries favouring younger minds, there's an easy explanation for that. You have thousands of inexperienced youngsters that try lines of thought more seasoned people would never think to try because they run counter to all they've been taught. Most of these youngsters get nowhere. The few that are both smart and lucky enough to try the RIGHT technique that goes against the ingrained way of doing things are hailed as superstars. When you've been staring at the same problem for 20 years the mind tends to fall back into the same patterns. Agile minds isn't the whole story though.
These posts express my own personal views, not those of my employer
Do you want to keep doing what you're doing, or do you want to be a manager? Being the manager is more lucrative if you're successful but you're not going to get to both play IT admin and manager of a large company. You'll be spread way too thin.
The job you don't want to do, you need to hire for. If that's the mangerial role, you need to make sure you don't end up a lowly surf by managing the hiring process very very carefully to ensure you retain control at all times.
If you are unable or unwilling to find any decent new hires, you'll get no where, possibly end up being sued, and you'll burn out doing it.
These posts express my own personal views, not those of my employer
Hire ids right out of "tech schools" with as or a few certs and put them in teams of 3 or 4 with a senior guy watching their work.
there are thousand of programs for it around the country and 100,000 of kids coming out of those programs every semester.
I'm currently have one more semester till my A.S. in Info Tech with certs in A+,net+, security+ and plan on taking ccna in the next few months. I just sent out my first real professional resumes for a job starting out at like 11 or 12 bucks an hour to get my feet wet.
my point is there are masses of more kids out there just like me eager to work and dot expect astronomical salaries.
get in contact with instructors and professors at collages in your area. They can send student to yo. Even if those "kids" are just installing software or Basie maintenance it will still free up your senior tech to do real much needed work.
and if there any one with out there near central KY looking for a decent teck guy for heap I'm looking 4 a job!
I think you might have missed the point the grandparent was making - possibly because you're a Unix admin, and not a Windows admin.
The Windows GUI is inseperable from the OS, and has to run on the machine's built in video adapter and keyboard. You can't fully administer it through a console connected via RS232, for example, without hardware hacks to virtualize the video. And as the GP mentioned, many operations tend to require or at least strongly encourage a session logged in on the server's console. This is all pretty strange, when you think about it.
It's not that hard, I've been in the same boat.
Step 1: Think of the coolest things you can do to make things quick and easy.
Step 2: Implement it.
Step 3: Document it.
Really, whether it's thin clients, giving all the pc's bootable nics, setting up network images for reimaging, using VNC or other remote service/admin tools... think outside the box and then do it. Because with 200 people you're not going to have time to walk to every machine, nor can you spend time bringing the boxes in for simple reimage.
And don't mind all the people saying "don't do it". The truth of a small business is that you have to have the business (and therefore the revenue) before you can hire the people. Just make sure that, no matter how qualified they may be, you stick them with the menial tasks for a while, just to make sure they're up to snuff, and also that they don't screw up your shiny new contract.
Step 3 is, of course, "DOCUMENT". Write out your procedures for dealing with everything, from single user failure to sitewide issues, new virii outbreaks, everything. And most importantly, write up a SLA (Service Level Agreement) that outlines how long your targeted fix time is for certain levels of issues. Site down? Fix in 1 hour is the goal. User wants a blue mouse instead of a navy one? We'll try and do that in a week.
Having a system to record incoming requests, categorize them by priority, and then give an ETA, is going to be a valuable part of your long-term success here. A few users are easy to handle. Get to 200 and you start getting the people who are whiny, pushy, or just plain don't want to work and can now blame their computers for it. The ticket system pays for itself the first time some peon goes "Of course, Mr CEO, I haven't done any work all week, those rotten computer people haven't fixed my problem yet!", and you can go "well, they never told us it was broken, see?". Trust me, it'll save your arse more times than not. Yes, it's a pain having to do it for the easy stuff, but in the long run it's better for the clients and for you.
Once you've started setting up the cool stuff, you have some peons hired to take the day-to-day load off of you, and you have some system processes and documentation going, then it's time to start the fun game of "What if", as in "what if the network rack caught on fire?" or "what if some user decides to plug in 120VAC into the phone jack?" or "what if our main service provider goes tits up?". Make contingency plans to handle everything, no matter how crazy it sounds. It doesn't have to be 100% service, it just has to be enough to keep the company going, whether it's access to the mainframe or incoming phone calls.
One of my most ludicrous plans was "What if all network activity died in a 5 block radius for some reason?". Our contingency for localized network or telecom outages was to borrow from neighboring businesses, but what if it all died? Yeah, I thought it would never happen, too, until lightning hit the main telecom service station, and in the process of doing the rush fix, the engineer fried the whole thing. Sure, maybe it's just a fun afternoon shooting the shit with your buds thinking up crazy scenarios, but maybe it's the fix that saves the day.
My answer, by the way, was to spend the extra dollars to make my on-call cell GPRS instead of just local cellular. While everyones network access was dead and telecom was mostly screwy (cell and POTS), I linked the phone into a laptop, did a little connection sharing to the switch that connected the 4 or 5 vital stations, and got them dialup access to the text-based mainframe in another province, and a little email as well. It was slow, it was only for an afternoon, and I really don't think they did anything overly important, but that little trick kept that account for me until I closed shop.
Oh, and the ever-forgotten Step 4... don't forget that it's OK to make profit. You're doing a lot of work for these people, 200 users usually requires 2-4 full time techs, at a cost that could be 40k - 80k each. Save them money, but make enough for yourself too.
If I knew the wedgies I gave you back in 6th grade would have resulted in this . . . I might have taken a moments pause.
I am a senior Unix sysadmin for a large company with a huge and diverse IT infrastructure. Name OS type, version and patch level and there's a good chance we'll have an exact match. Obviously, it takes a huge IT organization to support this inefficient diversity. Best and brightest IT infrastructure consolidation experts hired by our company over the years have failed to make any difference. There's is no way to get the worms back into the can, if they just got out of the biggest can there is. Don't be greedy chasing after clients and try to avoid too much diversity.
:-)
Hire experts. You don't want to save money on the quality of your IT staff. One grossly overpaid expert can do the work of twenty point-and-click "sysadmins" and still save you money. Train your staff. Regularly send them to training classes. Don't be afraid that they may leave you. Eventually most of them will leave you no matter how well you treat them. But try to keep them for as long as possible. Training is the key. There is no substitute for a sysadmin who knows what he's doing.
To summarize: stick to Solaris and hire Russians
1. Shell Scripting, can your person make changes to dozens of machines at one time when you have a crunch.
2. Obscene famialiarity with the OS.. A headphone jockey can walk a user through hell and back, if they know exactly what is popping up and why you can save a huge amount of time. If theyre good they can walk someone through a network stack rebuild with their eyes shut.
3. Wicked soft skills. With the right phrasing "Your machine is going to be a bit odd for a couple weeks" becomes "Just click the new shortcut, until we can get a nice block of time that more convenient for you." And you have suddenly moved from frustration to elation.
4. Good Work attitude, If the person is a pain in the rear to work with it will be a disaster.. Screening for the bad-neurotic is a hard trick. Probe the three innapropriate topics of conversation, Unix-Windows-Mac can be used for religion, but if the applicant keeps rambling in any direction, point them at the door.
5. Brainiac Troubleshooting.. If the person can manage to systematically reduce a problem to smaller problems then you have someone who can tackle some of the larger issues of a site.
6. Experience.. There are quite a few technicians that have already supported these issues If you manage to grab a Technician that has worked in both the homogenous and the heterogenous environments, you can find some pearls of wisdom on handling the issues with each.
Take some real time to screen the applicants.. sure 90% are going to be junk and of that you need to spend a bit of time making sure that your not going to go insane working with them. Check some references dig a bit.
Storm
Microsoft has tons of tools that are pretty good at managing large enterprises with few people, but its helpful to know something about Windows scripting, and locking down Windows to prevent user misconfiguration.
400 or so desktops might be one or two people at worst. As to the os install bits, I did that gig 15 years ago and had automated remote installs with one floppy, PXE gets rid of the floppy. I mean it the oh no my machine got eaten by a virus should be no harder than a please reboot hit f12 and go get a drink. In the background you should have setup the machine to PXE boot into the virus scanner image and exported it's VNC session to your desktop across the site to site VPN. Find anything that you dont like and ghost to box from the same image. If your not there yet your not working efficiently just longer and harder. There are four types of IT admins ones that will do the same thing over and over, one that try and automate and spend more time automating than they saved, one that try and avoid doing any work and get others to automate it for them and ones that automate and save time; hire as many of the last type as you can find.
As to your lack of finding good people, get cheap right out of school labor, put them to work taking tier one stuff, things that you can script and take the load off the guys above them, the good ones will start scripting and automating to make there life easier. These guys are the cheap labor that will a good environment will excel, do make sure you have a test lab they have access to cheap vmware server on top of Linux works wonders.
No sir I dont like it.
Well its a tough business out there. I work for a consulting firm that has doubled in size with clients ranging from 5 to 75 desktops. We have a number of clients who have larger server farms as well as clients with remote sites that also have a server or 2. In any case, you need to evaluate your current business plan and see if you planned for growth. You will need to start to look for personnel to handle the in-house operations. You will need a business manager to start organizing your business end of things. What you will also need to do is figure out how to keep your current tech staff current with upcoming technologies. Frankly if you are asking about ways to go from manual rebuilds to enterprise deployments then you certainly have a lot of catching up to do.
What I see happening, if you continue on your current trend, you will burn out for one. Then you will start to lose your current customer base due to lack of support. We have gotten a ton of new clients due to the lack of support from their old company.
Also you may want to look into hiring a consultant to help you plan ahead. You will need to drop some clients and get back to working normal hours.
As for hiring talented individuals, well remember, certs aren't everything but they do help clear up the piles. Look for experience, you will need people who are good with networking, Server systems, hardware and desktop support. Learn how to monitor systems and prevent major fires from happening. Thats always the best plan. As for helping with desktop deployments, there are a ton of apps out there that can help as well as built in services from Windows server solutions.
Good luck!
Dewser - all around techy "In the immortal words of Socrates - 'I drank what?'"
Here's a secret about big corporate IT deployments (I've worked in a couple very large ones, and observed a few more belonging to equally large "partners"): The tools they use don't really save you that much work.
You wind up maintaining solutions to help you maintain your solutions. You train people to train people. You automate your processes, and in return you get to create a whole slew of new processes to oversee the automation. You lose techs thanks to the new "productivity", but you gain project managers, and regular managers, and admin staff of all other kinds.
What I'm getting at is that scaling up your processes isn't going to magically help you do more with less. I believe -- wholeheartedly, or I would have gotten out of this line of work a long time ago for a nice part time career at Blockbuster Video -- that you can do more with less, but it's not going to be looking at how the big boys do things, because they're just as fucked up as you are.
Look at your business. Try to improve its efficiency based on the intimate knowledge you have of what you do on a daily basis. Just like refactoring code, anywhere you see yourself doing the same thing over and over, see if you can't find a way to remove that redundancy. Go in search of specific solutions for the tasks at hand.
Game... blouses.
FYI, Documentation is more important than you think.
And just as important, a *good* backup system. I had to fix a system down issue not that long ago where a configuration file had been strangely corrupted. Sure, I could probably find it somewhere in the system documentation what all the settings there was supposed to be, but it was a lot easier to check when it was last good, fire up the backup tool and say "I want file X from last friday". It's sort of like documenting code - for the love of Christ, document when you do something non-obvious, but the more of the code you can make self-explainatory, the less grief you will have. The code is the documentation that can never be out of date or wrong (the code might be buggy, but not wrong as in "different from the code").
Another classic is to have some sort of log of what's been done with the system. I recently had to deal with a system that hadn't been moved properly from one environment to another, but the whole move/inform/test routines were all apparently lacking. Or the database people who decided to tighten up security, removing vital access rights to stored procedures. While they still might have caused problems, if you'd start out with a clue that system changes X and Y has happened at least you wouldn't go "WTF this isn't working, what's happening?". Particularly when these sort of issues come up right next to deadlines.
Live today, because you never know what tomorrow brings
Just for the record, I am NOT an MS fan-boy. But, I do know what keeps me in hot dogs and beer. Learn about the resources Microsoft provides. Embracing the dark side instead of fighting it will save you hours of headache and make you a hero. Active Directory is a huge time saver in any shop with more than a handful of computers/users. There is a ton of stuff you can control with group policy. The number of policy elements and the granularity of control keeps growing with each OS release. TechNet has some good resources but you have to dig for them sometimes. For systems management, Windows scripting is a must. VBscript is not too difficult to learn. Microsoft provides scripting interfaces into Active Directory and the Windows management interface (WMI). Almost anything you can do in AD or at a computer console can be scripted in VBS with the AD and WMI interfaces. There is a very active scripting community on the TechNet web site. Lots of good sample scripts and documentation. Plus links to various webcasts. You can literally spend an hour a day for a week listening to the webcasts and learn enough scripting to save twice that much administration time every week. Deployment is a bit of a weak spot. RIS is complicated and quirky. You can spend weeks just getting the basic system working then not be able to use it because your new hardware doesn't support the bootp environment. SMS is nice as a systems management tool. It does have OS deployment capabilities if you jump through enough hoops. But, it will cost you around $10,000 for a small deployment and months of learning how to make it work right. (It is very NON-intuitive) If you have somewhat standardized hardware and software I would use Ghost. The trick is to make sure you document all the components involved in creating your standard image. It doesn't need to be a click-by-click instruction, just a bullet-point list will do. (Install this component, disable that one) Include just enough detail you can blast through the install on a new hardware platform and have it come out the same as previous machines. Do the same with the user configuration. Once you have a good image, sysprep it and Ghost it to an image file. Rinse, lather and repeat for other hardware platforms. There are a number of ways to deploy the images. Boot to CD or external hard drive works well in smaller environments where you might not have a dedicated image file server. Use a generic CD driver on a boot disk or check out BartPE (works well with external HDs).
The Practice of System and Network Administration by Thomas A. Limoncelli and Christine Hogan is the definitive reference to build, and more importantly, maintain any network and system infrastructure. It is written in an accessible style with plenty of real-world examples that focus on the importance of key infrastructure. It is not a "How To" book exactly, rather it offers advice and specifications for the kind of support infrastructure you have to build to be successful supporting large system and network infrastructure. If you are familiar with this book, please add your comments on it.
My God! It's full of Voids!
Like the other poster said, use an iLO card (or your vendor equivalent). It's certainly saved me a 4AM trip to the data centre to powercycle a box and look at a screen.
Not Meta-modding due to apathy.
Centralized Computing.
I have a similar situation in a two man shop where our largest client has gone from 2 to 300 users in the past two years. The only way that we have been able to keep up with the management overhead is by implementing thin computing. We have six clients, and a total of about 25 servers (including our Citrix Farm), but I am able to only work four days a week and my partner only three.
This may be flaimbait here, but we also are almost exclusively Windows on the Server and Desktop (even our thin clients). God knows I love Linux and put it everywhere it's feasible (which has only been edge of network so far), but Active Directory and Citrix allow me to have my business and my life.
The only thing I'd add is that you become familiar with the remote desktop takeover tools. Sometimes it just helps to see what the other person is doing.
When our name is on the back of your car, we're behind you all the way!
My crew is in the same boat. I am the fourth tech and we are growing quickly with large new clients. We don't want to destroy the magic our team that has produced and the resumes we were getting were not great.
We have had one of those big AOL call centers shut down in our town and one of my friends is a GM there. I had him collect resumes from their IT staff. I feel bad about capatalizing on the center's misfortune but at least we might be able to give someone a job who really needs it.
It may some mercenary but if there are any big companies pulling out of your town, they might be a good place to look. The bonus is that you will get someone who is already familiar with large corporate deployment and strategies...
To grow, you're going to be forced to be more specialized in your job functions. Find an experineced systems administrator who has worked in big shops and has experience working with software developers. This persion will be expensive if they're any good, and will be worth the money.
Above all don't pretend that you know how to manage IT infrastructure. Systems and infrastructure administration is a compeltly different world than most developers ever work in. Your automated systems and production environments need to just work, and your development environments need to quickly recover from breaks. You don't have time to sit around troubleshooting either one when there's work to be done. In my experience most developers want to tweak things all the time. This is a good trait for the otherwise tedious craft of writing software, but must be resisted in production. Sometimes there's a reason for a sub-optimal config. Study it in a lab if you like, but don't touch production until you're sure.
Also, software development is creative work. Administration is less so, and is more constrained. Patching systems on off hours is not like writing software when the inspration is greatest. There are time constraints and procedures to be strictly followed.
Systems Administration is the art and science of maintaining an unstable equilibrium of resource availability versus maintenance and optimization. experience and temperament are important.
Finally, in order to support the increased infrastructure necessary to meet your client's needs, you will need to find even more clients. You will probably need sales staff eventually. One or two big customers is a time-bomb. If they go away then you have to down-size, and if they find out that they are so important to your bottom line, then they can begin to dictate your priorities. The only defense against both of these is to have enought customers that the loss of any single one won't kill your business. This is not a step to be taken lightly.
Two comments:
Be honest with your new potential clients - if they still want to work with you as you transition, they'll be more understanding.
Second:
I remember when my team, which started out with me as a contractor, then me as a full-time staff member, then more people as time passed... reached the right level of staffing and were able to move from what I call "reactive mode" where we were constantly rushing from one emergency to the next, to what I call "proactive mode" where we were actually addressing potential problems before they became problems. Obviously, there is always going to be some surprises, but once you're adequately staffed and trained life is soooo much easier.
Try a hiring agency for work-to-hire people. That way, you can overstaff a little while you get caught up and trained, and then when your head is above water, keep the best of the bunch and let the other's contract run out.
Work-to-hire contractors often are the hardest working bunch, and they're highly motivated to keep their skills up. And, even though they seem expensive compared to hiring your own, it's not too bad of a deal since you don't have to cover their taxes, benefits and etc. So, for example, if you'd pay an employee $24/hr and instead you're paying the agency $33/hr you're really about breaking even because of the hidden costs of having an employee.
Comunikashion scills R empotant.
I hope you're cover letters and resumes read better then that post. Otherwise I foresee geek squad in your future.
Also never mention having an A+ cert. That's like claiming to be skilled in 'butt picking'.
John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
I'm currently managing 2000 workplaces, 3 mainframes, 50-some servers with about 15 people.
We didn't have any trouble or asset management except for what came with Active Directory and right now I'm implementing a central command, asset collector etc.
I can't elaborate on it due to NDA but if you're looking for someone to take care of your IT support management problems, contact me, I might be able to give you some tips.
Custom electronics and digital signage for your business: www.evcircuits.com
This home brewed and self taught IT professional cut his teeth on starting a local ISP with 2 partners while in highschool. Selling it, then bouncing around for a while (trying to find my fit in the industry) and then starting up a shop doing outsourced support and consultation for local hospitals before finally deciding to settle into an enterprise situation for stability during the first years of my marriage. However I receive a lot of respect in my area for not only my technical ability but for my adaptability. I also receive respect for the respect I give to others, titled or not. Experience is an amazing teacher, but unfortunately it can callus us and we no longer feel the changes around us. 30 years and 3 advanced degrees, unfortunately it didn't open your mind up any, or prepare you to accept change and avoid becoming callused... I guess I didn't miss anything by skipping college after all. However I have delt with plenty of viewpoints similar to yours, in the end it boiled down to that person feeling threatend by me. Which always gave me a better insight into their mind as I myself harbor very little ill will towards or for anyone, I can tell when it's harbored for me.
I second this. It is hard as hell to find anyone competent with REAL WORLD EXPERIENCE for less than $75,000 a year. There plenty of guys out there who you can pay $50,000 and throw into the breach, but as soon as they have to recover a corrupt Exchange message store or bring up a crashed domain controller, they're going to end up causing more problems than they fix. And forget about it if they have to troubleshoot anything outside of the server (like a switch or firewall).
You can do it, sure, with the right hardware hack, as I mentioned in my previous message. The point is, you shouldn't have to, and it's primitive, single-user-oriented software that makes you do that. That bias is pervasive, affecting the way software for the Windows platform works.
BTW, I'm not some Unix zealot. I grew up on DOS & Windows and have been using Windows 32-bit versions since the first betas of Windows NT and the Win32 API. But Microsoft got lazy, and stopped improving their architecture.
What I do now to optimize administration of servers is use Debian.
I work for a large corporation as a Unix/Linux system admin. While I only manage 25-30 server systems we have hundreds of customers with thousands of systems, and a couple hundred system admins.
A proper change management system is crucial, as well as well documented processes and best practices for maintaining the systems. You cannot hope to manage dozens or hundreds of systems without documentation and change management! http://www.infrastructures.org/ is a good resource to get started. Yes, its a lot of work, but its best to get your support infrastructure ready before you really take off with supporting more systems, so you aren't working those 60 hour weeks until you all burn out and quit.
Never underestimate documentation. Keep both copies that are on a secured internal system, and a hard copy as well. Update it frequently and make sure when you make system changes, they're documented in your change management software and the documentation. Wikis are great for documentation (its what we're using now). I've heard some people had a lot of success with Bugzilla for their change management. We have to use an internal, proprietary company package so I don't have experience with it for that.
And as a previous comment stated, you'll probably need to hire someone that is familiar with corporate IT. You might even need to hire more people that have specialized skillsets, instead of people that wear every hat.
That isn't to say that turning down work makes you successful, of course. Once you get to where you're turning away work, it means you can be selective. Also, you can start charging more.
BTW, if you're already doing 60-hour work weeks, you had better be making some really really good money (*), because the alternative is that you're already losing the game. What if one of you comes down with a flu that puts you on your back for 3 weeks? The other two gonna work 90-hour weeks? Doubtful. 60-hour weeks are for once-a-quarter crunch time. Right now, you're eating your seed corn.
(*) Even if you're making good money, it's probably wrong. The way to riches in computing is to identify an essential niche you are talented in, and raise your rates. 60-hour weeks make you 50% more dollars. Matching your talents to the right customers makes you 3x as many dollars.
We're now doing 60-hour work weeks, so re-education has remained passive
Errr whats unusual about this?
This may sound jokey but I am being serious. Every IT job I have had (7 in 10 years) I have worked these sorts of hours due to a) bad management, b) stupid sales people (promising stuff that is not ready/spec'd out for a months time) c) understaffing d) poor staff that are so bad at their jobs they should be fired e) general incompetence e) etc.. etc...
From experience the good programmers end up having to pull miricles to get the work done and the bad ones drift along trying to look busy. Managers and sales don't realise that because it is quick to say this does not mean it is quick or simple to do and the work just keeps on building up! (as a btw I am sat at my desk at 6am, and I have a meeting booked from 4pm until 6pm tonight - so another 12 hour day).
I'm afraid this is the standard in the IT business - it's sad, but true
Jaj
Figure out how many people you want and what average salary you want to pay. Then cut the number of people in half and double the average salary. Nothing kills enterprise IT faster than semi-competents trying to cover-up their mistakes. Plan to spend at least twice as long as you think you can afford on interviews and checking references. While interviewing, ask some questions that are too hard - if the candidate can't comfortably admit that they do not know the answer, they won't be able to admit when they make a mistake later on. Never punish the admission of a mistake. Seek candidates who are happy to teach *and* learn from coworkers. If they think they have the answer to everything, they'll get stuck on one answer and deafen themselves to alternatives. Make sure there is always a tasty carrot and a big stick, and apply them justly - good people love meritocracies.
Stop-Prism.org: Opt Out of Surveillance
Not that I would consider 200 machines a ton of devices to manage, but dealing with a larger network is definitely a different story. Policies, procedures, scheduled downtime, documentation...this is your world. Just like the person above said, you're not an auto mechanic....you don't just go in and start monkeying around with things trying to fix them. VMWare might not be a terrible idea but it depends on exactly what you're supporting. You need to test things like patches and upgrades before they get deployed.
You said something about CD builds...we use cd builds with network based updates on a network of 25K managed devices, so cd builds are fine. One thing you may want to think about is centralization. Get the data off people's desktops so "fixing a desktop" can most of the time come down to rebuilding it. This will save you time. Spending two hours debugging a problem with an application that's only happening on one machine is a waste of everyone's time. This requires some retraining of the users and you'll never eliminate the need to back up data off the desktop completely, but you can still save a lot of headaches as well as creating network shares where you can get real backups going.
Definitely get REAL familiar with windows. People can say what they want about windows but for managing an enterprise network where there are novice users? It's the only way to go. Microsoft is dumb in a lot of ways, but there's a lot you can do with their software in the enterprise. That's not to say you shouldn't use UNIX where it's appropriate of course, but really take advantage of what windows has to offer because you know 99% or more of the companies you deal with will be running it and will not be changing.
VBScript and Wise Installer will be your friends especially if there are a lot of apps you will need to support. Even just for office...learn about msi and transforms and repackaging. The more you can standardize a build the better. One offs take time and knowledge to recover from. You want to be able to have the recent grad you can go out and hire rebuild the CEO's machine because it's just that easy to do. The more exceptions you make and bullshit you allow, the worse off you'll be and the more time it will take to support. If you have exceptions and one offs that aren't thoroughly documented? Your ass will be grass when that HDD dies or the app gets corrupted so don't allow them or don't agree to support them under your usual SLA.
Don't get ahead of yourself, because it's only 200 machines. That's nothing. Don't buy tons of expensive software and don't make yourself crazy trying to over-manage things, just work in the direction of standardization and centralization as it is appropriate to the client.
I'm going through something similar now with my company. Recently we've been bringing on clients who have grown to the point they can no longer manage things in-house, and need outside assistance. We've been going in, and across the board, they want things done "right", meaning we get in, map out their networks, and then suggest changes. The companies this works best at are the ones who understand that it's better if we stick to this mapping plan, don't make any changes ourselves, and then come up with good, feasible, recommendations for improving their systems.
The bad ones are the ones where a week into a mapping, they decide that things must be "done now", and then it's a slash-and-burn saga with an exploding budget and lots of outages. Usually after this, these clients are reformed and become good ones. Otherwise, it's just unrewarding money for us.
No offense, but this is an extremely ideal and unrealistic manner to approach this problem. Businesses need to learn how to grow when needed, especially start-ups. Clients don't like it when you tell them "well we can't handle it right now, try us again in 3 to 6 months." Thats not cool. Clients will usually take their business else where and unless their competitor drops the ball, chances are you've lost a customer.
Moving from the small to medium sized organization is moving to a different universe. Spending 2 hours troubleshooting onsite desktop problems may be okay for the 3-10 client staff. A 200-user network will have constant issues. A 200-person network will get away from you quickly. You should have tiered support set up for these sized clients so that you focus your staff and keep them happy. Doing it all is a burn and at already 60-hour weeks, you are going to deal with burnout and performance issues that squash the romance of growing the business.
Either take on the 200 person networks and tier yourselves or just stick to the little guys. If you decide to take the mid sized clients, drop the small ones or your efforts will be counter-productive. Focus will protect your reputation, doing it all will ruin it over time with growth.
This looks like a good place to put in my 2 cents worth...
Spend some time thinking about your strengths and weaknesses and plan your business accordingly. If you are a into small businesses then stay there. Enterprise is a very different animal than small business. If you are going to play in the enterprise space then you need to hire experts in big boy games.
Have you asked yourself questions like "Why am I willing to take on the new clients?" and "How am I going to provide better value than the other guys?"
Frankly deciding to switch from SMB to enterprise sounds like a bad idea. It would be like a star gymnast deciding to play pro football. Just because you are good at one sport does not mean you will thrive in another.
Good luck,
Dan
This is just the kind of attitude that is ruining the IT industry, I have worked in corporate IT for 17 years now, and each year it is getting worse, with all this documentation, processes, change management, problem management, ITIL, blah, blah, blah, its just a cover for people that have got into IT and don't have a clue as to what is really going on. Sorry if you need everything documentated down to step by step, screenshots every time you click next then maybe a job down at the local supermarket is more your thing...
I suspect that he means "Has a CS degree" AND "Knows what the fuck they're doing".
People with CS degrees are easy to find, people who have them and know what the fuck they're doing are not.
LK
"Hi. This is my friend, Jack Shit, and you don't know him." - Lord Kano
But I'm sure you will explain why Linux is Da Bomb, MySQL can do everything Oracle can, and how your limited experience outshines my 30 years of working with computers and 3 advanced degrees
30 years in the field and 3 advanced degrees, yet you can't figure out how to log in to Slashdot?
LK
"Hi. This is my friend, Jack Shit, and you don't know him." - Lord Kano
Come on, dog - this is not a tools issue. It's process that will set you free. Good release, change, and config practices are essential in any environment - and can work with any tool set (even tho you may decide some of your tools suck and change them, any tool will suck in a crappy process). Remember - it's a poor craftsman who blames his tools. Check out the Visible Ops handbook. I got a free copy from the dudes at Tripwire at a SANS thing but you can get it on Amazon, too. Solid.
There are three key things that you need to do when you want to manage an IT shop (for anyone):
#1. Have a ticketing system in place or built. This is probably the most crucial thing you can do, because it will become the core of your job. Nothing gets done without a ticket, period. The ticketing system will also allow you to document (1) the problem, (2) the time spent on the problem, and (3) your interaction with the customers.
#2. You need to do an audit, so you know what services are being used (and required) by your customers. That means you need to be able to capture, at a minimum, (1) the machine configurations, (2) users & permissions for each user on each machine, (3) services provided for the network, (4) the network topology, (5) all the vendors of all systems and their support numbers, (6) the applications (including custom ones) that are used by all the users, and (7) whatever processes or requirements there are, even if they're informal.
#3. You must attempt to normalize the systems as much as possible. That means every system, as much as possible, should be exactly the same. This also implies a remote control system and, possibly, a software distribution/configuration management solution. You must gain control over the machines. If everything is the same, you spend less time thinking. Thinking is the one thing you don't want to do when something happens.
#4. Document everything. You'll need to figure out a way to document all this. ITIL may help, but there isn't any real documentation tool for the IT industry. This is unfortunate, and a real market opportunity for someone. I've been inside Global 500 shops and small shops, and nobody really has a good way to document their infrastructure. They have tools, and the documentation on the infrastructure is to an extent embedded in the toolset (ie: dashboards, impact diagrams, etc), but real documentation is pretty scarce. There is no single tool that can really represent the multidimensional nature of the IT infrastructure.
Oh, and #5. Customer expectations. You need to be able to manage customer expectations. If you get two more clients you may be OK - depending on how you handle your new customers. If their infrastructure is tight, you may not have much more work at all. If it's organic, then you may have small problems. Most organic IT shops function most of the time. Hire the smartest guy out of the org and boot the rest, then implement the above.
Good luck! Sounds exciting!
Oh, that's five things, not three! Doh!
The parent's comments about experimenting on a client's network are valid, but I find the "Hire someone who knows corporate IT" comment more interesting. Imaginary Friendly's company with it's clear lack of experience and long working hours do not sound like a firm prospective clients should have much confidence in. I believe this is the first problem that should be addressed. If the company has staffing issues before taking on new client's, then increasing headcount should be the number one priority. Contractors are a handy way to solve short term staffing issues as you can get them on short notice, and easily fire contractors that don't live up to expectations. With careful management contractors can be used to take the strain with new client obligations. With proper staff levels in place the problem of experience can be addressed.
Most posts on this thread seem to assume that the prospective client's infrastructure is well maintained. In my experience this is not usually the case. The original question does not say what familiarity with the new client's setup the company has. When taking on a new client the first task is to assess what you are taking on. To do this an in depth site survey should be taken, followed by a strategy session with the client. These are chargeable items. Some client's will take the output from this work and use it to find the lowest bidder for the maintenance work. The lowest bidder may not be your firm so do not do the survey for free, even if the client dangles the possibility of a large contract in front of you. A good client will pay for the work, a client unwilling to pay can be left to drive your competitors out of business.
In the larger scale environment standards go hand in hand with automation. Pretty much any task that can have a standard written for it can also be automated. Desktop machine installs should be automated at the absolute minimum. Automating the routine tasks lets you deal with the problems that you are there to solve. No matter what platform you are supporting automating the routine tasks will make the difference between small scale support and enterprise support. To start with buy everyone in your company a set of system administration books for your chosen platform so they can learn on the job. On the job experience is the best way to learn how to sysadmin in a corporate environment. Hold regular sessions with your coworkers to discuss current work and strategy. Assign responsibility for each task to a specific person so that they can make sure it gets done. Let the person responsible for the task figure out if it is a candidate for automating. Much of the work you do will be the same for each client, so a good desktop install script for instance can be used for all your clients.
Whatever you end up doing have fun and don't stop learning. I still learn new things every day even though I have been in the industry for years. Don't be scared of work on a large scale, someone out there will do it so why not you?
Steve
First off; I work in a role where I run a Tech Services organization that services both small and large enterprise so gauge my response as such.
Your question shows me that you're primarily technical and not so much on the business end of things. You talk of service functions and not necessarily the back-office issues that will accompany your growth. While I understand your position and viewpoint, I must stress that in order to sustain what you build, aside from the tech, you MUST address back office and operations issues.
Large enterprise usually has demands in terms of service and account maintenance. In order to remain viable you must not only understand the people that you will interact with on a technical level but the business stakeholders as well. If your company's costs show up as a line item on a budget with out any mindshare from decision makers, you will find that ultimate service doesn't matter.
Sad but true. I've see many other companies maintain accounts by diligent relationship management. They're tech service personnel aren't sterling but the way they handle preception is. (how sad/pathetic is that)
Some of the advice on the methods give so far are also good. One more suggestion is that if you embark on learning these things, you must also attempt to USE them internally so that you can understand them at in multiple ways. Incident Management for example; Take a look at a ticketing system like Request Tracker. Stand it up and use it to manage service requests. Then dive into the SQL and write some queries and views that can be used for billing support. Then improve the classifications used on the tickets to give better customer guidance. That one step offers you both technical service delivery benefits and also back-office operations support.
Well, I should stop giving away the crown jewels. G'Luck on your efforts!
Get yourself familiarized with Microsoft ADS, Ghost over networks, and other PXE systems like redhat kickstart installs. Grab a copy of CentOS (a debranded redhat clone) to familiarize yourself with a bit of the larger class linux world. I'm a gentoo fan myself, but as you can imagine when we do customer installs it's MS, RedHat, or SUSE only...the bigger companies care far more about support of a product rather than how good it is on the global scale. They'll accept a program that works 60% as good if you guarantee that you'll be on site in one hour if it ever breaks.
What kind of servers do your new clients have (HP, Dell, IBM, Sun??) I work with HP and IBM and both provide great system management software to help take on massive quantities of machines. HP uses System Insight Manager, and IBM uses Director. Both have paid license based deployment tools (Remote Deployment Pack & Remote Deployment Manager respectively) that give you the capabilities of managing firmware and software installs network wide from a central server.
By leveraging IBM's Director, Ghost, MS ADS, Anaconda, and HP SmartStart scripting toolkit into one beast we manage to deploy several hundred machines on a busy day (still only 8 hours) with only 4 people in the build room. (I work in an integration facility for a large VAR of several major brands)
The problem with self-taught people is that they are "experts" in operating whatever solution they taught themselves, and filter all problems through that tool set. One of the big reasons for this is the free/cheap learning material that they use are vendor-provided educational/marketing collateral.
So when these people move up the ranks and become Microsoft/HP/Cisco/IBM/etc certified they have a natural affinity to their chosen vendor.
Conformity is the jailer of freedom and enemy of growth. -JFK
First, are you really going to be a 'big shop'? If it's possible for you to know all the names of everyone on your staff, you're not a big shop. You're still going to hunt down your top performers by name when something important needs done. That's not big shop.
If you are indeed going to be a big shop, you've got to make the business bigger than its people. The business has to go on no matter who, so long as they're capable, is in a position. Because in a big shop you don't always know who is doing something and whether it's going to be done right, you MUST put a measurable process in place. You've got to be able to hand off a task as a transaction and to measure that it's being done right, not just say, "Oh, Joe's working on that and we know he's top grade." Define your process, work by a process, and when it doesn't work treat it as a special case. Either move that single particular task outside of process until you can put it back in, or consciously and deliberately adjust your process. DO NOT fall back into letting your top guys save the day. They'll get tired of that and go save the day for someone who can pay them more.
Because the business in a big shop is bigger than its people, you've got to do what's right for the business, even if good people won't necessarily be treated with the respect they might think they deserve. Your top IT guy who's been with the company from its start may feel he should be in a manager position. But if he can't manage, if he's not someone who can recognize when things need to be taken out of process and manipulated, or if he can't own and orchestrate the process, leave him where he is and hire a good manager. Insightful, assertive, authoritative management is going to save you and your customer relationships more times than a wizard coder. The wizard coder makes it a viable product. The manager makes your business around the product successful. You've all seen enough marginal product to know that wizard managers win over wizard coders.
- Sig this!
Because you "plebians" have no standard for which to set yourselves against.
In my current position I'm maintaining and upgrading code that the "genius, 5 years ahead of his time" previous lead developer left for my employer after Katrina. It's pure 100% unadulterated crap, complete with no documentation, no comments, no obvious planning, a broken templating system, classes that aren't objects in the vaguest sense complete with configuration data in the class file, no exception handling, and error messages that have no obvious meaning. Everything is either over-engineered or isn't at all. The entirety of his work here has been nothing more than a giant WTF, which has been the case with virtually every "home taught" programmer I've ever worked with. When I brought the obvious problems to the CTO, he was shocked because he had just assumed that it was the way it was supposed to be. When you've never had a standard that you HAD to code against, which is the case of most non-degreed programmers, it's easier for you to trick yourself into believing that you're designing and developing your software correctly.
I'm sure there are exceptions, and some of the all time greats come to mind. But while a degree doesn't mean that someone can code worth a damn, it sure weeds out some of the weekend hobbyists who couldn't cut it in the curriculum. When hiring someone without a degree, I require code and documentation samples in addition to the technical questions, just to weed out those who have memorized all the usual questions.
As for the "genius, 5 years before his time" guy, well, apparently he's run into a problem with employment and is now trying to sell himself in a consulting role. He petitioned the CTO with a pitch to revamp parts of our system using the MVC pattern... apparently he's picked up Design Patterns for Dummies, because MVC is totally inappropriate for the application.
Why hack at all? Just use an IP KVM switch. Full-remote, full-desktop.
Excuse my speling.
Making The Bar Project
If you do not have windows servers and cannot use RIS, you can use DiegoStart It is basically an unattended windows install for the OS and software. There are step by step instructions on how to configure your DiegoStart server (just a windows share with all software needed). I am currently using this method for Windows XP and Vista, even though I have not yet published the Vista changes to the website.
That's the same hack - you have to buy extra hardware to virtualize the machine's local video. The point is that Windows cannot support machines without local memory-mapped video. The purpose of local memory-mapped video is to provide high-speed local interaction for users of a workstation. The fact that Windows servers sitting in a datacenter need such video hardware, and then on top of that need other hardware to virtualize that video hardware, and even after all that its remote administration capabilities are inferior to that of other OSes, is the problem.
Could not agree with you more. After managing more servers than I care to mention the hardest thing for many folks (read techs, engineers) to understand is that a deskptop crash often on afects a user. A single server crash can wipe out the productiviy and livelyhood of many people. I agree Visible Ops, the ITIL or just about any documented, controlled and verified system will be better than "drinking any koolaid or buying any software". In my experience with good processes and controls two good sysadmins can run at least 300 servers. One of my old SMS admins could run several hundred desktops by his lonesome with the occasional help from another IT guy. Yes automation is key but without consistancy of practice (read process and controls) you will just be another fire-fighter boy.
Excellent post. Not being a sysadmin type I didn't know this (I'm an ASP.NET, SqlServer developer), but with my new job I've got considerable input and say over what the server and network setup will be for my organization for the next several years. We have about 150 desktops and every one is configured with the user running as local admin and most without password (horribly insecure, I know).
I want to change this but I don't want the expense of Active Directory. Can Mac OS X Server's Open Directory be used in place of an A/D PDC/SDC on a Windows network? If not, is there a true A/D replacement in the open source community?
They should be, but...
Right. And the point is that you shouldn't have to resort to that sort of thing just to do routine remote maintenance of a server. All the (Unix) servers I run have a remote console capability, but I almost never have to use it. The fact that Windows depends on this has a pervasive impact on the way server software for Windows is written: most server software uses the local GUI for administration, so even if you do set up e.g. SSH access to the server, it's crippled for many admin tasks. There's no way around the fact that server software that depends on local memory-mapped video hardware creates issues for administration, which no amount of hardware workarounds can completely fix.
Yeah. I first spec'd a Win NT system as a server for a client back around '94 or so. 12 years later, I'm tired of waiting for Microsoft to figure out that exactly how servers aren't desktop PCs. Besides, Debian licenses are a lot cheaper. :)
highly technical people will kill us all with their highly technical mistakes. It has been proven adnasuem that 80% of all IT outages are caused (yes this is real research from Gartner and many others) by "highly technical" IT staff not by security breaches or equipment failure...you have to manage the changes if you want to have stability. hmm..wonder why all of the best in class shops live and die by their change management programs. I have worked with shops that handle thousands of servers with just a handful of sysadmins because of their amazing change management programs. Just because some shops (read most) don't understand how to make change management work correctly doesn't mean process is evil. After building several IT organizations with server to sysadmin rations north of 150:1 I became a complete convert. I will say that if you have have bad change management it is better to have none. But if you have no (or ineffective) change management you are spending way too much time fire-fighting and not enough time doing the preventive work neccesary to keep things up. I wonder why most serious shops with any retail exposure ban most changes during the holiday season?? (hint: so their availability numbers go through the roof and they can cash in on all of the sales)
I have worked on IT for 17 years in many different countries all around the world, and my typical week has been 40 hours, 35 the last few years.
If there was the smallest hint that hours would become insane I was posting my CV to agencies and prospective employers faster than you can say "overtime".
IT people are all too willing to put put macho bravura performances, when in reality weeks of 60 hours are pointing to poor management and personel with lack of self respect.
IANAL but write like a drunk one.
Understandable, but I've dealt with BACS who have no true concept of structure in their coding either. Alot of it has to do with people just being lazy. Lead a horse to water... I believe your interview method is actually the best way to look at it, I mean seriously am I going to have an artist paint me a picture without ever seeing his work? I personally don't go for programming positions, but when applying for positions I've always supplied them with my portfolio which includes graphical, code, and completed projects along with a list of my skill sets. However when I was teaching myself to code, I was doing it from text books essentially that also promoted best programming behaviors. However being self educated I do hate the stereotype that I'm going to try and fit a square peg into a round hole so to speak, I'm adaptable. Unfortunately that stereotype gets tagged to guys with my background by fellow IT guys who may or may not have gone to school and yet try to do the same thing consistently. One of the worst things I hate when walking into a new environment then talking personally with clients. "Yea I use linux at home" I can practicly hear their buttholes tighten up as if I'm about to take away their linux virginity and rip them away from their courtship to microsoft. I then have to go into a spill about, why I think their present setup works, why converting to linux would be a bad idea at the time blah blah blah (I guess I can thank the zealots for that). I mean for christ sake, I use it at the house because I like it, not because I think it's the savior and brings light into a dark world. A crude example but the point is, I realize the limitations of technical solutions. I also realize the difference when it comes to needs and wants and what it takes to get there. Why ice skate uphill?
Oh and I meant to say, plebians = plebeians. I'm not sure if you intended for that mistake to occur or not. Also, ouch that hurts you arrogant twit.
Imaginary Friendly,
/. community; AutoIt (a scripting language) http://www.autoitscript.com/. It was originally created for pc rollouts, but has turned out to be quite a powerful scripting language. I understand you may not have time to learn this language, but if you can do it in your spare time it will eventually save you hours a week, and be able to start being profitable to you and your staff. The community is quite active and willing to help in most any situation.
There is a bundle of good advice on this thread, and I really appreciate you posting the question. I am a small IT consultant as well, with some experience with the big guys. What I have found to work wonders is if you can have the computers doing some work for you. Some items that you normally do that eventually get old and tiresome. You mentioned Manual Rollouts, and how that wouldnt really be possible for larger clients. You are correct, but with a little scripting you can have this all done.
I would like to recommend to you and to the
I would also like to recommend that you find someone who can manage your large accounts for you, and help show you the ropes. Finding a competent person that is willing to teach, and learn.
Also if there is any way I can help as an IT professional let me know.
JS
Right now I work in a school system that has over a thousand pcs, 6 novell servers, and only 4 IT staff. And it just works. Granted it's different than your situtation, that we don't have to deal with clients in the normal sense. Then again it's not so different, we have to make our administrators at each school happy or else the crap will hit the fan.
The biggest advice I could give would be training, when I first started a year ago I had about 3 days training and I was left on my own at the school I'm over. Not that anything was that difficult to administrate, it was just getting to know the school and users in the school. Next thing would have to be automation. While the school's technology department can only buy so much, I mainly use Ghost since all the desktops are Windows. Having to reimage an entire lab of thirty computers is a whole lot easier than to reinstall the whole lab.
Also make sure you have a someone that specializes in the different areas of IT. For example until I arrived the school system had to outsource configuring a few simple routes on a Cisco router at $100 an hour. We have someone who has had many years of hardware repair experience.
Just my opinion
Don't you hate glorious self-promotion? Visit my Blog
My recommendation is this. A 200 PC shop can (and should) support a full time IT administrator. If you can work the standards at the two shops (where possible) to a common management methodology (standards and such). You can get away with 1.5 for both. With this in mind, I would do a search for a high-end guy, and be ready to pay him. If you don't go cheap, you can more easily pick up some quality talent. Look for someone who knows how to interview the client and build a "strategic direction" for their IT that fully supports their business plans.
There are many things that can streamline support (whether outsourced or in-house). These all need to be analized and recommendations made to the client for future changes and purchases.
One is looking at thin clients using Terminal Services (assuming a Windows shop). This can cut out a huge amount of support headaches.
Another is simplification of the app set. This is, basically, reviewing the various needs of the org, the installed app base, and working to reduce apps with duplicate functionality.
Another option is to look for good enterprise open source apps that can be used to fullfill needs while not overrunning a budget. For instance Compiere is an excelent OSS ERP/CRM app (Enterprise Resource Planning/Customer Relationship Management, basically HR & Sales). You will have to, at present, invest in an Oracle license and optional support contracts, but it is well worth the costs, being cheaper than all other High-end ERP/CRM bunbdles out there. Also, if they don't need the high-end features of MS Office, see if they can use OpenOffice.org as a standard. OSS, however, is not always the way to go. This needs to be looked at carefully for each organization and application in question.
Another thing to consider is a strict (at least as strict as you can get) hardware standard. having fewer target platforms reduces the support nightmares. The only places where the standard should be compromized is on montors and printers, or, possibly CAD tools (pucks, styluses, etc.), if it is an engineering/architectiral firm.
Also, along with the standard build is a hardware refresh policy. Every three years or so, a PC or printer should be replaced. You can spread this out on a monthly basis where you have a sane amount of upgrades to deal with each month. Don't change your target hardware standard more than once a year, though. This limits the installed base to 3 base platforms.
Antivirus and firewalling are absolutely essential. Setup a central antivirus management console on one of the servers and push out updates from there. This reduces the external traffic. For firewall, I recommend Astaro. It has a comparable feature set to a Checkpoint firewall with a fraction of the cost.
These are just some of the options to cosider to bring a site from chaos to sanity. A good admin will bring these skills and experience with him. Feed him well and he'll keep your clients running smoothly.
If you are utilizing the right tools... .msi packaging and deployment...(www.wpkg.org) is a nice opensource deployment tool for this.
disk imaging(roll out an entire office full of workstations in 30 min)
antivirus server(pick your vendor)
backup systems(BackupPC is one I love, doesn't do bare metal restores though)
tools like 'Nagios' for server/device monitoring
Groupware packages like Zimbra
and of course VMWARE...why have 10 test machines taking up space and heating up the room...test new stuff on a virtual machine and then deploy once you learn and get kinks worked out.
The right tools will save you a LOT of time and manpower.
"Just Smile and Nod." --Huck
Sub contract an IT tech (team) from a good firm.
Interview for a qualified partner in crime (PIC) that has skills that fill the bill.
Hire, then train him/her.
The subcontractor will cost you plenty but will allow you to play catch up and snag the accounts.
Be up front with your customers about this, either they will support you in this, or you find out what you're really up against early.
It's all about being "reasonable".
Spend money - make money.
Eh?
~hylas
So all you really did was put more of a load on hotmails spam filter.
Cool!
Feed the need: Digitaladdiction.net
If you're too busy, then you're not charging enough for your work.
Our support is currently crap but it doesn't sound like you'd be able to provide anything better.
As others have noted, the consequences of doing business poorly can be worse than the results of not doing it at all. But the other dimension is what I call "bullshit factor". It works like this: You're a small IT shop serving, let us say, 30 small-business customers on a fairly regular basis. You're approached by a business with large needs -- needs you can't handle (say you're a sysadmin shop, and they need programmers). You decline, and you offer them a lead to a shop you are pretty sure you can handle the work.
This has two positive effects for you:
1) You have built a quid-pro-quo with the other shop. One day you may suddenly find that they have been referring customers with needs they can't handle to you. This can go forward and eventually become a formal partnership or even a merger to the benefit of both businesses.
2) This company you declined to take on now know that you're not feeding them bullshit just to get business -- you care about the quality of your work more than the quantity. People remember that -- the kind of people you want to work with do, anyway. Chances are pretty good that when they have needs you can meet, that they'll come back to you, or if you expand and become able to take on their needs, you can approach them later to see how happy they are and throw your hat in the ring if they've got problems.
Even if you never hear from or of BigCompany again, as others have noted, at least BigCompany isn't badmouthing you to their partners and hauling you into court because you screwed up (avoiding negative reputation and lawsuits is as much of a win as gaining positive reputation and money, though it's harder to quantify).
The OP does have serious staffing problems by his own admission, and if he can bite the bullet and hire some competent staff to take care of his existing needs, he'll be better served in the long run, not least because he'll have gained an idea of how to grow in an orderly fashion.
-- Old Man Kensey
Resumes are so unhelpful, these days. We bring a broad smattering of people in. Some have just a littl experience post-BS, some have a lot. Virtually all of them fail the interview where we have them write a trivial algorithm.
When you say "looking for" you probably mean, how do we pick the resumes. HR does keyword searches on resumes posted on Monster and other resume farms. We also post jobs and look through all applicants who are interested in US. We contact, with the desire to bring in for an interview, everyone who mentions the things we're looking for (Java, SQL, Oracle, perl, etc -- whatever the particular position is). They routinely (90% of them) cannot crank out a five line program for a well-known mathematical function.
What they can do is rattle off buzzwords. Inspection usually reveals that they don't have any detailed knowledge in the topics or real understanding of how they work.
If what you mean is "how can I get a job?" -- I'd strongly suggest you put open source projects in a prominent place (near the top of the page) of your resume, and put keywords in a skills section down low so that HR will find you on a search (spell those keys in different ways for maximum possible hits). In the interview, KNOW HOW COMPUTING WORKS. KNOW HOW TO WRITE A PROGRAM. And KNOW HOW TO BREAK A PROBLEM INTO PARTS. You'd be surprised how far that'll get you.
My point in the original post is that a CS degree used to imply that you'd be able to do those few things. It doesn't anymore. I'll take an English major who has some experience working on an Open Source project over someone with a CS degree and some work experience from which they aparently learned nothing any day.
the point isn't the OS used, it's insecure the way things are done, running things like a home computer.
Your company should also have a financial plan to guide the management of assets, cash flow, and the like.
Finally, if you intend to grow, you're going to reach a point where you will need to separate the functions of technical operation and administration. If a customer is having problems, you can't be trying to get out the payroll, and if you've got to calculate and pay the quarterly tax payment, you can't drop that to solve a customer problem.
These are all business problems which successful businesses have gone through, and unsuccessful businesses have neglected. There's probably a business networking or entrepreneur's group in your area that you could link up with that can share insights and lessons learned. Alternatively, there are plenty of savvy business people in non-competing businesses that you could meet by joining the local business and services groups to developed connections, learn who has expertise, and build relationships that might lead to future business arrangements.
Growing a business is a lot like chess: not all moves are equally valuable and not all pieces are equally powerful; it's knowing the right move to make and when to make it that controls the flow of the game.
Sure, it's appealing. But you have to remember that moving to Enterprise means you must maintain a constant supply of Dilithium crystals, and the Warp Drive is notoriously difficult to maintain.
... and then they built the supercollider.
Just noticed this response and i thought i'd chime in for Internet posterity.
Documentation does not necessarily include "step by step, screenshots every time you click".
It includes things such as the REASON we opened port 4773 and which application will break next time we upgrade the firewall (four years from now) and forget to open it.
It includes things like... what is the administrator password of that database we haven't touched in 2 years, but still has critical data in it.
It includes things like WHICH subnets route through our Cincinatti office and which subnets route through our Denver office.
it includes things like the static IP addresses at the warehouse and why there is only two computers out there running DHCP when the rest are static.
As for change management, obviously, you've never worked for a large organization.
Shall I relay the things that happened to me THIS WEEK regarding undocumented changes.
Well hmm... Last week, our entire call center (120 staff) was down for 4 hours because some administrator ran some patch on some server somewhere in a different office, that caused back-end process to hang, which led to our telephone database loading dummy values instead of customer information, and then crashing the front end dialer, because of null-terminated values....
Now, it was a 3 minute job to roll back the patch once we realized the problem... but nobody had ANY idea this administrator had messed with the server today and he took off for a long lunch and left us troubleshooting problems on the wrong side of the world.
This week, an administrator changed the IP address on a database server. He tested thoroughly all of the applications that use this database, ensuring they were running from the hostname, rather than the IP address. But he didn't realize that another database had a hard-coded link to the system, which database updates on a different customer database. Had this system been in our newly-implimented change management system, the system administrator of the second system would have been emailed with the change a few hours in advance in order to telephone the guys who were handling the change and coordinate, but it wasn't.
We do have a new change management system in place, and the records are being imported gradually. Just this week, it has saved two major issues with systems changing patch-levels or having downtime that would have been a serious issue without the change-management procedure.
I'll admit that when you have a 1-man IT staff, documentation needs only to be lightweight reminders, and not much more. When you have a 3-man IT staff, documentation gets more important so you don't end up in the situation "I think Bob changed the IP address, but I don't know why or to what".
In an organization such as my company that has over 80 IT staff and 16 locations (about 2500 employees), even using group emails becomes daunting... at any given time, we have 10 or 15 "bits" of work going on with various systems and networks in different locations, it is frankly impossible to coordinate and be informed of changes, unless you have a procedural system such as change-management in place.
the formal procedures for change-management that are defined in something like ITIL are only useful for organizations who's IT structure is too large to have an "all hands" meeting without using a baseball stadium. In that case, it is impossible to address functional requirements of all departments and system groups without using established procedure as a starting-point from which to develop specific aims for your organizational systems.
It's great that you're clairvoyant and can instantly know when one of the other 18 server administrators touches a system that you depend on. I'm happy for you that you are immediately aware when one of the other 9 networking specialists changes a static route or updates a DHCP scope. It's great that you are subconsciously aware when one of the other 14 Desktop support techs pushes out a new, buggy patch to the entire organization and breaks half your applications.
Most people don't have that level of psychic energy and require systems like change-management to inform them of what is going on.
Stew
There are 10 kinds of people in the world. Those who understand binary and those who don't.