Slashdot Mirror
Ask Slashdot: Documenting Scattered Sites and Systems?
First time accepted submitter capriguy84 writes "Six months ago I joined a small firm(~30) where I am pretty much the IT systems guy. I was immediately asked to work on couple of projects without much going through the documentation on what currently exists. So I created new wiki topics everywhere and whenever needed. I am now in a situation where information is scattered across multiple pages and there is lot of overlapping. So I have decided to start a project of re-organizing the wiki so that it makes sense to me and easily accessible for others. I am dealing with 2 disjoint sites, 4 data centers, managing all flavors of Unix, windows, networking, storage, VMware etc. Along with that I have HOWTO guides, cheatsheets, contracts, licensing, projects, proposals and other things that typically exist in a enterprise. Any tips with how to approach? Dos & Don'ts? Recommended reading?"
Doing all the documentation for a few small IT projects, I've found that I'm better served creating task-based and user-based documentation than holistic documentation of the system. If you've a limited amount of time, I suspect it will be better spent creating easy-to-use guidelines for the most common interactions with the tech you're working with that people other than you will have. Step by step, idiot resistant, and with the technical nitty gritty just deep enough under the surface that somebody who understands, will, and that end users won't be troubled by it. It's a wonderful thing to imagine documenting all of it in some detailed, top-down and holistic way, but chances are that you (or whoever they replace you with) will be the only one looking at those guidelines, and that nobody will appreciate all that work, compared to a set of PDFs allowing anybody in the company with authorization to do X, Y or Z which make you look both useful and benevolent.
Most used stuff on top, however you see it. (or the user community). Extremely vague advice, but just something to help keep rational.
Why is the advice on Ask Slashdot so often "quit your job" as if running away from the problem is a good solution?
I have an issue with my boss --> quit your job
We are using a system that I don't like --> quit your job
In my company, I have this difficulty --> quit your job
As for the original question, I don't think I understand the problem: Of course you have an amount of heterogenous tech to look after. You did a good job organizing it into a wiki. So you already have your information centralized, and just need to maintain it. What's the trouble?
Not sure if the following is applicable to you, but in the last few weeks I dealt with documenting a software package. With sphinx (not limited to python) you write the documentation in the code, and it extracts it to make a nice website. The benefit is that documenting in the same place where you make configuration/software changes is more likely to be maintained. Maybe you can do something similar and build your documentation from the config files, so that centrally, you only tell which config files (etc) you want to pull. The benefit of a wiki is that you don't need any software to change text, and everyone can contribute.
NB: The message above might reflect my opinion right now, but not necessarily tomorrow or next year.
Sometimes that is the correct solution, however I doubt that's the case with this situation.
The best thing is really to decide on organization ahead of time as you then can keep it organized from the start. Ideally every time one input information that was of general use, one would make some sort of notation about it and create a page linking back to that section of text.
Doing it in retrospect is probably going to be a lot of tedious work as it's unlikely that a simple search is going to reveal all the common information without overwhelming with false positives.
Having worked in small environments I suggest pretending you're the CIO not just a low level IT drone.
Create your imaginary virtual employees and organize your data appropriately.
Unless you have a really good reason to go project based at the top level, I'm guessing as "CIO" your employees would be mgr operations, mgr development, mgr security/auditing who basically watches the other two or something like that. Take a wild guess what your top three level directories or top level three wiki pages I'm suggesting.
It become an interesting role playing game, sorta. So, if we were big enough to have a guy who did nothing but R+S CCNP/CCIE type stuff, he would probably report to the operations mgr, so R+S type stuff has a wiki page linked from the operations page. If you had an admin/intern of license collation and recording working for the operations mgr, that would probably link off the operations wiki page, etc.
This is also hyper convenient if the boss graciously grants you a summer intern, you can almost instantly trivially drop that person right into your pre-existing "system". Look kid, you're now officially an instant licensing admin, or whatever.
Also in your summary of "stuff" you overlooked written EULAs you provide to your internal customers, request forms to fill out, whatever ticketing system you use, whatever project management system you use... And it seems helpful to have a public or private or whatever wiki or something documenting what metrics, if any, you provide to your boss for review time.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
Consider who will be using the information, why, and when.
Determine the most common use cases for the information you have collected. Then organize the information to suit.
Server died. What do I do?
Add a user: What do I do?
etc. Think about where you will be in a year, and how this information will be used when you have forgotten most of it, or in panic firefight mode, or gone on to another job.
there are 3 kinds of people:
* those who can count
* those who can't
Last year i used MediaWiki's SemanticWiki to describe the systems, projects, human-resources, external-urls and their dependencies of a telecom.
Besides trivial parent-child relations among developers/employees, departments, etc,
i described system dependencies as semantic-relationsships with names like this: part of, invokes, build by, implements, deployed on, etc.
I described developer responsibilities with names like this: maintained by, coded by, external contact, etc.
Finally, the documentation pages and the refs to external-URLs of projects were reorganized by semantic-relations, like this: javadoc, docUrl, webApp, section of, help page, etc.
Agreed, don't bother. In a small company situation, it's good to be irreplaceable.
Capriguy84,
Over the last few years, I have worked with a service provider on what to an outsider would sound very similar. The organization's processes, applications, and information had been distributed throughout the company on an ad hoc and project basis, resulting in an irrational de facto architecture that contributed inefficiency to practically every activity. For any organization, addressing these systemic issues will always be a work in progress. Although my case differs in scale by more than a factor of 100, I think the lessons I've taken may apply for you as well.
1 - Dysfunction is not a product of the technical situation. It is a product of management. Thus, it can only be addressed by management. For you, this means you have the opportunity to lead the organization to the right outcome, and you will have to get management support to get folks to change their processes. It looks like this: a) Identify the outcomes the org is looking for. Not "FAQs on the same server," but "Knowledge Management - Processes and technology to get and KEEP key company information accessible when necessary and secure." b) Obtain management support on the outcome. c) Explain the steps and costs needed - technical and non-technical. Management must ensure that new knowledge lands in the right place and people to go to the right place for the info. d) Implement - the technology and the culture changes e) Evaluate.
2 - What you are doing is not unique to your organization. Practically every firm has this problem in varying degrees. Go scan through the book "Faster Cheaper Better" by Michael Hammer to see an articulation of the issue at a very high level - not specific to knowledge management.
3 - There are frameworks that help guide the architecture process. Have a look at TOGAF. These are model driven, and frankly, they are very hard to consume and live by. Nevertheless, the point of these is roughly the same. When the problem is too large, divide and conquer. Model the organization and subject matter by dividing it into its parts. Prioritize, strategize, etc. Having the model enables you to ensure that solutions at the micro level are in harmony with those at the macro, etc. TM Forum does this for telecoms.
4 - Since the most important part of the job is getting buy in from management and those having to live through the culture change, your soft skills are more important than the tech skills.
All this might look like killing an ant with a sledgehammer. I suggest that you take a little time glance through material on how this is done at large scale and apply whatever seems pertinent.
Good luck!
Do you know what ITIL is? Find out. Then get yourself a CMDB. There's open source ones so you're not going to break the budget.
http://www.cmdbuild.org/en/index_html?set_language=en&cl=en
Then get yourself a document management system. Alfresco? It's a monster.
If you can tame those systems, then you can look for a massively well-paid job with a bigger company that wants to leverage enhanced ITIL capabilities in an enterprise solution situation scenario. F**k yeah.
But if you can't be replaced, you can't be promoted.
In a small company that doesn't have a CTO position, and thinks all of that technology isn't very complicated and can be managed by a single guy acting as "pretty much the IT guy" you are already (falsely) perceived as interchangeably replaceable. I remember one particular interview where I met with the CEO to interview and he wanted to low-ball my already quite low suggested salary. He told me that a high school kid could do what he needed. I simply replied: Well then you need to get yourself a high school kid."
I recommend that you start focusing some effort on changing the false perceptions of upper management, because if they don't understand what is involved and why it is important, all your effort will go unappreciated and even the best solution will fall by the wayside the moment they need to save money to buy more staples.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
Even if you came up with something that will work fine today, it will require a lot of maintenance and probably have to be changed regularly to keep up with the new type of data you will need tomorrow.
I have had to deal with such problems under different environments, and what I found is that organization is much less important than a good search tool.
Make the search tool easy to access and use, and make it work, and people will come to use it by default.
Make sure your tool creates a list of failed hits and sort it by occurrence, and you know what to work on next.
You may also find an inventory or asset management system useful to make sense of what you have, if its relevant in your circumstances. There is web based or software like http://www.ocsinventory-ng.org/ http://www.tracmor.com/ http://www.pukkapanel.com/ and others mentioned here.. http://www.cyberciti.biz/tips/open-source-it-inventory-control-systems.html or if you use RT there is http://requesttracker.wikia.com/wiki/AssetTracker
There's also a previous /. discussion on asset tracking stuff although perhaps a bit outdated now.. http://slashdot.org/story/06/08/20/0214256/it-asset-tracking-and-helpdesk-software
You need to find out where you are before you can develop a plan to "get from here to there."
So, Step ZERO should be to look around all the systems until you find the previous sucker^WVictim^IT persons' name and email address. You might find it stuffed in a comment in a file called by a cron job, or in the header of a random web page, or some source code, or even in a README file.
Then for Step ONE, you contact them to get the real story of what happened (or at least a second perspective, from someone who was there and doesn't have a reason to downplay the stupidity that caused the current situation), and can begin making informed decisions - such as whether you want to stick around or not.
If you DO decide to stick around, then Step TWO becomes adding YOUR contact info (at least your email address - it can be a throw-away one) to a few places, for the next person. It's just professional courtesy to do so (and in engineering, it's pretty much a requirement - you touch it, you sign and stamp it - we should demand the same level of accountability and professionalism in IT).
Step THREE ... brush up on your BOfH, because you'll need the inspiration and moral support when management doesn't understand why what you think is essential is wasted time to them. And why stuff that makes sense to you simply will NOT GET DONE BY USERS.
Step FOUR ... forget wikis and stuff. The won't edit them, they won't use them. Ditto for bug trackers, ticket systems, etc. Their impression is that you work there in house so they shouldn't have to do that sort of stuff. So, simply tell them to email you for every issue, and you'll look at it when you get the email. If they can't be arsed^troubled to put it in writing, you can't expect to take time away from (point to overflowing inbox) all the ones that people have taken the time to write up properly. Make sure that every time you find an issue yourself, you email yourself as a "reminder". You should quickly end up with a couple hundred emails, just from yourself.
Half the time, they won't bother with the email - they'll go back and figure out that, for example, to print something they have to click on the printer icon (sad but true story). Or they'll ask someone else, while you've just CYA by showing that you already have LOTS to do.
Then at the end of every week, do a simple check of how many emails vs. how many resolved issues. If you can email the boss with a "this week, I had 157 emails about 43 different issues, and 36 were resolved" (and make sure that you send out emails asking for confirmation of every resolved issue for more CYA goodness) you've made progress where it counts - the office politics brown-nose department.
More usefully: write this shit up for yourself. Point your boss at it to show you're on top of stuff, but write the doc you would want to read, and so you know what the hell you did six months ago. This is what I do on our intranet wiki. I then point other people at it 'cos that's where the info actually lives, but as far as I'm concerned it's my online notepad.
http://rocknerd.co.uk
If you want to write a book, consider using Lyx (Link: http://www.lyx.org/Features) instead of Word 2010. It keeps track of all indexes, links, references, footnotes and TOCs, produces PDF output with links to everything you desire to cross-reference, and gives you a printed output much better than any version of Word. Give it a try!
You know it's time for the next revolution when your rulers' names end with roman numerals.
I have a similar setup (less locations, but the same sort of cluster foxtrot of non-documentation) and I use a wiki. I'm not sure what you're using, but I used MediaWiki. I created a name space for "servers" (actually VM's) and document their function, specs and what hardware (as a link) they are on. Now I can go to any hardware page and click "what links here" and see all the VM's on it. Of course that isn't perfect because there can be other links to the hardware, etc. I'm going to be trying out Semantic Mediawiki next because it'll let me query better (What servers are under X ip address, or Y location).
It's not a perfect system, but it works for me. I like working with MediaWiki and know it fairly well. It also allows me to keep other documentation with it that, yes, sometimes just tossed in there but at least I have search. I've tried some specialized "rack inventory" software but I haven't been terribly enthused by any of them.
snowulf.com
SemanticMediaWiki(SMW) can utilize triple-stores (proper RDF databases). But even without using one you can make still run queries from within wiki. You create the new query by using a special query-page and after you are satisfied with it, you embed it in any wiki-page. Next, whenever you view this page, the semantic-results come always fresh on that page. Compared to a RelationDB, SMW comes bundled with UI. You just have to learn a new syntax for running the queries.
Now to get the results in different formats you have multiple choices. I remind you that each wiki-page is at the same time a REST API. And SMW follows this path, providing specific query-parameters for getting query-results as CSV, as XML, as text, as html-tables, or any html-structure. And there are numerous extensions for embeding the results into google-maps(assuming they are geo-coords), event-calendars (assuming dates), graphs, trees, and many more.
Almost 15 years ago I went to work for a company in a very similar role. I did everything from coding to user support to network admin. Now we have about 100 employees and I'm the director of the small IT department (6 employees and two contractors).
If the company didn't care about IT they wouldn't have hired a full time tech dude. Jobs like that can be extremely challenging but also extremely fun because you control everything.
Having the technical capabilities we have in a company our size (that isn't a tech company) is rare and allows us to land contracts we'd otherwise not be able to get.
My job has its downsides. It's not for everyone and there's been times I've seriously considered leaving but mostly it's been good. All companies are different and so are the people that work in them. If he likes the job there's no reason he shouldn't keep it unless he thinks the place is headed down the toilet.
Unless you've been told to do it, don't do any of this at all, for two reasons:
1. Documentation efforts are often perceived by management as a waste of time
2. By documenting everything extremely well, you create a visibility that it would be easy to find a replacement for you if you e.g. demand a raise or screw up one day.
Cynical, I know, but in most places, those are the rules of the game.
What does the firm do. It seems to me like you need 1 file server, 1 mail server and maybe a web server. All can be combined into 1 unix box or MS small office server or an apple server. Step 1 is to simplify.
Step 2 is to give up on the wiki--use a couple of spreadsheets and an word doc (or apple/unix equivs); no one wants to or cares about your 82 page wiki
I'd start pruning like crasy.
"Quitting" is only an option for those who are independently wealthy, or are egotistical enough to think they can just grab another job by snapping their fingers. Get a new job first before quitting.
"I created new wiki topics everywhere"
I think that's the problem. Wikis are awful for gathering together information. Every individual page is too isolated. No reader can get an episcope, an overview of how all the topics fit together. Sometimes people suggest "make another topic that shows how the topics fit together" but this adds to the problem rather than solving it.
You see the same problem with things like Javadoc which are great for creating lots of tiny little informations but bad for conveying the overview of how things fit together.
The solution? Write FEWER topics, long ones. That way people can scroll through to get an idea. The brain's really good at skimming through large documents, understanding what the domain is, drilling down where it's needed. That's why books are so successful!
Why is the advice on Ask Slashdot so often "quit your job" as if running away from the problem is a good solution?
because it is precisely their job to deal with the problem themselves. if they don't even know where to start, they aren't quitting as if to run away... they are quitting because they are not qualified.
You may have a point there. You don't get doctors asking on the internet how to treat patients.
you're an idiot.
And you lost it on the pointless abuse.
> I far prefer wiki's. it takes seconds to correct and version a spelling
> fix in a document in MediaWiki.
How do you deal with the MediaWiki code? That's the biggest stumbling block in my org...nobody wants to learn the syntax of Wiki.
PRIME DIRECTIVE: Regarding some of the above posts, If you are having to strategically "leave gaps" or otherwise write bad documentation for the purposes of monkey wrenching your replacement or making yourself indispensable, you suck to the 10th power. I have dealt with this sort of fuckery more times than I can count over the years, and every time I clearly see the signs of a small mind at work. Don't be that guy.
I've have to do this routine a gazillion times in my role as an small/midsize biz consultant. Here's the formula as I see it:
Find the keys to the kingdom, and document these. Don't worry about getting to the nitty-gritty yet: just find the info that will let you find everything else. Be extra careful to track down any crypto-related stuff (keys, passphrases) that can't be replaced or cracked. The further the old IT person recedes into the past, the harder this crap is to track down. Identify the scariest bits of the network as quickly as you can after you get hired and trumpet to the hills about how fragile, dangerous, and not your fault they are :-) Document all of this in plain-old-textfiles or something stupid simple, with bonus points if you keep it in version control.
Set up some bug-tracking/ticketing software and use that to track all of your day-to-day documentation and troubleshooting. Redmine is my personal favorite, but RT and Trac are also good choices. They have simple, built-in wikis that are perfectly sufficient for this purpose. Use the time-tracking and project management features in the software: when the boss asks you where all of your time is going, you run a report and show him. Track every minute of your day: this time is excellent leverage for you when dealing with management.
Examine the backups of the system (or implement them, worse) from ground zero. Use the backup audit as the trail of breadcrumbs that your documentation follows.
Beg, borrow, or steal a chunk of hardware that you can stick Xen/HyperV/VmWare/AcmeHypervisor on and start test restoring various systems/apps/environments. Document the hell out of the test restore process. That's the most precious documentation that you can possibly have.
Don't document things that document themselves. You're much better off paying $299 for a copy of LanSweeper or the like to reach out across your networks and document all of the mundane details in real-time so that you can focus on making shit work right. I've seen a million cases where the IT guy spent a month making a beautiful set of Visio network maps that became useless a month after they were created. That's a waste of your otherwise precious time.
Now, start making recommendations about how to fix the fragile/scary/dangerous systems. Use your ticketing/project management app to track your recommendations and leave a paper trail of your process. There will be a fair amount of CYA involved here - you're going to recommend that the boss spend a bunch of money on $x, so you had better document the reasons for it with care.
Whenever you make a recommendation that the company buy product $x to resolve problem $y, document it very very clearly in your system, and if the bosses nix your recommendation, document that in writing too. Yes, more CYA, but as the IT guy you're often the staked goat when something goes wrong. You need to be able to PROVE that you had recommended a sane course of action.
Last: Try not to stress. IT is fun if you do it right.
Don't Panic!
A lot of this advice is very good, including generating documentation form configuration. However, please be careful and secure the website where this information is posted, because it will contain enough information to allow hackers to really hurt you if they choose to try to attack your system
It's an easy read, well written and with a sense of humor [note: it's a technical book on ..managing enterprise content. So it's not like "The Hitchhikers Guide to the Galaxy" easy... it's just not "gouge your eyes out after the first chapter" hard], and covers EVERYTHING you haven't thought of. And that most everyone here hasn't thought of. And everything I've forgotten after reading it. Chapter 1 alone will explain the very problem you just described, why it's bad, and why everyone does it.
Seriously - one time through that book has saved me on documentation questions (where to put it, how to organize it, how to write it...) time after time. And I still don't use 90% of what was in it. This is the book that you will read one time, and the concepts stick with you - and save your ass. And make you look like a rock star. A weird, librarian kind of rock star, that isn't fantastically awesome like a real rock star, but that seems to know how to use the dewy decimal system with that extra panache, rock star. People will forget all about it a couple of minutes after checking out their books, but for that one second when they find what they need when they need it and it's all because of you... you'll be the rock star.
Still - your life will be easier.