Slashdot Mirror
The New IT Crisis
Matey-O writes "You've succeeded in delivering 5 9's, your server farm is a well oiled machine, the helpdesk lines lie dormant. No? Well then how do you get credit for the work you do, when all that's noticed is the downtime? When the IT budget has to be justified, and you're overworked, undermanned, and you have to apply three patches to 100 servers before Close of Business, what has to change in IT before we melt down? Marc Andreessen has an interesting article on what has to happen to IT next."
capitalize on other's work!
It's the easiest way to success.
Details here.
Quote: it's not as if he needs to sit down in front of his screen and busy himself with the notoriously arduous task of hacking out a few lines of software, which, astonishingly, is something he has never done in the short but spectacular history of Netscape Communications Corporation [...] Not a single line of computer code. Never.
A message from the system administrator: 'I've upped my priority. Now up yours.'
Ok seriously, does anyone RTFA anymore? How about the comments? This is a clear PR stunt aimed at producing more leads for Marc's new company. And ZDnet, that fine bastion of even-handed IT reporting, has once again saved us all by printing only the relevant facts. Just once I'd liek to believe that one of my old IT heroes didn't sell out and become a corporate whore (can you say RMS anyone?).
-- People who think they know it all, really annoy those of us who do!
I've worked in a few large outfits and in my own small business, and I can testify there is almost zero correlation in a large office between the work that you do and people's perception of what you do.
The people who have the most problems, the ones who have a terrible catastrophe which just always seems to happen to them, are seen as the problem solvers. Despite the fact that their own lack of organization, incompetence, or laziness often brings these things upon them. No matter, they can proudly trumpet how they once again "saved the company" and worked 30 hours straight. The ones like me who prevent the problems, who organize their day so that nothing exciting happens if it can be avoided, and quietly solve problems on their own without assistance before people notice them, are seen as either invisible or lazy.
And no, I don't work a 30 hour day. Ever. I don't need to. I'm not bitter... gak!
The baby's fine -- please stop sending business cards.
You've got to be kidding me. This problem is as old as IT itself.
I think a better point to critique on his phone analogy is the implied point that the phone system isn't held together with "bailing wire" or "chewing gum." Nope, it's all pretty standardized, well-integrated equipment. Why is the phone service so much more "professional" than IT services?
;^)
Because phone service is a relatively well-defined, consistent, limited problem domain. Internet servers, dynamic web sites, and local security are loosely-defined, constantly shifting, open-ended problem domains. They're very different, and you can't compare one to the other.
However, for certain applications, there are well-defined standards, and well-defined practices. Still, for a lot of IT, it's a matter of custom engineering and architecture. For example, online content management: you can buy one of the management engines off-the-shelf, which will probably do most of what you need in a structured manner. For CRM, well, there's about a dozen of those. These packages are well-behaved in that it provides a well-defined interface, but that's not always an option (i know, i used to do data migration for small- to medium-sized businesses. at the low end, when you change systems, you'd better damned well know perl or some other text processing language to massage the data--that is, you need to be good with your bailing wire).
In the future, this situation will hopefully be better with standardization (mostly using XML it seems, even though the actual encoding doesn't really matter.. we could have standardized years ago, but nobody saw any benefit then). Having done data migration in the past, i'm all for keeping things disparate and non-standard, but that's because the work pays well and is fun
A better analogy might be a pool of corporate autos. Except that you don't have to interconnect the cars to get them to share load dynamically, or access content generated on one to form a report on the other, etc. A lot of IT is like trying to drop a big old hemi into a metro, or getting a suburban to go anywhere with just metros to provide power (two in front and one in back, it might go up a hill!).
Overall, I was not impressed with this article, but I'm afraid it's going to carry more clout than it should. oh well.
I am not typically supportive of
"My old approach was to say to the boss "What ya gonna do? Fire me? I'm too low paid and I claim slacktime NOW." Wierdly, it usually worked and I got my slack off time. Turns out he needed me more than I needed him."
I usually just try to always look busy. I have such a reputation for hard work, that in the (rare) slack times I can usually screw off and get away with it.
But you get to that point by ACTUALLY working hard when there is work to do.
Thing is, when you are a network admin for an enterprise, there is ALWAYS work that you can be doing.
Corporatism != Free Market
No doubt. Andreesen is trying to fabricate a crisis so he can save his worthless company, Opsware (LoudCloud), from insolvency and his career from the where are they now file.
Someone you trust is one of us.
Telephony is a mature technology that doesn't completely change the way it works every five years. For 80 or so years the way the signals got routed didn't change much at all. Then exchanges went digital, and, for the transition period, it was all bailing wire like the article says, except that the telephone companies had - dare I say it - telephone number budgets to pay for the changeover.
By comparison, the rate of change in IT is still very high. We've gone from mainframe to micro, from thin client through peer networking and back to thin client, from standalone to the Internet, we've done dial-up, ADSL, wireless...
... and one of the main reasons for all the bailing wire is that no company can afford to throw away all it's infrastructure every 24 months. If telephone systems stored data, and if handsets ran bespoke software, there would still be a few manual exchanges in use for backward compatibility.
Virtually serving coffee
One of the biggest problems as I see it is that management dont appreciate how important IT has become in their company. Looking at my company, I think they still relate the IT department to the same IT department of five years ago when to be fair the technology was a little easier to grasp and there were much less computers in offices.
Take networking for example - it used to be in our place BNC and the occasional run of UTP cable - all attached to relatively unintelligent devices. Now its all Cisco switches, fiber and cat5e - and it really is a full time job in itself managing a large network with so many 'intelligent' devices.
Also taking into account the addition of so many more servers (SQL, Mail, Finance stuff, Student Records, DNS, Proxy..) - the list is endless. Again, these systems have really bloomed in the past 4 or so years, at least for where I work.
I guess they dont see how much goes on behind doors when it comes to this business..
"Hey! Unless this is a nude love-in, get the hell off my property!!"
Why don't we just go back to the mainframe way of doing things?
When users had dumb terminals on their desk, they had the illision of a full power computer, but it was actually a small box with few moving parts that was linked to high-powered computers (or cluster of computers) who were actually doing the heavy lifting. Since all of the functional components were in the Computer Room, there was rarely a need for tech staff to touch the dumb terminals, and the tech people could work in their own distraction-free environment.
What's more, failures could sometimes be abstracted away from users. Hard drive failures happen, that's a fact of the technology. However, if a HD fails on a user's desk, it means that user has lost the use of their computer until it's fixed. If an HD fails in the datacenter, there's usually a backup of the data which can be put into play immediately by mounting the backup on a good drive that's already spinning. To the users, the disk crash can be practically invisible.
There's already tech technologies such as X-Windows and Windows Terminal Service with which to create GUIs on a dumb terminals. Why does the common secretary need a full-powered PC on her desk anyway?
Well, I work as a sysadmin at a hosting co, and the farm of FreeBSD and Red Hat machines I administer is running beautifully. Automatic upgrades, email notifications of all the important stuff, distributed shell program to run commands on all machines at once, Tripwire and firewalls to keep an eye on the hackers. If I'm not working on any projects, I only actually "work" maybe 4 hours a day.
How do I justify my existence? Easy, the colo machines that I don't administer get hacked or broken almost once a month. Each time, I send a report and I also tell my boss "by the way, I already took care of this, we won't have this problem".
(Maybe I should send a report each month: "security holes that DIDN'T affect us this month".)
After a while he realized that I'm like an insurance policy (as well as my Windows counterpart who does basically the same thing).
Now I just hope that all these colo customers don't sign up for our monitoring service, that will really make me invisible.
Another less ethical option is to leave a few unsolved but safe problems in the machines so that you have a small fire to put out each week, to make yourself look busy.
Comment removed based on user account deletion
Mind you virtually no one is fooled by the alt-tab: When everytime you walk near somebody their screen flickers as they swap windows, or they fervently hit the keys whenever someone nears, you know that they are slacking, but to many people the thought process will be that they're looking at porn or playing a game.
If you aren't willing to do it as "research" (for instance as funny as it sounds Slashdot can be a work-related website for many tech firms. It is "putting your ear to the ground" in a sense), then I recommend what another recommended which is that you browse in a manner that is inconspicuous and requires no screen swapping: Lynx, or grahics-less.
If you haven't checked into nagios yet, you owe it to yourself to do so. Now. It's a monitoring application that can take action on problems. That's the first step to automating things in the datacenter. It's open-source, and it's highly useful, if a little tricky to get working.
I moved out of a group running a lot of big Sun machines (I set up an E10K for them) because of managerial issues. Before I left, we had a budget item for about $250,000 to set up a monitoring and job-scheduling application. It was going to take *another* Sun box to run, and we were being told that it would take 3 months to get it all set up and configured.
With Nagios, I can do everything we they were talking about implementing. I spent 3 weeks, and it cost me nothing. I employed a dual PII 266 that was collecting dust. (I also used an old P166 as a dedicated kiosk for showing the web page.) My boss and my co-workers think it's great. I'm dying to show it to my old group...
Acts 17:28, "For in Him we live, and move, and have our being."
One thing that thin clients don't solve is... Laptops. A lot of companies deploy a bunch of laptops, cause they're not much more expensive than desktops (exp. if they lease everything), for those employees that need to work from the office, bus/train, home, all day/all night. So if you design you network to be server-centric, laptop users are left out in the cold.
The problem here isn't really in applying the patches (especially MS patches), it's the damage those patches do to the applications running on those servers.
Microsoft routinely sends out patches that negatively affect their own damned applications...much of the Sys Admins time is spent editing the goddamn registry to restore order.
Not that I need to say this, but if you automate bad code, your problems rise exponentially....
Error encountered in IAWebSig.clsSig.Create: Last Procedure: sPrc_Ins_tblSig
God.. I pity you peeps..
Seriously, any time I go to work for a new company, the first 2 months are spent automating everything across the board. I'll generally get no sleep quite a few nights writing software, and when I'm done, I just tell management that because of errors in the past, all work orders will be processed through this new interface. I give 'em an internal web addy and then go play Quake/EQ/etc. for the rest of my time there (which has never been less than 1 1/2 years which I get vested options from). The websites handle all the dummy checking, logs all processes to another system to check periodically to make sure all works well, and performs whatever request they want.
I order spares for about every piece of equipment in the building including spare switches and 1-2 spare servers for the occassional *frying* motherboard. If the servers are set up in non-redundant fashion, I make sure load balancers (or happy cisco 6500) are ordered, every server has a backup, all backups are automated and working properly, and put in the DR (Disaster Recovery) report JUST incase they want to go that route.
Frankly, I have MAYBE 1 hour of downtime a year, and that's usually attributed to my tripping over a cable that some numb-nutz (who's gonna get chewed out for an hour) layed outside of the wire maintenance tray. Only reason I move jobs is because I just get WAY too bored. Admining is easy for those who aren't inept.
-What have you contributed lately?
Back in the early 80s (when Marc was in what grade?) the business press ran articles about how software was getting so good that soon we wouldn't need programmers, because writing software would itself be handled by software.
I predict that we'll have software that can write major software at just the same time as we have software that can write convincing novels. In both cases you have the task of putting together language that respresents a broad swath of messy reality.
Now, systems administration may be more like writing a good technical manual than writing a good novel. Ever notice how many good tech handbooks there are out there? You haven't? Maybe it's because novels are easier. Good systems administration is about leveraging people strengths with machine strengths, and vice versa. Automation without the human element is as uncompetitive as, well, the human element without automation these days.
"with their freedom lost all virtue lose" - Milton
Why?
Well because the software we use today doesnt lend itself volontarily to extencive automation. You can automate patch installations, users added to all relevant systems at one click of a button, backup and all such things. The problem that some people seems to have a hard time grasping is that software sucks mostly. It is ridden with faults that make any automation fail randomly no matter how well it is implemented. Thats where most IT staff is doing their job, straighten out faults in the software and installing it. Support is also very hard to automate.
Before any automation can be used on a daily basis software must get much better and have much less bugs at shipping date than today. Its a very wrong approach to go backwards and automate fixing of faults related to bugs. Fix the bugs instead.
HTTP/1.1 400
Your Loudcloud was supposed to do this for us and it failed.
Why did it fail?
1 No one really knows how to do it
2 The infrastructure is too expensive
3 Customer requirements are too dissimilar from one another
4 No one has the balls to tell customers their requirements are crazy and impossible
5 Transition costs are poorly understood
6 Exeutives are measured by overhead and customer satisfaction and not doing the right job the best way
7 People are not a resource they are an overhead item
Maybe I'm on crack, but...
WTF is this? What is he trying to say? Where's the paradigm?
The piece (it isn't even really an article) says we (IT) aren't staying on top of our workloads, but doesn't even begin to suggest what to do to change that.
This isn't news, this is flaimbait.
On the other hand, it was nice to see one or two paragraphs pandering to my IT-ego, telling me its ok, someone undertsands my pain.
On the gripping hand, everyone I come into contact with at my company (and even a lot that I don't) know that I do my job well. I am consistantly complimented on my performance, my eagerness & abilty to help others, etc. As an added bonus, because my users are well informed, they're able to give me good error reports if something goes wrong. Because they are able to effectively report problems, those problems are solved quickly. Because of that, I have very little downtime.
Perhaps Marc is thinking about the BOFHs, and how to allow them to keep up with the rest of us?
It all sounds a lot like what IBM's Sam Palmisano was preaching back in October, about "eBusiness on demand".
The idea that impressed me then was the thought that nobody would seriously consider generating their own electricity now that it's a utility. But back in the early days companies and communities did just that. Same thing today with computing, but tomorrow...
It strikes me there's a shade more to IBM's vision than there is to Andreessen's, though. Check out the IBM version here, with links to some more in-depth material.
(Marc Andreesen)
I'm an egotistical, talentless hack who's latest
stupid idea, loudcloud, failed, so I sold off half
of the company to a bunch of Unsuspecting, good old boy rubes, since my status as "Internet Goldenboy" is in question.
(/Marc Andreesen)
I used to work for a rather lame start-up, which was run by a member of the aohell/nutscrape
cronie network of good old boys (that racist, ignorant, sexual harassing homophobic prick, The only person I know of to have a wired article about how much of a jerk he is. Opsware was
crap. It was slow, buggy, and caused us downtime
that wasn't really downtime according to loudcloud's incredible staff of marketing and law
employees. We were a startup with low funding, yet we spent $800k a month for service from them that we could have built ourselves at exodus or equinix for $200k a month.
If anybody wants information on a REAL movement
in automated systems administration, go to Infrastructures.org A movement based on Steve Traugott's Usenix presentation Bootstrapping the infrastructure.
"And how can this be? For he is the