Slashdot Mirror
The New IT Crisis
Matey-O writes "You've succeeded in delivering 5 9's, your server farm is a well oiled machine, the helpdesk lines lie dormant. No? Well then how do you get credit for the work you do, when all that's noticed is the downtime? When the IT budget has to be justified, and you're overworked, undermanned, and you have to apply three patches to 100 servers before Close of Business, what has to change in IT before we melt down? Marc Andreessen has an interesting article on what has to happen to IT next."
Well then how do you get credit for the work you do, when all that's noticed is the downtime?
I feel that if I work hard (and smart enough), then I deserve free time every once in awhile. After all, I earned it.
But, managers don't understand this. So, I relax by reading The Onion or Freshmeat at work, but always make sure my hand can quickly hit ALT+TAB to get back into my work window (usually Emacs).
and you have to apply three patches to 100 servers before Close of Business
Not flaming, but a fact.. Microsoft SMS will get that done for you with a couple of clicks.. literally..
And we're expecting to receive a beta copy of SMS 2003 (incl. Active Directory, the big feature).. so i'm kind of looking forward to that..
I have to say.. even though we run 2000 on our servers.. SMS is a dam good product and a timesaver!
"The ones who dont do anything are always the ones who try to pull you down" -- Henry Rollins
Create presentations, monitoring
systems with a lot of 3d and 2d diagrams
and histograms.
Bosses love that.
Since when do telephone systems maintain themselves? Last I heard, my voice was still running along wires and branching about in switching stations. Unless I'm mistaken, they still require maintenance. Granted that the operating costs have been reduced (fewer operators = fewer wages), but you could take a similar stance on IT. What about self-help forums where you can search a knowledge base to find answers? These can replace a lot of man hours of technical support work. There's always going to be some kind of human element to whatever equation. We're never going to find empty power plants that can generate their own electricity indefinitely, because there's always going to have to be human intervention.
Some businesses demand complex solutions, and I fail to see how these complex solutions are going to be met by turnkey solutions -- where a manager can go out, purchase a server, turn it on, and have it run his business for a year without any kind of customization whatsoever.
Ok. Just becuase it Marc Andreessen doens't mean that it's news.
This is an editorial, not an article.
How about talking about Grid computing. Or Organic IT. Developing systems and monitoring capabilities that go beyond telling you things are down, or when they're too busy. They actually add capacity on the fly through virtualization, taking from inactive systems to cover for the active system.
I think Marc once had a vision, but I'm not sure how strong of a visionary he is these days...
-- You can't idiot-proof anything, because they're always coming out with better idiots.
Ok seriously, does anyone RTFA anymore? How about the comments? This is a clear PR stunt aimed at producing more leads for Marc's new company. And ZDnet, that fine bastion of even-handed IT reporting, has once again saved us all by printing only the relevant facts. Just once I'd liek to believe that one of my old IT heroes didn't sell out and become a corporate whore (can you say RMS anyone?).
-- People who think they know it all, really annoy those of us who do!
what has to change in IT before we melt down ? Lots
What WILL change? nothing
for every burnt out admin thats going to quit theres 5 more waiting to take his place
good companies keep good employees
eather your not a good employee
or your not working at a good company
The More Knowledge you have the Luckier you Get- J.R. Ewing
I've worked in a few large outfits and in my own small business, and I can testify there is almost zero correlation in a large office between the work that you do and people's perception of what you do.
The people who have the most problems, the ones who have a terrible catastrophe which just always seems to happen to them, are seen as the problem solvers. Despite the fact that their own lack of organization, incompetence, or laziness often brings these things upon them. No matter, they can proudly trumpet how they once again "saved the company" and worked 30 hours straight. The ones like me who prevent the problems, who organize their day so that nothing exciting happens if it can be avoided, and quietly solve problems on their own without assistance before people notice them, are seen as either invisible or lazy.
And no, I don't work a 30 hour day. Ever. I don't need to. I'm not bitter... gak!
The baby's fine -- please stop sending business cards.
Well then how do you get credit for the work you do, when all that's noticed is the downtime?
:)
The very reason I left. When something crashes, who gets blamed? When users forget their passwords (which are usually something as simple like their friggin' username), who gets hasseled? When admin lays down an impossible time table with ridiculous performance expectations, who gets told "make this work or else?" When the company starts loosing money due to poor business decisions and/or the economy being in the toilet, who's the first dept to get cut?
Not as appealing as those tech school commercials make it out to be, huh?
I think a better point to critique on his phone analogy is the implied point that the phone system isn't held together with "bailing wire" or "chewing gum." Nope, it's all pretty standardized, well-integrated equipment. Why is the phone service so much more "professional" than IT services?
;^)
Because phone service is a relatively well-defined, consistent, limited problem domain. Internet servers, dynamic web sites, and local security are loosely-defined, constantly shifting, open-ended problem domains. They're very different, and you can't compare one to the other.
However, for certain applications, there are well-defined standards, and well-defined practices. Still, for a lot of IT, it's a matter of custom engineering and architecture. For example, online content management: you can buy one of the management engines off-the-shelf, which will probably do most of what you need in a structured manner. For CRM, well, there's about a dozen of those. These packages are well-behaved in that it provides a well-defined interface, but that's not always an option (i know, i used to do data migration for small- to medium-sized businesses. at the low end, when you change systems, you'd better damned well know perl or some other text processing language to massage the data--that is, you need to be good with your bailing wire).
In the future, this situation will hopefully be better with standardization (mostly using XML it seems, even though the actual encoding doesn't really matter.. we could have standardized years ago, but nobody saw any benefit then). Having done data migration in the past, i'm all for keeping things disparate and non-standard, but that's because the work pays well and is fun
A better analogy might be a pool of corporate autos. Except that you don't have to interconnect the cars to get them to share load dynamically, or access content generated on one to form a report on the other, etc. A lot of IT is like trying to drop a big old hemi into a metro, or getting a suburban to go anywhere with just metros to provide power (two in front and one in back, it might go up a hill!).
Overall, I was not impressed with this article, but I'm afraid it's going to carry more clout than it should. oh well.
a) The phone industry had one task to do - get data over wires from point A to B, and however they did that was fine - users wouldn't have to be retrained if you replace old cables with fiber optic. That task is relatively simple, compared to the complexity of what is running and being distributed over the internet today. Automation works best when the target is static and clearly definable. I'm not sure either applies with servers/IT/internet.
b) A significant amount of trouble with maintaining systems comes from having to figure out lots of different pieces of hardware. Lots of random equipment makes IT support a great deal more difficult. There are two solutions:
1. Standardize all company hardware on a small number of systems/components, say one type of desktop, one type of server, and a few special purpose machines, and then only support those. Tools like VASystemImager then can make tasks like upgrading and bug fixing vastly simpler.
2. Use inexpensive thin clients interfacing to some powerful central server, ala Largo, and only have to maintain that central machine and swap out cheap, dumb clients. Also simplifies things tremendously.
People will no doubt point out that you have to run different types of OSs for different jobs and so on, but you can still use the central server/thin client approach and just make the connection to whichever OS you need transparent. It takes thought to set up, but once it is working you don't have 4000 individual support headaches to deal with. Only a few machines to upgrade, support, etc.
Unfortunately, this won't happen. First, you would have to have a truly MASSIVE infastructure upgrade, which replaces a working system. Riiight. Second, you need to have management willing to try something new and be patient to wait for the long term results. That's not how they think - they think next quarter profits. There is also sheer mental inertia to contend with.
It would be much easier for new companies to adopt this idea from the get go, than for older companies to adopt it. That may be where new, useful IT principles get applied.
The only way current companies will do something is if the system BREAKS, and I mean just totally stops functioning. Thats when they will wake up to the fact that significant changes are needed.
"I object to doing things that computers can do." -- Olin Shivers, lispers.org
Ok, I read through the article and came away with absolutely no information. He says some things we already know: data centers are expensive, IT people are overworked, and the rest of an organization only notices the technology folks when something breaks. So, what should we do about it?
.NET environment. It's basically the same as Java applets, but they call it "Smart Clients" to give you the impression that it's something they invented. Sounds a lot like Network Computing to me -- which simply means that Network Computing is a good idea after all! And now that Microsoft has "invented" it, the idiots who make up most of the world may finally start to adopt the idea. Make the desktop a stateless device like it was 20 years ago when we all had dumb terminals on our desks, and IT overhead will drop like a rock.
Well, here's where you expect an innovator like Andreesen to come up with a brilliant idea that's going to begin the next IT paradigm shift, but all he says is that we need to find some revolutionary way to automate our own stuff -- basically, to automate the act of automating things. And how? Well, he doesn't really know. He makes some vague reference to sending out automatic updates to hundreds of servers at a time, and that's it.
Real bright there, Marc. Automatic patches and updates. As if that's the answer. In the real world, you don't have a huge farm of servers that all run the same patchlevel of the same operating system. I've got a few hundred boxen behind the glass, for example, that are a mix of Linux, Solaris, FreeBSD, Windows 2000, and Windows NT. And I'd guess that at least 50 percent of them would experience some sort of problem if we were to just push updates out to them unattended -- different applications require different patchlevels and break on others.
Let's not forget the fact that there's more than just servers. There's infrastructure such as routers, firewalls, and switches. And of course there is the dreaded desktop, which is probably the source of 90+ percent of IT headaches. Until the IT world wakes up and gets the hell off local desktops, the maintenance nightmare will continue. Seen what Microsoft is doing lately? Their vision of the future is one in which applications are loaded through a browser and executed in a local
The other trend you're going to start to see is outsourcing. People are realizing that it's expensive to build and run a data center. Fortunately, you don't have to. All you have to do is run your servers at a hosting center that knows how to do outsourced IT (as opposed to just hosting web sites, like the first generation of centers like Exodus did).
There are ways of streamlining IT after all. Unfortunately, Marc Andreesen didn't touch on any of them. I give this article a "C minus."
Tired of FB/Google censorship? Visit UNCENSORED!
I am not typically supportive of
Going Postal?
Yes, going Postal. It's hot, it's new, it's... Satisfying.
"Cool and smooth, I don't know how I dealt with those bastards that kept asking 'this mouse?' without going Postal" -- Satisfied customer.
And now you too can try it, for the low low price of a single aluminum hardball bat. Spun aluminum with a non-slip rubber grip means never having to say "For all that is good and holy it does not optimize your connection!"
Keep your packets off my GNU/Girlfriend!
Wow, that was insightful. So you mean IT is overworked and that the it'd be nice if IT could become a more automated process? Wow Marc, you've really stunned me with your insights!
DUH! I mean c'mon, the reason why IT is such a mess is because all the IT staff are being expected to do more with less and do it faster. They are overworked fighting fires which means they aren't given the resources to do advanced planning and put together systems that would really get things right. He makes it sound like this is the responsibility of the IT people, but it's really the responsibility fo the business as a whole to have some foresight and help these IT people out.
Overworked sysadmins do what has to be done to keep something working RIGHT NOW. Sysadmins with some free time will spend their efforts writing scripts, automating tedious tasks, and making sure fires don't happen. When fires do happen, they have the time to deal with them effectively because they've had a chance to automate a lot of the other tasks.
I have known many a sysadmin and I have never met a one who wasn't constantly pissed off because he lacked resources. A friend of mine was sysadmin for a company that wanted to have 24 hour uptime for their systems. He was the ONLY sysadmin. That sort of crap happens constantly in the IT world because the other members of the company have the wrong atttitude about IT, that it's an expense. If they looked at it like they look at factories and buildings, as an infrastructure investment, then you'd probably see a lot more happy IT managers out there.
This sig has been temporarily disconnected or is no longer in service
One of the biggest problems as I see it is that management dont appreciate how important IT has become in their company. Looking at my company, I think they still relate the IT department to the same IT department of five years ago when to be fair the technology was a little easier to grasp and there were much less computers in offices.
Take networking for example - it used to be in our place BNC and the occasional run of UTP cable - all attached to relatively unintelligent devices. Now its all Cisco switches, fiber and cat5e - and it really is a full time job in itself managing a large network with so many 'intelligent' devices.
Also taking into account the addition of so many more servers (SQL, Mail, Finance stuff, Student Records, DNS, Proxy..) - the list is endless. Again, these systems have really bloomed in the past 4 or so years, at least for where I work.
I guess they dont see how much goes on behind doors when it comes to this business..
"Hey! Unless this is a nude love-in, get the hell off my property!!"
"Not only that, our company recently fired our most talented engineer without cause (the week before Thanksgiving) becuase he could hire a "paper tiger" (ie: MCSE) for $5K a year less, and a H1-B who he brought in and treats like a slave.
"Don't get me started about this H-1B thing. You will get my britches in knots again."
The "paper tiger" MCSE/CNE, I object to. Not the H1-B. He was hired 3 months before the boss let my friend go (who was my mentor, and largely respobsible for why I'm as good as I am). The H1B in question is bloody brilliant. One of the hardest working and smartest people I've ever met. He remembers anything I teach him after showing it to him ONCE... I can only WISH I were that smart.
I object to H1-B more because it VICTIMIZES those who come here on that kind of visa more than I object to skilled immigration. The USA is better off letting people like him into this country, especially when it deprives an opressive country (Iran) of a brilliant mind. But scumbags like my employer shouldn't be allowed to bring him in on a H1-B "slave" visa...
Corporatism != Free Market
This is what feeds Andreeson's IT crisis today: the fact that technology professionals took their apparent suffering and feelings of being not understood, and used it to isolate themselves. They refused to act like businesspeople in an organization that lives and dies by its profit and loss statement. They complain about how management doesn't appreciate them, but how many learn to do a cost analysis that will show the business reasons for buying software X or hardware Y? In other words, the glorification of the geek in the 90s gave a lot of geeks the idea that they didn't have to learn the language of business to survive there. That's why they're underfunded, underappreciated, and harassed.
I've had the benefit of a boss who demanded a business analysis for any significant technology initiative at the company. He doesn't get computers, but he understands ROI. He understands a well-presented business case for anti-virus software. We have a wireless network in our new facility in Texas for a real-time inventory management system for one reason: my cost analysis showed that the implementation costs would be recovered within a year because of labor saved from eliminating batch-mode downloading, and that the cost over five years of our wireless system was ~15% of the batch-mode system.
When geeks figure out that they have to speak the language of the business, then the IT department gets properly funded, properly respected, and properly treated.
Anyone who loves or hates any language, platform, or manufacturer, doesn't know what they're talking about.
If you haven't checked into nagios yet, you owe it to yourself to do so. Now. It's a monitoring application that can take action on problems. That's the first step to automating things in the datacenter. It's open-source, and it's highly useful, if a little tricky to get working.
I moved out of a group running a lot of big Sun machines (I set up an E10K for them) because of managerial issues. Before I left, we had a budget item for about $250,000 to set up a monitoring and job-scheduling application. It was going to take *another* Sun box to run, and we were being told that it would take 3 months to get it all set up and configured.
With Nagios, I can do everything we they were talking about implementing. I spent 3 weeks, and it cost me nothing. I employed a dual PII 266 that was collecting dust. (I also used an old P166 as a dedicated kiosk for showing the web page.) My boss and my co-workers think it's great. I'm dying to show it to my old group...
Acts 17:28, "For in Him we live, and move, and have our being."
Back in the early 80s (when Marc was in what grade?) the business press ran articles about how software was getting so good that soon we wouldn't need programmers, because writing software would itself be handled by software.
I predict that we'll have software that can write major software at just the same time as we have software that can write convincing novels. In both cases you have the task of putting together language that respresents a broad swath of messy reality.
Now, systems administration may be more like writing a good technical manual than writing a good novel. Ever notice how many good tech handbooks there are out there? You haven't? Maybe it's because novels are easier. Good systems administration is about leveraging people strengths with machine strengths, and vice versa. Automation without the human element is as uncompetitive as, well, the human element without automation these days.
"with their freedom lost all virtue lose" - Milton
My team runs just under 6 dozen web and database servers ( Solaris and Linux ) for the University of Michigan using an open source system management suite called 'radmind' and I can't say enough good things about it ( I'm not one of its developers, so I can get away with this ): fast, secure, stable, standards based, and makes a little thing like patching several dozen servers a breeze ( though ... what kind of freak patches in the middle of the day? ).
Incidentally, the CTO of loudcloud ( a.k.a. opsware ) is Tim Howes, of LDAP fame and formerly of the UMich RSUG ( the same group that has since developed radmind ). small world.
is the one you never see. That idiot running desk to desk doesn't have any idea what they are doing. The one that shows up at 8 and leaves at 6 and appears to be reading screen after screen of pr0n. That SA is the bomb.
Worship the SA, do not replace the SA with a small shell script.
This
Seriously, I don't get the money to buy redundent Cisco routers. I don't get to buy extra switches and motherboards.
And the little internal web addy, that has been in place for 2 years? Some tool of a manager who is having a hissy fit, just walked into my office and asked why I haven't fixed his problem that he never told anyone about.
Please......
Whenever you are dealing with people who have almost as many deadlines as you do are involved. Things are never that easy.
Now where I will agree with you are scripts. Learn them. Use them. Love them. But do you have an IDS system? Who verfiys those alerts? Who checks your server logs for descrepencies? Admining is rarely easy.
I'm not drunk, I just have a speech impediment. And a stomach virus. And an inner ear infection.
I know exactly what he's writing about. I was just fired from my job of two and a half years. The reason? Check my sig. That's right, we started a personal website in our own time, with our own reasources, and because the CEO didn't like it, we were out. "Decreasing company morale" was the reason we were given. They fired me, who was the Sysadmin, the webmaster, and the only PC technician.
Forget the fact that for the last two and a half years, I haven't had a real vacation, because I got a call EVERY FREAKIN' DAY about some minor issue.
Forget the fact that I worked a minimum of 2-3 hours every night on company stuff, 'cause I wasn't allowed to make ANY changes during the day. (The night before I was fired, I spent 3 hours writing a script to fix a problem that was affecting only 1% of the users. No real problem, but I didn't want them to have to deal with ANY issues.)
Forget all of that, just get the fuck out. I take some solace in the fact that two days after they fired us they sent out a global e-mail of "Please bear with us, it'll take up to five days for your calls to be answered." And, e-mail was down for almost a week because no one but us knew how it was setup!
IT needs to get the respect it deserves. In this era of decreasing budgets, the only way companies will be able to make any money is to increase efficiency, and that means automation.
Which is more painful? Going to work or gouging your eye out with a spoon? Find out!
http://www.workorspoon.com
Patching servers, fixing machines, crawling under desks. If you want a job with a fair bit of action, work at a school. If there's anyone who can find a way to screw something up, or do something unexpected, it's a 15-yr-old with a keyboard and mouse. That, or it's my 57-yr-old grandparents that somehow magically manage to screw things up that should be unscrewupable.
Combine that with archaic computers, underfunding, etc etc... and it's an exciting job.