Slashdot Mirror
The New IT Crisis
Matey-O writes "You've succeeded in delivering 5 9's, your server farm is a well oiled machine, the helpdesk lines lie dormant. No? Well then how do you get credit for the work you do, when all that's noticed is the downtime? When the IT budget has to be justified, and you're overworked, undermanned, and you have to apply three patches to 100 servers before Close of Business, what has to change in IT before we melt down? Marc Andreessen has an interesting article on what has to happen to IT next."
Well then how do you get credit for the work you do, when all that's noticed is the downtime?
I feel that if I work hard (and smart enough), then I deserve free time every once in awhile. After all, I earned it.
But, managers don't understand this. So, I relax by reading The Onion or Freshmeat at work, but always make sure my hand can quickly hit ALT+TAB to get back into my work window (usually Emacs).
Create presentations, monitoring
systems with a lot of 3d and 2d diagrams
and histograms.
Bosses love that.
Since when do telephone systems maintain themselves? Last I heard, my voice was still running along wires and branching about in switching stations. Unless I'm mistaken, they still require maintenance. Granted that the operating costs have been reduced (fewer operators = fewer wages), but you could take a similar stance on IT. What about self-help forums where you can search a knowledge base to find answers? These can replace a lot of man hours of technical support work. There's always going to be some kind of human element to whatever equation. We're never going to find empty power plants that can generate their own electricity indefinitely, because there's always going to have to be human intervention.
Some businesses demand complex solutions, and I fail to see how these complex solutions are going to be met by turnkey solutions -- where a manager can go out, purchase a server, turn it on, and have it run his business for a year without any kind of customization whatsoever.
Ok. Just becuase it Marc Andreessen doens't mean that it's news.
This is an editorial, not an article.
How about talking about Grid computing. Or Organic IT. Developing systems and monitoring capabilities that go beyond telling you things are down, or when they're too busy. They actually add capacity on the fly through virtualization, taking from inactive systems to cover for the active system.
I think Marc once had a vision, but I'm not sure how strong of a visionary he is these days...
-- You can't idiot-proof anything, because they're always coming out with better idiots.
what has to change in IT before we melt down ? Lots
What WILL change? nothing
for every burnt out admin thats going to quit theres 5 more waiting to take his place
good companies keep good employees
eather your not a good employee
or your not working at a good company
The More Knowledge you have the Luckier you Get- J.R. Ewing
Well then how do you get credit for the work you do, when all that's noticed is the downtime?
:)
The very reason I left. When something crashes, who gets blamed? When users forget their passwords (which are usually something as simple like their friggin' username), who gets hasseled? When admin lays down an impossible time table with ridiculous performance expectations, who gets told "make this work or else?" When the company starts loosing money due to poor business decisions and/or the economy being in the toilet, who's the first dept to get cut?
Not as appealing as those tech school commercials make it out to be, huh?
a) The phone industry had one task to do - get data over wires from point A to B, and however they did that was fine - users wouldn't have to be retrained if you replace old cables with fiber optic. That task is relatively simple, compared to the complexity of what is running and being distributed over the internet today. Automation works best when the target is static and clearly definable. I'm not sure either applies with servers/IT/internet.
b) A significant amount of trouble with maintaining systems comes from having to figure out lots of different pieces of hardware. Lots of random equipment makes IT support a great deal more difficult. There are two solutions:
1. Standardize all company hardware on a small number of systems/components, say one type of desktop, one type of server, and a few special purpose machines, and then only support those. Tools like VASystemImager then can make tasks like upgrading and bug fixing vastly simpler.
2. Use inexpensive thin clients interfacing to some powerful central server, ala Largo, and only have to maintain that central machine and swap out cheap, dumb clients. Also simplifies things tremendously.
People will no doubt point out that you have to run different types of OSs for different jobs and so on, but you can still use the central server/thin client approach and just make the connection to whichever OS you need transparent. It takes thought to set up, but once it is working you don't have 4000 individual support headaches to deal with. Only a few machines to upgrade, support, etc.
Unfortunately, this won't happen. First, you would have to have a truly MASSIVE infastructure upgrade, which replaces a working system. Riiight. Second, you need to have management willing to try something new and be patient to wait for the long term results. That's not how they think - they think next quarter profits. There is also sheer mental inertia to contend with.
It would be much easier for new companies to adopt this idea from the get go, than for older companies to adopt it. That may be where new, useful IT principles get applied.
The only way current companies will do something is if the system BREAKS, and I mean just totally stops functioning. Thats when they will wake up to the fact that significant changes are needed.
"I object to doing things that computers can do." -- Olin Shivers, lispers.org
Ok, I read through the article and came away with absolutely no information. He says some things we already know: data centers are expensive, IT people are overworked, and the rest of an organization only notices the technology folks when something breaks. So, what should we do about it?
.NET environment. It's basically the same as Java applets, but they call it "Smart Clients" to give you the impression that it's something they invented. Sounds a lot like Network Computing to me -- which simply means that Network Computing is a good idea after all! And now that Microsoft has "invented" it, the idiots who make up most of the world may finally start to adopt the idea. Make the desktop a stateless device like it was 20 years ago when we all had dumb terminals on our desks, and IT overhead will drop like a rock.
Well, here's where you expect an innovator like Andreesen to come up with a brilliant idea that's going to begin the next IT paradigm shift, but all he says is that we need to find some revolutionary way to automate our own stuff -- basically, to automate the act of automating things. And how? Well, he doesn't really know. He makes some vague reference to sending out automatic updates to hundreds of servers at a time, and that's it.
Real bright there, Marc. Automatic patches and updates. As if that's the answer. In the real world, you don't have a huge farm of servers that all run the same patchlevel of the same operating system. I've got a few hundred boxen behind the glass, for example, that are a mix of Linux, Solaris, FreeBSD, Windows 2000, and Windows NT. And I'd guess that at least 50 percent of them would experience some sort of problem if we were to just push updates out to them unattended -- different applications require different patchlevels and break on others.
Let's not forget the fact that there's more than just servers. There's infrastructure such as routers, firewalls, and switches. And of course there is the dreaded desktop, which is probably the source of 90+ percent of IT headaches. Until the IT world wakes up and gets the hell off local desktops, the maintenance nightmare will continue. Seen what Microsoft is doing lately? Their vision of the future is one in which applications are loaded through a browser and executed in a local
The other trend you're going to start to see is outsourcing. People are realizing that it's expensive to build and run a data center. Fortunately, you don't have to. All you have to do is run your servers at a hosting center that knows how to do outsourced IT (as opposed to just hosting web sites, like the first generation of centers like Exodus did).
There are ways of streamlining IT after all. Unfortunately, Marc Andreesen didn't touch on any of them. I give this article a "C minus."
Tired of FB/Google censorship? Visit UNCENSORED!
Glad you mentioned it. Yes, this is just Mark pushing his own product (Datacenter management tools). In fact, he already has a few big customers. And this is the same direction Sun, for example, is going with N1. But it is going to have problems.
You can't abstract away problems. And software is designed to be customized more than plug-in and exchange like lego blocks. Further, the 'leaky metaphor' problem hits these kind of systems big. (An example from a previous article, "UDP does not guarantee data delivery, and certainly not in the correct order, but TCP/IP does." So where is that guarantee if I cut a 1 foot break in the ethernet cable?)
Software is going to have to get a lot more dumb for this sort of thing to work.
Wow, that was insightful. So you mean IT is overworked and that the it'd be nice if IT could become a more automated process? Wow Marc, you've really stunned me with your insights!
DUH! I mean c'mon, the reason why IT is such a mess is because all the IT staff are being expected to do more with less and do it faster. They are overworked fighting fires which means they aren't given the resources to do advanced planning and put together systems that would really get things right. He makes it sound like this is the responsibility of the IT people, but it's really the responsibility fo the business as a whole to have some foresight and help these IT people out.
Overworked sysadmins do what has to be done to keep something working RIGHT NOW. Sysadmins with some free time will spend their efforts writing scripts, automating tedious tasks, and making sure fires don't happen. When fires do happen, they have the time to deal with them effectively because they've had a chance to automate a lot of the other tasks.
I have known many a sysadmin and I have never met a one who wasn't constantly pissed off because he lacked resources. A friend of mine was sysadmin for a company that wanted to have 24 hour uptime for their systems. He was the ONLY sysadmin. That sort of crap happens constantly in the IT world because the other members of the company have the wrong atttitude about IT, that it's an expense. If they looked at it like they look at factories and buildings, as an infrastructure investment, then you'd probably see a lot more happy IT managers out there.
This sig has been temporarily disconnected or is no longer in service
Remember the September story on how IBM, Sun, etc. wants servers to administer themselves? Remember how sysadmins had either lukewarm or negative reactions to it across the net?
So how do you soften people up to the idea? Wait a couple of months, release a low key but suggestive "article" to get the concept back in peoples' heads, then launch the offending software/hardware/schema again about a year later onto a public, now resigned to seeing the new 'feature' as inevitable. It's a standard pr tactic.
We saw it with Intel's P3 PSN fiasco, numerous webmail service privacy policy changes, and the XP activation scheme. And I'm not saying that all this is the work of evildoers, just that this is what it is.
I'm really not trying to troll here, and even though he makes some interesting points, it's very difficult to trust the business opinions of a man that has one major dropped ball in his experience and is now trying to push his automation software as the Next Big Thing in IT. I don't honestly think he has any clue what the NBT is, but neither does anybody else.
"Not only that, our company recently fired our most talented engineer without cause (the week before Thanksgiving) becuase he could hire a "paper tiger" (ie: MCSE) for $5K a year less, and a H1-B who he brought in and treats like a slave.
"Don't get me started about this H-1B thing. You will get my britches in knots again."
The "paper tiger" MCSE/CNE, I object to. Not the H1-B. He was hired 3 months before the boss let my friend go (who was my mentor, and largely respobsible for why I'm as good as I am). The H1B in question is bloody brilliant. One of the hardest working and smartest people I've ever met. He remembers anything I teach him after showing it to him ONCE... I can only WISH I were that smart.
I object to H1-B more because it VICTIMIZES those who come here on that kind of visa more than I object to skilled immigration. The USA is better off letting people like him into this country, especially when it deprives an opressive country (Iran) of a brilliant mind. But scumbags like my employer shouldn't be allowed to bring him in on a H1-B "slave" visa...
Corporatism != Free Market
Well, here's where you expect an innovator like Andreesen to come up with a brilliant idea that's going to begin the next IT paradigm shift, but all he says is that we need to find some revolutionary way to automate our own stuff -- basically, to automate the act of automating things.
What exactly has he done that was truly innovative? He happened to be at NCSA when they happened to be writing the first graphical web browser. To the best of my knowledge, he's just been somebody who was at the right place at the right time.
This sig has been temporarily disconnected or is no longer in service
There are many products that can do this ... even a well crafted batch file making use of task scheduler....anything that can save time doing redundant work is a good thing...
The problem lies in the work order - and you have to apply three patches to 100 servers before Close of Business .... unless you are rolling out said patches to 100 identical servers (OS/Patch Level/Hardware) or uptime is optional, you have slim chance for success...where does integration testing, application QA and a controlled pilot fit in before the end of day?
The quote "Fast, cheap and stable...pick two" applies here...
"Hatred is the coward's revenge for being intimidated"
This is what feeds Andreeson's IT crisis today: the fact that technology professionals took their apparent suffering and feelings of being not understood, and used it to isolate themselves. They refused to act like businesspeople in an organization that lives and dies by its profit and loss statement. They complain about how management doesn't appreciate them, but how many learn to do a cost analysis that will show the business reasons for buying software X or hardware Y? In other words, the glorification of the geek in the 90s gave a lot of geeks the idea that they didn't have to learn the language of business to survive there. That's why they're underfunded, underappreciated, and harassed.
I've had the benefit of a boss who demanded a business analysis for any significant technology initiative at the company. He doesn't get computers, but he understands ROI. He understands a well-presented business case for anti-virus software. We have a wireless network in our new facility in Texas for a real-time inventory management system for one reason: my cost analysis showed that the implementation costs would be recovered within a year because of labor saved from eliminating batch-mode downloading, and that the cost over five years of our wireless system was ~15% of the batch-mode system.
When geeks figure out that they have to speak the language of the business, then the IT department gets properly funded, properly respected, and properly treated.
Anyone who loves or hates any language, platform, or manufacturer, doesn't know what they're talking about.
It's not the up-front cost of the machine, it's the ongoing support costs. Thin client has a much longer working life, and takes much less time to support.
If one of my co-workers had been fired for that reason, I'd have quit on the spot. And so would most of my colleagues. Why? Because of loss of faith in the leadership (boss not understanding morale concept, and not knowing who'd be next, leads me to prefer to dictate my own future).
The boss would be left with an empty department to explain to his VP and justify his bad decisions. Not that it would happen - I have the total faith in my boss and the morale is good, people help each other out.
You Americans need to learn to fuck people back and not take shit from nobody. The SINGLE reason managers (in IT and elsewhere) can treat people like this, is that the people being screwed react with bending over and asking for more.
Umm no... That is not how most management calculates ROI.
Using your example, the following is how we calculate ROI before giving green-light to a project.
Assumed life of IT system = 5 years.
Current Cost of 5 monkeys over 5 years @ 30k/monkey/year = 30k * 5 * 5 = $750k
Cost to develop IT system = 3 high end IT people = $200k.
Cost to maintain and to keep a help desk jockey or two =around = $30k-50k per year = $250k over 5 years.
Total Investment over 5 years = 200k + 250k = 450k.
Return on Investment over 5 years= 750k - 450k = 300k.
You implied that the management does not take into account the ongoing costs into the ROI calculation. It very much does. That *is* the "investment".
The "return" is the the money saved on the monkeys.
As a guy who got his first start programming on an IBM mainframe, I just want to break down in hysterical laughter.
Early nineties: everything was to be decentralized. Get rid of that antique iron! Go Intel/Microsoft! Can the tech Luddites, bring in the Bright Yunguns, who will show us the way.
Mind you, I was one of those calling for the company to modernize on a client-server model.
But I've watched for, what, 20 years since this all started. I watched the expensive mainframes go into the landfill, and the new servers take their place.
I've watched those servers get more expensive. I've seen the communications hardware become massively expensive. I've watched the new Data Priesthood come into being. I've watched companies being wiped out converting to, and maintaining, the new paradigm.
Not that it would have been good to stay on the old iron. BUT...
We've come full circle. The hounds are baying for simpler solutions, and most importantly, for the elimination of the new mainframers, the admins.
I predicted it over ten years ago. The PC's will cluster into data centers, pseudo-dumb clients will spread onto everyone's desks and laps, the services will automate, and the present IT industry will convulse. A shakeout is coming. Time to get that teaching career started.
As a sidebar: in my experience, far more money was spent on executive games and perks than was ever lost on IT spending. Cost accounting is applied where management wants application.
Seriously, I don't get the money to buy redundent Cisco routers. I don't get to buy extra switches and motherboards.
And the little internal web addy, that has been in place for 2 years? Some tool of a manager who is having a hissy fit, just walked into my office and asked why I haven't fixed his problem that he never told anyone about.
Please......
Whenever you are dealing with people who have almost as many deadlines as you do are involved. Things are never that easy.
Now where I will agree with you are scripts. Learn them. Use them. Love them. But do you have an IDS system? Who verfiys those alerts? Who checks your server logs for descrepencies? Admining is rarely easy.
I'm not drunk, I just have a speech impediment. And a stomach virus. And an inner ear infection.
I predict that we'll have software that can write major software at just the same time as we have software that can write convincing novels. In both cases you have the task of putting together language that respresents a broad swath of messy reality.
Actually, we do already have software that can write software, it's called CASE, Comupter Aided Software Engineering.
Unfortunately, it's takes about as much effort to explain to CASE what you want your application to do as it would to write the application yourself anyway!
The long term result is a drastic decrease in quality products and service yet at higher cost. Higher price for lower reliability is not good business. Many companies are now asking, "What am I getting for my IT budget?" and the answer has not yet been provided except in marketing spew, thus further entrenching the cycle of waste and inefficiency. Ask any sucessful small business owner (except for those that are on some welfare tit, like with many government contracts) and what they tell you is that trust is the key to long term business relationships. Marketing is supposed to be about advertising your existence and what you offer, then giving facts on why it is worth their effort. Cute bullshit is funny, but is not professional except to those mindless sheep that let such superficial bungwipe actually overpower any business sense or logic and reason. Get rid of the bloat and focus on giving the customer what they want. If you have an IT department then ask what it is there for. If it is to fascilitate the rest of the business exchanges and dealings then it is wise to ensure that such an infrastructure is well placed. After all, would a Car dealership ever hope to stay in business if the customers had to deal with being rained on (leaky roof), smelling open/malfunctioning sewers, or being assaulted by biting dogs or other critters? Then why would a company allow its infrastructure to be underfunded? Problems in that department will undoubtedly affect the rest of the company.
I find it hard to take anyone seriously that would scoff at an underfunded and undermanned IT departments lack of efficiency, yet then turn around and use mickey mouse hardware and software for said IT departments. The fact that the IT industry pumps out such crap is a direct result of poor business leadership in which idiots are put in charge of purchase and deployment and whom are sold on "cutsey" adds and FUD. Personally I go for what works and I could care less how charming the sales staff is or what the actual commercials are currently using as their limirick of the month.
I know exactly what he's writing about. I was just fired from my job of two and a half years. The reason? Check my sig. That's right, we started a personal website in our own time, with our own reasources, and because the CEO didn't like it, we were out. "Decreasing company morale" was the reason we were given. They fired me, who was the Sysadmin, the webmaster, and the only PC technician.
Forget the fact that for the last two and a half years, I haven't had a real vacation, because I got a call EVERY FREAKIN' DAY about some minor issue.
Forget the fact that I worked a minimum of 2-3 hours every night on company stuff, 'cause I wasn't allowed to make ANY changes during the day. (The night before I was fired, I spent 3 hours writing a script to fix a problem that was affecting only 1% of the users. No real problem, but I didn't want them to have to deal with ANY issues.)
Forget all of that, just get the fuck out. I take some solace in the fact that two days after they fired us they sent out a global e-mail of "Please bear with us, it'll take up to five days for your calls to be answered." And, e-mail was down for almost a week because no one but us knew how it was setup!
IT needs to get the respect it deserves. In this era of decreasing budgets, the only way companies will be able to make any money is to increase efficiency, and that means automation.
Which is more painful? Going to work or gouging your eye out with a spoon? Find out!
http://www.workorspoon.com
Patching servers, fixing machines, crawling under desks. If you want a job with a fair bit of action, work at a school. If there's anyone who can find a way to screw something up, or do something unexpected, it's a 15-yr-old with a keyboard and mouse. That, or it's my 57-yr-old grandparents that somehow magically manage to screw things up that should be unscrewupable.
Combine that with archaic computers, underfunding, etc etc... and it's an exciting job.
Posolutely. I couldn't agree more.
I'd even take this a step further. Let's just say, for the sake of argument, that we reach Marc's promised land of "it just works" hands off IT administration. Then what?
Marc is missing the point. IT is not a necessary evil, it's a competitive advantage. I've sometimes told people who spend too much time attributing their problems to their computer that they should stop using it then. If you can do your job better without the computer, then by all means do.
I have yet to see anyone take me up on this challenge. They know, we all know, that despite their occasionally infuriating peccadillos, that computers make us more productive.
You cannot remain competitive by sitting on your hands. Marc's world of "it's all better so we can rest now" is pure fantasy. The problem with this article is that it portrays IT as a problem, rather than a solution and a competitive advantage. Who wouldn't like to compete against the company that decides it's achieved all essential IT objectives? You want to get your ass kicked, then stop trying to figure out better ways of doing things.
--Lawrence Lessig for Congress!
- Ask your non-geek boss how to make a business presentation on the business merits of a technology initiative. Ask her about cost analysis. Ask her about justifying new software to her bosses (what they're looking for).
- Pick up a textbook on managerial accounting (which is different than what accoutants do): it'll go into great detail on how businesses analyse costs and plan with them.
- Look at your company's profit and loss statement and figure it out. I started at the company as a cost analyst, and now I know exactly how the business is financially structured.
- Try doing it yourself: start a spreadsheet. List all the costs of something, say, upgrading to the latest version of Office. Include every cost you can think of: retraining, troubleshooting, labor spent installing, licence fees, costs to convert all the documents on the server. Don't just list dollar figures, list your calculation: 30 hours at $20.00/hr (but don't forget to include the cost of benefits and other expenses: an employee paid $10/hr costs the business $12-14/hr). Then list all the monetary benefits: time saved with zippity feature X, etc. Compare the bottom line: will the business benefit financially from the upgrade?
At it's heart, understanding the internals of a business means understanding it's programming language: finance. You don't understand how a business works until you understand the flow of money through it. Figure that out, and you'll be a respected geek who can talk to managers on their own terms.Anyone who loves or hates any language, platform, or manufacturer, doesn't know what they're talking about.
And IT must start taking steps forward, if it wants to be an enabler of growth
Mr. Andreessen,
I must applaud the points you've made in your article: "Welcome to the new IT crisis". I agree completely with everything you've said.
I'm tired of working long hours, hand-holding my users with software they should already know, and applying countless service packs, patches, and firmware upgrades. My staff is over-worked and under paid...but we are happy to have jobs in this economy. I loved the article, but it left me with one question....
HOW THE HELL DO WE DO THIS?
-ted
Isn't this the same Marc Andreessen who once responded to a criticism of all of the new bugs in Netscape with ``Hey, we don't have the time to do it right"?
And if you use software that is written to ``do it right", won't uptimes of 5 nines happen by itself?
I'm just one guy who spent a year trying to get Andreesen's software to work on hundreds of computers. I really don't have any opinion about this multi-millionaire.
Geoff
I think I see a trend here. Maybe for them it really would be easier to muzzle the entire internet than to produce p
There's just a few gotchas. These revolve around consistency.
Are your servers consistent? Are your OS's consistent? Are your applications deployed the same way in each of your remote offices? Do your users have the same applications on their computers?
Or is your IT infrastructure a giant ad hoc hairball resulting from IT decisions being made by non-IT personnel? Do you use anything because "that's what everyone else uses, so it must be good enough, therefore opinions to the contrary are wrong"?
Have you implemented things like network management and application/workstation management only to find that the investment is worthless because your organization cannot adhere to a set of standards long enough to make such systems effective, resulting in such an expanse of policy variance that the "management" systems themselves become a net drain on your management resources?
If so, then you've come to realize that the IT problem does not have a technical solution, rendering the entire premise of the article false.
The problem is cultural. IT people are not trusted to make IT decisions, such as meaningful policies with regard to how technology is implemented and what resources are required to deliver a given level of service.
What all of this means is that your Great Answer may not come from the vendor/consultant of the month, but by simply asking your people what is going on, listening to them, and giving them the leeway to make things right. And yes, this was written by a consultant.
I was a systems administrator for about five years. Let me tell you why I think this article is wrong and most other people are wrong about cutting IT costs and increasing IT effectiveness.
The problem is not that IT people are doing it wrong. The problem is that the job of IT people isn't mostly the tedium of manually configuring software the way the vendor recommends. The problem is that the vendor has written software without understanding what the software's users need to accomplish. So, it needs to be glued together. Or, the problem is that the vendor has written software which is just plain broken, and they don't even understand that it's broken, so getting a patch for it is hopeless and the system admin is left to figure out what the devlopers didn't care to think about and come up with a solution on his own. Or, the problem is that nobody in your company even UNDERSTANDS that they need software for certain purposes (like firewalls, backups, etc.). Or they know they need to solve a problem (they want reliable e-mail) but they don't have the time to figure out how to make it happen.
Yes, IT folk spend some of their time applying that one patch to 100 systems, but anyone vaguely competent can automate that in a short period of time. The problem is not applying the patch. The problem is
Basically, most if not all software and hardware sucks. The vendor fundamentally doesn't care enough to build you a product that works, or maybe they don't know how. Usually it works well enough to use, but it just isn't perfect, and you have to learn its quirks to make it do what your company needs. And even if it were perfect, that wouldn't be good enough, because a perfect product, by itself, would just sit on a shelf and not be bought and later installed by someone who has determined that their company needs it.
In short, the IT guy's job is to be an advocate for his company, understanding what they need from their computers and what problems they are having, and then taking what's available and making a working system and helping the users be able to use it to do what the company needs done. NOBODY else is going to do that. Not vendors, and not consultants, and certainly not some company that automates away the part of the sysadmin's job that he does while he's thinking about the more important parts. :-)