Deferred IT Maintenance Is a Ticking Time Bomb

snydeq writes "The underfunding of routine hardware replacement purchases and the degradation of aging enterprise apps pose systemic risk for many IT organizations, thanks to a ballooning 'deferred IT maintenance debt' in the decade since Y2K fears pushed enterprises to invest heavily in essential system upgrades, InfoWorld's Bill Snyder reports. And with sysadmins 'scrambling to keep systems up and running with budgets that barely cover the basics,' this 'IT debt' promises only to increase in the coming years, especially as IT continues to defer routine maintenance in favor of new 'cost-saving' initiatives, particularly around the cloud."

Comments

How is this newsworthy? It's just common sense.

[mschaffer]

Deferring any maintenance can have calamitous effects.

I fail to see why this is newsworthy? Is it just because IT people whine louder?
If you are in the US---just look around. Infrastructure systems are crumbling away because of "deferred maintenance". It's not just IT. It's roads, bridges, state governments, municipalities, houses, businesses---it'severything!

Re:How is this newsworthy? It's just common sense.

[Desler]

InfoWorld needed some adclick revenue so they posted this completely duh story

In other news...

[Anonymous Coward]

Water is wet and the sun rises every day.

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

How is this comment worthy?

How does that work again?

[jeffmeden]

And with sysadmins 'scrambling to keep systems up and running with budgets that barely cover the basics,' this 'IT debt' promises only to increase in the coming years, especially as IT continues to defer routine maintenance in favor of new 'cost-saving' initiatives, particularly around the cloud.

The point of using "the cloud" (a hollow buzzword, I admit) is that you can offload the servers, software, and maintenance to a firm that specializes in such things. Theoretically, taking advantage of the cloud where it fits your organization will offset the "maintenance debt" problem. YMMV, of course.

Re:How does that work again?

[sycodon]

CIOs and organizations blissfully march towards disaster while quietly chanting to themselves, "The Cloud will save us all".

Re:How does that work again?

[Nikker]

Something to do with eggs and baskets comes to mind, just because you rent the Limmo you are still the one responsible for the drivers mistakes, etc, etc.

Re:How does that work again?

[jeffmeden]

Don't blame the cloud... If it weren't around they would simply chant about outsourcing, virtualization, or right-sizing whilst marching to their doom.

Re:How does that work again?

[DragonWriter]

Theoretically, taking advantage of the cloud where it fits your organization will offset the "maintenance debt" problem.

"Cloud" (as in, dynamic server provisioning) has very little to do with it.

Outsourcing IT functions to a firm that is contracted to actually perform the maintenance that was being deferred on the in-house systems (whether hardware, infrastructure software, application software, etc.) obviously can address problems related to deferred maintenance, not because of the outsourcing itself, or because the vendor to whom the operations are outsourced happens to use "cloud" technology to power its offerings, but because the maintenance is actually happening.

OTOH, its not a magic wand to deal with maintenance debt with regard to information systems. You still need to conduct ongoing evaluation and updates of business processes and the supporting applications not merely to meet generic needs but to meet the particular needs of your business. If you are using generic apps provided by a "cloud" vendor, your flexibility to keep them up to date with your processes may be limited (the same is true of locally hosted COTS software, of course.) If you are using custom apps -- or scripted customization of off-the-shelf apps -- hasted by a cloud vendor then, just as with similar local-hosted apps, you have to maintain the software as part of that continuous maintenance of the business operations.

Keeping operations -- whether implemented in hardware, in software, or with organized groups of people -- in tune with the changing needs of the business is a fundamental need of business which is largely technology-independent. Using vendor-provided, cloud-hosted services may be a way to outsource some of the more generic parts of that (e.g., someone else gets paid to, among other things, apply basic OS patches and patches to software shared with other users) and may provide tools that simplify some of the rest (if all your key apps are cloud-hosted web apps, the mechanics of rolling out updates may be trivial), but it doesn't eliminate the basic need or make it so internal staff don't have to do anything to address it.

Re:How does that work again?

[CaptainJeff]

Not a hollow buzzword, I'm afraid. Simply the name given to having a service provider handle (1) infrastructure, (2) an application environment, or (3) applications for you while taking care of all of the maintenance of said provided service. Normally, whatever it is is running in a virtualized environment and you are not provided with insight, nor control, over where it is running physically. Pretty straightforward concept and such an approach does offset some of the maintenance debt...if you can pay someone else to do this work (physical upkeep of VM host servers, monitoring of VM guest servers, patching and associated commodity maintenance of operating systems and associated services (antivirus, etc)) for less than you can do it yourself, then you win. Like it or not, these services are commodities now.

Re:How does that work again?

[turbidostato]

"The point of using "the cloud" (a hollow buzzword, I admit) is that you can offload the servers, software, and maintenance to a firm that specializes in such things."

Yes, because it's a demonstrated hard fact that those companies providing infrastructure for the cloud can't lower their operational costs by neglecting maintenance; of course they wouldn't do that anyway since it's those infrastructure companies' very valuable data what is at risk if maintenance is neglected instead of their customers'.

Oh, wait!

Re:How does that work again?

[DigiShaman]

I'm going to play dumb for a moment. What you just described IS "The Cloud". Or were you just being facetious?

Re:How does that work again?

[eth1]

Outsourcing IT functions to a firm that is contracted to actually perform the maintenance that was being deferred on the in-house systems (whether hardware, infrastructure software, application software, etc.) obviously can address problems related to deferred maintenance, not because of the outsourcing itself, or because the vendor to whom the operations are outsourced happens to use "cloud" technology to power its offerings, but because the maintenance is actually happening.

This still doesn't really work. I work for an IT services company that does just this. Companies that don't want to run their own email servers, firewalls, or whatever will turn them over to us to run. However, it's very rare that they will pay as part of the contract for us to actually keep hardware and software up-to-date. That's usually something that we have to go back to them and say, "we need to get this upgraded," to which they answer, "It's too expensive!"

Fortunately most of the contracts have something in them about at least keeping up to a version currently supported by the vendor, but that still usually means they can get far enough out of date to have to do lots of gymnastics when it finally does get upgraded.

Re:How does that work again?

[jeffmeden]

No one gets my jokes around here.

Re:How does that work again?

[doomicon]

CIOs and organizations blissfully march towards disaster while quietly chanting to themselves, "The Cloud will save us all".

Of course it will, because the same publication that brought us this gem about deferred maintenance costs, also told their readers the It's Economics Stupid.

http://www.cio.com/article/591812/Cloud_Computing_It_s_the_Economics_Stupid

Re:How does that work again?

[DragonWriter]

This still doesn't really work. I work for an IT services company that does just this. Companies that don't want to run their own email servers, firewalls, or whatever will turn them over to us to run. However, it's very rare that they will pay as part of the contract for us to actually keep hardware and software up-to-date.

With "cloud" vendors providing packaged solutions, usually part of what you get is updates (this isn't the same as someone turning over existing servers to an outside firm.)

But, yeah, its possible to outsource without addressing deferred maintenance, too.

Re:How does that work again?

[SanityInAnarchy]

What makes it hollow is that it has several meanings which, while not necessarily mutually-exclusive, are also not necessarily related -- and all of these things already have words. What you've described is, more or less, "outsourcing foo" where foo is 1, 2, or 3.

There was no need to sex this up with "cloud". I was fine with it when it was "Amazon's Elastic Compute Cloud", since then it was just part of a name of one particular offering, but now it's just confusing. Are we using "cloud computing" if we use a traditional pay-by-the-month VPS, but we're hosting a web app we use instead of equivalent desktop software? Or is it only "cloud" if we use pay-by-the-hour pricing for on-demand VPSes, as EC2 offers? Is it "cloudier" if we use an entirely managed environment like, say, Google App Engine? (Or maybe Windows Azure? Haven't looked into it enough to know.)

When someone says "cloud", it's not entirely clear whether they mean an on-demand VPS with a thick client (think game server on Amazon EC2) or an in-house, non-virtualized web application. The fact that these things have nothing whatsoever to do with each other is what makes "cloud" so vacuous.

Like it or not, these services are commodities now.

I don't mind it at all. I like most of the stuff being marketed as "cloud", though I am of course wary of giving up control when I don't see a clear benefit. I use some "cloud" things, and some things I host myself.

What I mind is that when Microsoft says "To the cloud!" I have to stop and think. Do they mean, to my on-demand Linux VPS? No, of course, they mean servers, in the broadest possible sense -- nothing necessarily to do with Web technology (since they'll liberally apply it to native-app integration), nothing necessarily to do with virtual machines (since who knows or cares what they're hosting it on?), and their target audience would've been equally impressed with "To the Internet!"

Re:How is this newsworthy? It's just common sense.

[I8TheWorm]

I think it's a setup for the "IT Industry Invaded by Incompetent Idiots" and "CIOs Found Replacing Working Systems with Crap Made By Their Hunting Buddies" articles.

A large portion of /. readers are in IT and already knew this. However, seeing it in "print" in a newsrag you might find in a CIOs office is a little noteworthy. It means it's only a matter of time before someone comes rushing to your desk to say "Our CIO just read an article about infrastructure and we need an ans..."

Hang on, someone's at my desk.

It's a governance issue - plan and simple

[GPLDAN]

Too many CIOs of too many western corporations report to the CFO, not the CEO. There are WAY too many CIOs who come into organizations with an eye, or a reputation, for cost cutting instead of tech innovation. Pick up any copy of CIO magazine and look at the toadies who make the top CIOs in the nation, and ask yourself - what innovation did they bring to make that list? What business process did they improve with tech? Only a handful make the cut. Most are there because they are good at pinching out costs, kicking out the older IT workers and either outsourcing or bringing in college grads.

I routinely see job ads for experienced Java developers, people with hard core experience in integration, esp. with telephony or security technologies, need 5-10 good years, offering $70k tops. Good luck with that, but again it is the CIOs who get the jobs telling people they can staff cheaper, run leaner, cut the corners - that get the job because it is the CFO who is doing the hiring and the performance reviews.

The big corporation IT C-level execs are a fear driven lot, there are no Gates or Zuckerburgs in their midsts. The action is being with the cloud providers, or the web service providers themselves. Enterprise IT is really a shit place to be outside China. It's a world full of EDS consultants and chickenshit CIOs who won't think how a business could use IT to expand. And the social media space is going to tear a bunch of them new assholes, because none of them know how to leverage it. The startups do.

Re:It's a governance issue - plan and simple

[Anonymous Coward]

There's a lot of truth to what you say, but from a business perspective, there are two basic strategies: low cost and differentiation. For a lot of firms, IT is not their differentiator, so that means IT is driven by a cost strategy. I agree that more firms should view IT as a differentiator and a source of sustainable competitive advantage, but they dont all have to. For those firms, going low cost and outsourcing are the right decisions because IT is not their core competence.

Re:It's a governance issue - plan and simple

[Anonymous Coward]

Too bad I don't have mod points.

This deserves +5; Insightful.

Re:It's a governance issue - plan and simple

[mlts]

Nail, head hit. I have worked for people who had bosses who had zero interest in anything other than cost. The same people that I lambast for putting basic security precautions as an extreme low priority, due to their attitude of "security has no ROI."

What is ironic is that even though this makes the quarterly figures look good, it is killing competitiveness long term. Another example is R&D. Not product research, but true R&D where people discover something, shrug, put it on a shelf for 20 years, then it becomes immensely marketable (think Corning and Gorilla Glass as the prime example of this.) Instead any groundbreaking research ends up being done overseas or by small firms which are bought out, or have their IP infringed upon. If research is done, it is for a product to cough up this quarter or perhaps this FI, and usually it is how to add a gewgaw to something existing and palm that off on the market.

You mention China. Chinese companies know how to lock their doors down. They know what happens if you run your company with your fly open, and most of the companies over there wouldn't even be in business had it not been for "borrowed" IP from the West. Take Foxconn as an example. For a company that size making so many Apple products (including ramping up production on unannounced items), they are quite airtight about what their factories produce even with the hordes of workers they have. Had this been a company run by the typical PHB here in the US (with their usual lack of interest in security), everyone and their brother would know what the iPhone 5 looks like, and perhaps even have the source code for that rendition of iOS.

I wish the US would start borrowing from China in this regard. Even with security aside, just because a company's IT infrastructure works today doesn't mean it will work 5 years, or perhaps even six months from now without major issues. Taking a charge off quarterly earnings to fix problems now means a lot of less wasted money when the upgrades have to be done post-haste.

Re:It's a governance issue - plan and simple

[hedwards]

That's not entirely the CFO's fault, a lot of that has to do with the US tax code. If the capital gains rate for investors didn't start to kick in until 2 years down the road and took another year to fully kick in, you'd see longer holding periods. Also, if the short term tax rate was higher for individuals holding for less than a month.

As it is there's very little reason for corporate executives to think down the road by more than a couple quarters, as chances are the people who own shares now won't still own any much beyond that.

Re:It's a governance issue - plan and simple

[hedwards]

It's not until they loser your records like TD Ameritrade did mine, or are unable to find a file vital for handling an account.

Re:It's a governance issue - plan and simple

[c0lo]

Too many CIOs of too many western corporations report to the CFO, not the CEO. There are WAY too many CIOs who come into organizations with an eye, or a reputation, for cost cutting instead of tech innovation.

Assuming the corporation is not in the IT business but only uses IT, I thing that's only fair to use risk management as the second term of the contrast (instead of tech innovation).

Re:It's a governance issue - plan and simple

[Lifyre]

It's amazing how right this is. The problem also stems from companies that could or should be using IT as a way to improve their core competencies and improve their competitive position aren't because of the recent economic issues. Many people are getting power and influence by riding the penny pinching wave instead of making good long term decisions. We're going to be facing the aftermath of having these people over promoted for a long time to come.

Re:It's a governance issue - plan and simple

[Anonymous Coward]

And the $70k Java developer gig is in Fargo.

Re:It's a governance issue - plan and simple

[gtall]

"I wish the US would start borrowing from China in this regard." I wish you were right. However, if an American firm did this, they'd be sued by Chinese companies in American courts using American lawyers...and their Imaginary Property would be protected and the American firm set up for serious damages. And the Chinese know this.

Re:It's a governance issue - plan and simple

[Anonymous Coward]

You've obviously never been to China let alone Shanghai. Most if not half the small and medium business run pirated versions of Windows and Office on 2nd hand PCs. Often cobbled go together with spare parts. Most of the time, they're already hacked and DON"T CARE! As long as MS Word opens and they can browse the internet, ef it.

That is China. It's not what you think it is or would like it to be.

Re:It's a governance issue - plan and simple

[tommeke100]

have you seen the pictures of the FoxConn factory in China?
They have housing buildings for their workers, where they can sleep 8 ppl per room
They also have "suicide nets" around the building so ppl wouldn't jump off
You want that in the US?

Re:It's a governance issue - plan and simple

[mlts]

My point got lost here. The point is that Chinese companies actually seem to devote resources to computer security and IT infrastructure because they are seeing the mistakes companies are making on the other side of the Pacific. Even the Chinese government is laying fiber like there is no tomorrow.

Here in the US, we don't need corporate dorms -- we escaped that stuff in the Gilded age (but with the political climate, we may be heading back towards that direction.) Instead, more than just a token effort is needed in a lot of businesses when it comes to IT infrastructure and security.

Re:It's a governance issue - plan and simple

[IICV]

The same people that I lambast for putting basic security precautions as an extreme low priority, due to their attitude of "security has no ROI."

To be fair, on the normal USA business timescale (that is, the next two or three quarters), it really doesn't. Security only matters if you're still heading the company when something bad happens, which to most of these people is pretty much unthinkable - because they know they'll be gone in a few years.

Re:It's a governance issue - plan and simple

[Anonymous Coward]

Nail, head hit. I have worked for people who had bosses who had zero interest in anything other than cost. The same people that I lambast for putting basic security precautions as an extreme low priority, due to their attitude of "security has no ROI."

What is ironic is that even though this makes the quarterly figures look good, it is killing competitiveness long term. Another example is R&D. Not product research, but true R&D where people discover something, shrug, put it on a shelf for 20 years, then it becomes immensely marketable (think Corning and Gorilla Glass as the prime example of this.) Instead any groundbreaking research ends up being done overseas or by small firms which are bought out, or have their IP infringed upon. If research is done, it is for a product to cough up this quarter or perhaps this FI, and usually it is how to add a gewgaw to something existing and palm that off on the market.

You mention China. Chinese companies know how to lock their doors down. They know what happens if you run your company with your fly open, and most of the companies over there wouldn't even be in business had it not been for "borrowed" IP from the West.

Maybe the tech companies, but certainly not all Chinese companies. They regularly have knock-offs that are exact duplicates of the original merchandise because some factory leaks the designs.

Re:It's a governance issue - plan and simple

[CAIMLAS]

Companies which have been improving their services and products through better IT practices are actually not doing too poorly. And no, I'm not wholely talking about IT industry companies, either: I know of manufacturing, architectural, and a handful of general 'service' companies (including banks) which have significantly improved their IT utility, both internally and as they face the customers.

No, XP isn't off the desktops at many of these places. (They are at the architectural place.) But many of the machinations have changed because there are people who realize "hey, it's a tight market; we need to leverage our existing assets - the people and their knowledge and connections. How can we do this?"

Re:It's a governance issue - plan and simple

[jimicus]

Had this been a company run by the typical PHB here in the US (with their usual lack of interest in security), everyone and their brother would know what the iPhone 5 looks like, and perhaps even have the source code for that rendition of iOS.

I wish the US would start borrowing from China in this regard.

Give them a few years, they will. It'll start to happen when the first PHB demands evidence that a chinese factory has had no more than three information leaks in the previous 2 years, and they'll get a very bemused email back asking them what sort of information leaks would they like engineered because that's the only way anything's getting out.

Re:It's a governance issue - plan and simple

[Anonymous Coward]

As usual, it's a balance thing - and at the moment, there's little to be personally lost by a CIO by letting an IT infrastructure
crumble. But vast personal gains to be made by cutting costs, regardless of means.

I disagree that working in a neglected environment is a bad place to be though. By the time the rot has set in, the likes
of EDS will have left the building. Become known for your innovative save-the-day antics and it's a job for life. Legacy
support has worked for me for most of my career !!

- c.

Re:It's a governance issue - plan and simple

[Archangel Michael]

Not only is the nail on the head, it is worse. The CIO reports to the CFO but doesn't actually know any of the terms that the CFO uses, so all they have are "saved X by not doing Y". A REAL CIO would have the CFO terms down and be able to say "by doing T we can do S things more productively".

I remember when this hit me about a dozen or so years ago, when I was the business manager for a small regional ISP (when they still existed), and had to process some logs for billing purposes. I had one computer that I used, bought when we started the ISP, and its processing time had slowly increased to 3 to 4 hours to process the logs the way I wanted. I bought a new computer and the process no longer took the 3-4 hours, but ran in less than 15 minutes.

The way it hit me was I was considering not running the processing because of the time it took my computer to process the logs, leaving that computer unavailable for other usage. The result is instead of NOT doing something, because it took so long, I was able to use a new computer to do it more efficiently. Cost savings on the books (-$1500), productivity gains 3-4 hours week. Those hours do NOT show up on the balance sheet anywhere.

Computers allow us to do more in less time, and time is money, except to bean counters. All of the technology cost savings I've ever seen end up cutting productivity, not just in IT staff, but across the board. Until CIOs can speak to the CFO in terms of productivity and opportunity costs associated with other non-financial costs. Yes, while it is "cheaper" on the books to run computers to 4 and 5 years of use, the cost is often more than just hardware. People need to do more with the computers they have, and tech time fixing equipment failing because it is beyond support life is a cost, it just doesn't show up on the books in obvious ways.

Re:It's a governance issue - plan and simple

[marcosdumay]

So, throw all the computers out of the window, and go back to paper and pencils. With no more colstly IT I bet any company can dominate the low cost markets.

Re:It's a governance issue - plan and simple

[Lifyre]

Exactly my point. The ones that ARE leveraging it are seeing the benefit and until CIOs everywhere are the same path there will be continuted long term issues.

While I personally dislike XP at this point it is still a viable corporate OS. There are benefits to moving up to Windows Vista/7 I'm sure but not significant enough yet to justify costs for every company.

Re:It's a governance issue - plan and simple

[Rich0]

I dunno. Even if investors REALLY did invest for the long term, the way that CEOs are compensated would have to change quite a bit. What is in the shareholders' financial interest is rarely exactly the same as what is in the CEO's interest.

Re:How is this newsworthy? It's just common sense.

[fuzzyfuzzyfungus]

I suspect two reasons: 1)(and most important): This is being published by Infoworld, ergo it focuses on IT stuff. 2) Much of the worst rot in IT is largely invisible to the layman.

Slow computers with styles that were pretty neato back in 2000 are obvious to the poor office drones who have to endure them; but anything that new can, largely, be forklift upgraded for the cost of the new systems and some grunt labor. Turning a 3 year desktop refresh cycle into a 5 year(or 7 year, *cough* *cough*) desktop refresh cycle doesn't make anybody happy(particularly once warranties run out, the scavenging and improvising begins); but is architecturally a small problem. You don't really accrue much "debt" over time. The cost will be "1 forklift upgrade to present day PCs" whether that upgrade takes you one generation ahead or three.

It's the complex software, the highly specialized proprietary industrial controller cards, and suchlike widgetry where there is real hell to pay, and most of that is invisible...

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

"I fail to see why this is newsworthy? Is it just because IT people whine louder?"

Exactly. He'd better check the bridges he's traveling over or under every day.

Re:How is this newsworthy? It's just common sense.

[Toe, The]

Because much of this IT is stuff that affects individuals who have no influence over it.

When a company puts off investing in security, for example, and when they also collect and store my credit card info / medical info / personal demographics / shopping history / etc., they are putting me at risk.

I have to trust that their IT department is on the ball. Something I am beginning to think is never a good idea. But it's impossible to not give companies some info on me and still be a normal modern human, and thus I am forced to trust them all the time.

So if they're further neglecting their IT, it means my data is more vulnerable. Not that's there's a damn thing I can do about it.

Maybe it's ok..

[iONiUM]

Maybe the cloud will mean less in-house IT stuff, which means the IT debt won't even need to be paid off.

Now if Amazon or Microsoft is putting off the IT work for their cloud systems, that might.. be a problem..

Re:Maybe it's ok..

[sjames]

One way or another, it's going to be paid. Either buy and deploy internally or pay an outsourcer to buy and deploy. Meanwhile, you can't just export re-writing your old creeky VB6 apps to the cloud. Even if you find a cloud vendor offering a suitable replacement app, you'll have to pay for that and pay to have your vital business data pried from the cold dead fingers of your old app and imported to the new app.

It may come out a bit cheaper having the cloud provider do it, but be sure of the business arrangements. In the modern world, they'll leave you high and dry in a cold millisecond if it becomes profitable for them to so so. Read the fine print carefully!

on the other hand

[blair1q]

The price of replacing things may be getting cheaper faster than the implied cost of the risk of not replacing it is going up. It may be cheaper to wait until it breaks than to buy something with a rapidly depreciating value. The extra cost of dealing with an emergency may be paid for by the lower cost of the gear and labor. And for those risk events that never happen, the ratio of preventative replacement cost to emergency replacement cost will be infinite.

Only your CIO knows for sure. But I bet he's planning it this way.

Re:on the other hand

[tqk]

The price of replacing things may be getting cheaper faster than the implied cost of the risk of not replacing it is going up. It may be cheaper to wait until it breaks than to buy something with a rapidly depreciating value.

This implies that the box isn't doing anything of any value. In the real world, they are, and it's insane to risk the corp. on guessing whether it'll continue to stay up.

That CIO is one in name only, and should be shot.

Re:on the other hand

[marcosdumay]

"The extra cost of dealing with an emergency may be paid for by the lower cost of the gear and labor."

That is quite unlikely... And, by the way, the biggest problem out there is with deferred maintence on software. That causes low, but repetitive costs most of the time, until it gets so bad that it causes hight, and repetitive costs.

Re:on the other hand

[blair1q]

Or it may be doing something distributed that can be handled by the others since there's overhead designed into the system.

Google is an extreme example of this. They don't replac.e blades. They add capacity so fast that when one blade goes bad they just leave it dead.

Re:on the other hand

[tqk]

This implies that the box isn't doing anything of any value. In the real world, they are, and it's insane to risk the corp. on guessing whether it'll continue to stay up.

Or it may be doing something distributed that can be handled by the others since there's overhead designed into the system. Google is an extreme example of this.

I wasn't thinking of ops that run high availability failover servers. I was thinking of the ones who're still running ancient Solaris boxes running "critical" FTP servers. Yeah, they do tend to run for damned near forever, but betting "critical" infra on that is short-sighted at best. If it's really critical, it's going to cost big-time when it does eventually fail. That's what that nominal CIO is betting s/he can get away with until s/he moves on.

*yawn*

[lennier1]

In other news: The sky is blue!

Re:*yawn*

[c0lo]

In other news: The sky is $500 billions-worth of blue!

FTFY

I read it like this:

[TheRedDuke]

"The underfunding of routine exercise programs and the degradation of aging overweight sysadmins poses systemic risk for many IT organizations, thanks to a ballooning 'deferred weight loss program' in the decade since Y2K fears pushed enterprises to invest heavily in dudes who live in their parents' basements", InfoWorld's Bill Snyder reports. And with sysadmins 'scrambling to keep their bodies up and running with foods that barely cover the nutiritional basics',' this 'IT chub' promises only to increase in the coming years, especially as IT continues to defer routine workouts in favor of new 'cost-saving' initiatives, particularly around the refrigerator."

I didn't have my glasses on, though.

Re:I read it like this:

[Anonymous Coward]

I didn't have my glasses on, though.

They're probably stuck under one of your chins, again.

Re:I read it like this:

[Bing Tsher E]

The important message, though, is: "The IT Guys know that they should be getting paid more money for doing more new stuff."

Many people don't understand it.

[khasim]

Yes, it is "common sense" to you.

But how many "IT managers" understand that part of their job involves pruning? Killing old systems. Deprecating other systems. Weeding out the "one-offs" that pop up when no one is looking.

The last place I worked has 7 different database servers. Because they were running 7 different database platforms.

Virtualization means that you can reduce the floor space needed. But management also needs to look at reducing the systems needed.

The longer you leave it, the cheaper it gets

[petes_PoV]

With hardware prices dropping every year, the longer you can defer hardware upgrades, the less money it will cost you. Given that basic piece of information, it's hardly a surprise that companies don't upgrade until they absolutely have to (anyway: why would they until there's a need?). If they can give their kit a mid-life kicker with some more memory or swapping in some faster CPUs, isn't that better than spending 10s of thousands or more on a new box. Better, that is for everyone except the hardware manufacturers who will counter the drop in sales volume by lowering prices even more.

Re:The longer you leave it, the cheaper it gets

[dangitman]

If they can give their kit a mid-life kicker with some more memory or swapping in some faster CPUs,

That's just stupid. Simply "dropping in" a faster CPU doesn't do all that much, because you keep the same bus speeds on the motherboard that limit bandwidth. As for memory, haven't you noticed that buying older, slower memory tends to be more expensive than buying the new faster memory, because the old stuff is no longer in production?

Aside from those factors, there's the added labor in installing those components and testing them. On top of all that, most mass-market desktops that are used in this space aren't particularly upgrade-friendly when it comes to internal components.

Re:The longer you leave it, the cheaper it gets

[petes_PoV]

Those who have no experience and say it's "stupid" should look and learn from those who are actually doing it and getting their employers good value for money in proven, real-life situations. It *does* work. It's a quick, reliable and scalable solution in many, many cases. Obviously you need a background in performance analysis so that you've correctly identified and quantified the bottleneck - and no, merely playing with a "point and shoot" GUI "tune your system" toy program that you downloaded for free off some website doesn't qualify you.

Re:The longer you leave it, the cheaper it gets

[PreparationH67]

That is only one side of the problem. Workstations can probably go one for a while with some basic hardware updates after a few year (like more RAM). Issues like that are the ones that a visible to the users and the higher ups mostly because more people will complain when their 4 year old PC is running slow. The real problem is with the infrastructure, as in the background stuff only the IT guys can really see, understand, and complain about. A lot of the comments on here hit the nail on the head. Why spend money of Enterprise level hardware when you can get consumer grade for cheaper? Why rewrite this old outdated program when you IT guys can figure something out to extend the life? Why do you IT guys want to get paid so much? What do you mean professional software developers are expensive? I can get some kid to write it for far less. It does all boil down to short term thought overtaking long term planing, and that problem isn't just an IT one.

I feel the pain...in state Gov...

[jzarling]

I am having trouble getting basic hardware replaced - I can't get a 500-750 dollars to replace some network switches let alone enough scratch to update my primary DC. Our Budget Analyst does not see the need to plan for future needs, or periodic replacement of vital equipment as warranty cycles expire. I have documented our needs, but my boss the CIO is afraid to push the issue.

Re:I feel the pain...in state Gov...

[Script Cat]

Because the operations are planed to get transfered to China soon.

Re:I feel the pain...in state Gov...

[Herkum01]

I see replacing equipment as less of issue than purchasing new equipment for additional capacity. Assuming you are doing backups, you can always move hard drives from old-system to a new-system. Yeah that is some work to get it back in running, but it is still a possibility.

However, if you have servers running at capacity, and they refuse to purchase new equipment, you are asking for problems.

Re:I feel the pain...in state Gov...

[c0lo]

I am having trouble getting basic hardware replaced - I can't get a 500-750 dollars to replace some network switches let alone enough scratch to update my primary DC. Our Budget Analyst does not see the need to plan for future needs, or periodic replacement of vital equipment as warranty cycles expire. I have documented our needs, but my boss the CIO is afraid to push the issue.

My advice: add some risk analysis argumentation. You know? Something on the line of:
1. probability of equipment failure over time - use the "cumulative hazard function" not the "probability distribution function".
2. impact the server crash will have on the business (make sure you slip-in some "lost face" apropos - after all it would be the manager's face to be lost). If you can express the impact in $$$ and plot the "risk x impact", chances are the budget analyst will "get the picture" easier.

Re:I feel the pain...in state Gov...

[turbidostato]

"Assuming you are doing backups"

And that's exactly the problem, my friend. Too many people round here seem to imply that "deferred it maintenance" means not replacing servers when the guarantee period ends up. But maintenance means having two sysadmins when you formerly had three or maintaining the three sysadmins when capacity has grown 50%.
Lowering maintenance costs means that it has been a year that nobody has the time for a test restore so nobody has noticed that the backups are failing since six months ago because a minor glitch in the tape reader. Lowering maintenance means that your sysadmins have no time to "play" with new agile or devops concepts and tools that would allow for safer and more effective practices and that their knowledge is rusting with time so you are more dependant on external consultors that will squeeze money out your nose.

Re:I feel the pain...in state Gov...

[Anonymous Coward]

And make sure to send it by email. Print the email to pdf, save it.
Eventually the failure will occur, the manager who refused to approve the expenditure will try to shift blame - its what they do best - often with an email to their boss.
Then you can reply with the original email adding - oh but we did warn about this.
You can do this quite innocently and frequently (not always) without repercussion.

Re:I feel the pain...in state Gov...

[CAIMLAS]

Forget the risk analysis arguments. If people refuse to see simple logic, hitting them with more of it will do no good. These people will dismiss your reasoning with superstition, thinking along the lines of, "My VCR is 10 years old. It works fine. I see no reason to upgrade our network equipment until my vcr dies."

For things like that, the only reasonable solution is a loose mains wire and some bridging rod to the offending equipment in question. Then, when you get your new equipment overnighted at 20% markup, you're able to perform a Miracle (looking useful in their eyes, as opposed to someone who plans ahead and documents things, who's scuffed off like a scab) by restoring the data from backup.

Re:I feel the pain...in state Gov...

[Anonymous Coward]

Does your employer know your mindset?

Water still wet

[uvsc_wolverine]

This is just a "status quo" article. There are occasional spending upticks centered around events (like Y2K) but we typically get our marching orders from the C-level people and are expected to just get it done.

I work for a large university (32k students) and we've got roughly 30 people taking care of about 4,000 computers, 10 important web servers (there are a whole bunch more that no one cares about), Active Directory and Novell Netware (we're in the process of dumping Novell), Groupwise, Magic Service Desk, VMWare, network file storage, multiple POS systems, and a whole bunch of backend stuff that makes all of these systems talk to each other for authentication purposes. That 30 people includes all of our support personnel, network admins, AD admin, programmers, DBA guys, and our email admin. We're also moving from Netware to AD, and from Groupwise to Exchange. If you look at just our desktop support personnel we've got 13 full-time technicians to do desktop level support for 2,000 employees and 32,000 students. We're all looking at this as an opportunity to get good experience to put on the resume and then jump ship for decent money.

Part of that need for maintenance is a need to have good people to do that maintenance. We finally got the school to cough up funding for IT personnel training (we were paying for our own training/certifications), now we just want to get paid more than the high school dropouts working for facilities.

How about weeding out enterprise standards?

[hsthompson69]

Just one minor gripe with the parent - a lot of times, what should be weeded out isn't the "one-offs" (which are often times built way under budget with way more capacity and way less maintenance cost), but the actual official enterprise standard that got put in because some CIO was buddies with some sales rep. "One-offs" are a signal that the current standards (either of technology, or product development), are having problems. While not all "one-offs" may be worthy of keeping, when going through the weeds, don't assume the enterprise standard is perfect, and don't assume the one-offs don't have something to teach you.

Examples of enterprise standards that should be weeded out where I work -> Lotus Notes, StarTeam, Windows XP.

Re:How about weeding out enterprise standards?

[hawguy]

I think the one-off's he's talking about are things like the purchase tracking system that Finance installed on a workstation under someone's desk. IT was not aware of it being purchased and of course it has no way to export data into any of our other systems so we can integrate it with the rest of our financial systems.

Eventually we sucked it into our VMWare infrastructure to get it off of the desktop machine (no RAID disks, and the backup consisted of someone in accounting copying the database to a flash drive "periodically", which was anywhere from weekly to bi-monthly).

We're going to spend a bundle of money to get the data out of this system to import into our ERP system we're working on - had Finance come to us for this system in the first place, we would have pointed them to the add-on module for the accounting system they were already using. Oh, and it would have cost less than the system they purchased behind IT's back and would have saved them a lot of labor in double entering data. But the CFO swore by this system in his previous company so that's what they used.

Re:How about weeding out enterprise standards?

[hsthompson69]

I agree, some of the one-offs (especially when driven by business monkeys masquerading as poor techs), are going to be problems worthy of getting rid of. But I've noticed a lot of "enterprise standards" are simply declared by fiat (often by more business monkeys masquerading as poor techs), rather than because anyone actually *tried* to deliver anything with the damn thing. Buying into glossy brochures and power point presentations spells trouble again and again.

On the other hand, if you've got some reasonable techs trying to solve big problems with reliable software (say, CVS back in the day, Subversion, or hell, even wordpress and mediawiki), often times they'll be able to do so with less money up front, more reliability, and less money in the long run.

As a leader in IT, it pays to actually look deeply into what is going on at the ground floor, and discovering internal best practices, rather than listening to a sales monkey and then mandating best practices based on zero real world experience. Even better, if you've done your due diligence to find out what the grassroots best practices are, you'll also probably have found some pretty good technical leaders to mentor up into the organization (anyone who manages to left-hand a working, reliable system under the radar in order to make up for deficiencies in an "enterprise standard" is probably *exactly* the kind of guy who has other ideas they can't try out without more backing).

Heaven help us from CFOs installing rogue systems under their desks :)

Re:How about weeding out enterprise standards?

[pasamio]

You think that's bad. A former organisation I worked at had a similar thing happen where the Finance group bought a software package which was sold on "no IT department required" however the way we found out was even before they deployed it. It was of course a computer program so instead of the Finance department using their own budget allocation, the bought it out of the IT departments budget. When the CIO and the CFO had a meeting the shouting could be heard from the other end of the building.

Suffice to say the "no IT department required" bit was trash and at one stage we had a few people working on trying to deploy the system out.

Re:How about weeding out enterprise standards?

[cinderellamanson]

"Suffice to say the "no IT department required" bit was trash and at one stage we had a few people working on trying to deploy the system out."

Some "IT department", can't even use an "easy button" :p

Re:How is this newsworthy? It's just common sense.

[ceCA]

Don't worry be happy.. 12-21-2012 is just around the corner. Who cares? I welcome our new overlords, the 4 horsemen of the apocalypse. We don't manufacture anything. We outsource everything. Damn I think the 4 horsemen are Indian and they all have a Texas accent too !!!

It's such a Ticking Time Bomb that...

[StickyWidget]

... it could seriously reduce 2nd quarter earnings for Cisco, Juniper, Microsoft, Peoplesoft, Oracle, Sun.....

~Sticky

Re:It's such a Ticking Time Bomb that...

[Anonymous Coward]

That would be my take on it.

People aren't buying enough: scare them into shopping!

Most of the panic is seeded by people who sell software and hardware; If nobody's buying they're not making money. We've had more than one group of wise consultants way over spend our money, getting the CFO to sign up for NAS sytems that were completely stuffed with 5 times more storage than we'd need for the next 2 years, but ensuring we'd have to buy a new NAS in order to increase storage. And that wasn't the worst they did - the rest is just too complicated to describe.

degradation of aging enterprise apps?

[jolyonr]

How, exactly, do enterprise apps degrade?

Do they suffer from bit-rot, and have some kind of half-life?

I understand that eventually apps will fail to be supported by the developers, won't potentially work on more modern operating systems, and in some cases require updating in order to be able to work correctly with the rest of the world.

But it's a bit disingenuous to call this "degradation". The app continues to do what it always did. You're just wanting more out of it than you did before. The app didn't change, you did.

Re:degradation of aging enterprise apps?

[Maxo-Texas]

1) The languages, special hardware, libraries and controls become unsupported on new hardware.
2) The languages, special hardware, libraries and controls become unsupported on new versions of the operating system.
3) The operating system becomes unsupported.
4) The hardware becomes unsupported.

Example: VB6 program uses bar code scanners.

2004? VB6 unsupported as a language.

2008, VB6 unsupported for security patches (so any required security patch could kill VB6 program)

2009 bar code scanners unsupported (change to optical recognition with new software interface)

2009 VB6 Outlook/Word integration fails.

2010 Hardware and operating systems to support VB6 start becoming unavailable. All are unsupported by vendors.

Cost to redevelop VB6 program-- about 1.6 million dollars.

At some point- basically find a new packaged product (cost $100k + $500k user licensing & support + loss of ability to differentiate business) which provides 80% of functionality of the VB6 program and toss it. Can't be changed to match your business - you must change business to match it.

Re:degradation of aging enterprise apps?

[Anonymous Coward]

You're obviously new to IT, or your grasp of the English language is loose... there are many ways enterprise apps can degrade, first off there will be bound to be enhancements, bug fixes, patches, etc which adds on potential load to the apps, there can be more users as the business grows which adds on load over time, even if user base doesn't grow, the data will which affects the platform (i.e. disk, network, etc) which affects software performance, etc.

Either you're taking the term "degrade" too literally or you're new to IT..

Re:degradation of aging enterprise apps?

[c0lo]

But it's a bit disingenuous to call this "degradation". The app continues to do what it always did. You're just wanting more out of it than you did before. The app didn't change, you did.

Hmm... While theoretically you are right (the app doesn't change, the environ/data changed), it doesn't make the application run properly over time, the effort/cost to correct the application is still needed.

Examples:
a. application/system that don't function as expected anymore due to a security patch applied on the OS (damn;d if you don't apply the patch, damn'd if you do).
b. applications/system that don't scale anymore with the number of concurent users
c. applications/system with performance strongly affected by the "data bit rot" (lost/dangling references/relationship accumulation).

Re:degradation of aging enterprise apps?

[benjamindees]

Here's the way it works.

If you implement an app that is well-designed and comprehensive, you are married to it for maintenance, your boss will not give you other projects, HR will wonder what they are paying you for, and eventually you will leave due to boredom and wage stagnation.

If you implement an app that is mostly functional but half-assed, you will be promoted and given a new underling to maintain your half-assed app for you and bring you coffee.

So, you see, this way enterprise apps naturally tend towards being just a bunch of Excel macros.

Re:degradation of aging enterprise apps?

[Red Flayer]

At some point- basically find a new packaged product (cost $100k + $500k user licensing & support + loss of ability to differentiate business) which provides 80% of functionality of the VB6 program and toss it. Can't be changed to match your business - you must change business to match it.

Larry Ellison? Is that you? That's been standard Oracle-speak for nearly a decade, now.

I think you may have meant it sarcastically... but I think it's a good idea.

How much benefit do you really derive from your unique business practices? Why not standardize? Besides the customization and integration savings, it's cheaper to train employees (since many applicants will have been using the same workflows at other employers).

Larry might have his flaws, but this is one thing that I agree with him 100% on.

Re:degradation of aging enterprise apps?

[Anonymous Coward]

Huh? What current MS operating systems don't support VB6? Check the core runtime binaries - they're all protected OS files now on newer platforms. I've got VB6 apps (well, my clients' apps) out in the wild running on everything from Win2000 to Server 2008 R2, including being hosted apps on Citrix. I'm not *happy* about maintaining these dinosaurs, but they're working fine, including support for theming (built-in controls only, though, but that almost goes without saying).

MS isn't stupid enough to kill VB6 app compatibility. It won't go away until Win64 kills off Win32.

And what Wintel hardware doesn't support VB6 apps? I don't see how that even makes sense.

As for the optical scanner, MS Outlook, or pretty much any other recent API, the only truly nasty problems I can think of are these two: COM and/or native Win32 API is dropped in favor of only .Net, Java, and so on, much like Crystal Reports after version 11.5 (a.k.a. 11R2); that forces a lot of re-work to use an alternative API (e.g. use less convenient native Win32 instead of COM) or creating a translation layer (e.g. via C++ to get to a Java-only API via JNI). Those are ugly solutions, but it's hard for me to imagine that a half-million+ outlay for an 80% replacement that limits business differentiation would be preferable.

Another common problem is EOLed third-party controls, but anyone who lived through the VBX to OCX transition should know better than to use those unless absolutely necessary or they came with pure VB6 source code and use/deployment rights for updated binaries.

I'd love for my clients to replace their aging VB6 apps, and I remind them the clock is ticking (if only slowly).

[I can't believe I'm about to offer this; I must be a budding masochist.] If you can't find competent VB6 maintainers, that's one of my job descriptions; respond to this post, and we'll work out how to discuss terms away from slashdot. Heh, chances are, I'd charge no more than half the cost of the sub-optimal replacement product you mentioned...

Oh, and $1.6M for a complete rewrite? I know it's the worst metric, but how many KLOC are we talking here?

- T

Re:degradation of aging enterprise apps?

[sjames]

It's not so much that the app rots, but it's support structure and fitness do. Requirements slowly grow until they exceed the app. The OS the app depends on gets a stream of updates until one of them introduces an incompatibility. People who actually know that app move on, demand for knowledge of that app declines and so less bother to learn it. Vendor EOLs the app or one of it's dependencies and support for it. Ancient install media gets damaged or lost.

How quickly the lessons of Y2K are forgotten! Everyone remembers the big yawn (for the most part), but few remember the magabucks spent on hiring senior managers to come back and wallop the COBOL decks again to fix the problems at the last minute (without which, Y2K would have been a lot more exciting). I still periodically see odd date screwups around the new year that appear to be cases where a temporary last minute patch was made in 1999 with a note to fix within 10 years and guess what doesn't happen!

Some of those COBOL enterprise apps are still kicking around and the last person who knew anything at all about them is dead now. They only run at all due to heroic measures including in some cases running an emulator inside an emulator on a partition in an aging mainframe. You REALLY don't want to let things go on that long, it becomes very expensive.

Re:degradation of aging enterprise apps?

[CAIMLAS]

There is a solution for 1 through 4, and it's not one you'd like (unless you have a shotgun):

Virtualization. Virtualization will give those ancient systems another stay on life, and you will be able to "phase them out" (not really) over the following decade, as other products start to get used.

Virtualization turns your old crap systems into ever-living zombies. "This is old crap, we'll just virtualize it on new hardware". Doing this should last the system itself 1-2 hardware refreshes. By the time it's time to replace them, we'll all be digging in trash cans and fighting each other off with sticks...

The above matches most scenarios. For all others, you're making a fairly good argument for in-house open source development. (What would it cost to employ 5 'decent' programmers to maintain a system they have access to the code for? Somewhat less than dropping $600k every 3-4 years on new systems, I imagine - especially if they've got open source collaboration.)

Re:degradation of aging enterprise apps?

[Skal Tura]

Sometimes the differentiating business practices is your competitive edge. Infact, it many times is just that.

I should know - we do loose some business because we cannot adapt the software to suit our needs (closed source portions/original design approach), and are forced to have our business to fit within the software specs. That sucks.

For startups the standard software package might allow cheaper start up, but eventually there will be things which does not suit the business.

Biggest competitive advantages are reached via business practices, look at Toyota and how their heavy emphasis in manufacturing automation keeps them ahead. Hint: Their automation solutions are pretty much everywhere, even their competitors buy heavily from Toyota to get their OUTDATED automation solutions in use. You could say Toyota is the Google of the auto industry. Toyota's automation procedures & know-how is like Google's datacenter procedures & know-how.

Re:degradation of aging enterprise apps?

[Ash-Fox]

Virtualization. Virtualization will give those ancient systems another stay on life, and you will be able to "phase them out" (not really) over the following decade, as other products start to get used.

That wasn't an option for one of the systems I maintained that relied on some very specific ISA hardware. Even tried a ISA to PCI adapter at one point but the software freaked out over it and virtual machine software wasn't really helping either.

Re:degradation of aging enterprise apps?

[Maxo-Texas]

The software degraded (lost key features) when virtualized under winXP. Also required supporting hardware (external, non-PC device) became extremely expensive. Using the new inexpensive hardware would have required a software change.

Re:degradation of aging enterprise apps?

[Maxo-Texas]

The software didn't run correctly when virtualized.

One required security patch disabled a feature of medium importance. There was a risk that any new required security patch might kill the software.

Per microsoft, VB6 was unsupported and they no longer tested it with new patches to see if it was broken or not by the new patches. Any support would require $400,000 ($450k this year, $500k next year).

It wasn't hard for the business to imagine- they had a new directive- use packages (closed or open source okay) over custom software. That allows them to eliminate multiple support positions. Each support position also had to be an expert in both the system rules and VB6 so in several areas, it was coming down to "I sure hope Bob doesn't get hit by a bus or we are screwed".

Re:degradation of aging enterprise apps?

[marcosdumay]

So, life at the closed software world is as usual.

Re:degradation of aging enterprise apps?

[marcosdumay]

That would be true if requirements remained constant. The problem is that they don't, and most business software needs constant development. Constant development has a nice feature:

• At the begining, the system is small and organized, everything is well;
• Then, comes the next iterations, and it gets bigger, but still organized, and things are still well;
• With time, the developping tem will change, and newcomers won't understand the original organization, not to say that some of them will be simply bad developpers. They'll organize things their own way, and the system will become disorganized. Things won't be well by now, but that isn't a disaster yet;
• With time the users of the application also change, ditto for their managers, and the newcomers won't have any kind of idea of what was the porpose of the application. They'll want it to do every kind of smart and dumb things. Worse yet, the developers also won't know the porpose of it, and will implement every kind of smart and dumb things on it. Scope increases fast, and at the same time the code is getting disorganized. It still ins't a disaster;
• Then, a disaster happens. Anything can trigger it, it can appear anywhere at the application, and it may come from the business level, from within the code, from interfaces between code and external tools, or hardware, or realy any place. And nobody is able to solve the problem anymore, because the application is a monster too big and too complex to be mastered by any team.

Re:degradation of aging enterprise apps?

[Anonymous Coward]

The software didn't run correctly when virtualized.

I've yet to run into this. I'm very interested in any details that might help me cover my own ass if you have the time and are allowed to discuss it.

One required security patch disabled a feature of medium importance. There was a risk that any new required security patch might kill the software.

If you're at liberty to say, what did the feature involve? Was it implemented with only core VB6 capabilities in line with well-known MS security best practices (e.g., don't write to Program Files folders), or did it rely on third-party controls/drivers or esoteric Win32 behaviors?

Per microsoft, VB6 was unsupported and they no longer tested it with new patches to see if it was broken or not by the new patches. Any support would require $400,000 ($450k this year, $500k next year).

I recall them asserting support for the core libraries on supported OSes, but I can't find the link. There was a security patch that corrected some core DLL or other within the last year or two, long past when they dropped support for VB6 itself. Gotta love the pricing on those custom MS support contracts...

It wasn't hard for the business to imagine- they had a new directive- use packages (closed or open source okay) over custom software. That allows them to eliminate multiple support positions. Each support position also had to be an expert in both the system rules and VB6 so in several areas, it was coming down to "I sure hope Bob doesn't get hit by a bus or we are screwed".

Given the extra details, it does make good business sense. I'm used to seeing internal apps for which there really is no off-the-shelf replacement, let alone an 80% one, and where the relevant staff would need to know the ugly details regardless of whether the solution was custom or shrink-wrapped. Good luck with all that.

- T

Re:degradation of aging enterprise apps?

[Maxo-Texas]

Well, having to be mildly vague but it involved Office automation from VB. Something that used to happen automatically stopped working and had to be done manually. This was in the non-virtual environment. Basically VB was using an office function to do something automatically.

The virtualization issues centered around something called XDK (I think?) and some out of date 3rd party unsupported closed source controls. Oh and something about synchronization with handhelds. The handheld library wasn't even supported on win7 any more (not even virtually). It wasn't installable.

Re:degradation of aging enterprise apps?

[Anonymous Coward]

Thanks for the responses.

Well, having to be mildly vague but it involved Office automation from VB. Something that used to happen automatically stopped working and had to be done manually. This was in the non-virtual environment. Basically VB was using an office function to do something automatically.

MS doesn't keep the Office automation API as strongly backwards compatible as the Win32 API. Semantics and even interfaces change in unexpected places. I recall having to version-checking code for Word 2000 vs. 2002/XP when automating Word Mail Merge a long time ago. MS had simply changed some crucial automation API function, and without any notice AFAIK.

The virtualization issues centered around something called XDK (I think?) and some out of date 3rd party unsupported closed source controls. Oh and something about synchronization with handhelds. The handheld library wasn't even supported on win7 any more (not even virtually). It wasn't installable.

Bummer for you, but good news for me as those clearly aren't part of core VB6.

Heh, captcha is "slumming" - marvelously apropos for a discussion of VB6 app maintenance in 2011.

- T

Load of crap

[dave562]

Everyone I've been talking to my field is telling me that corporations are spending like crazy on IT in the last two quarters, and are going to continue spending large amounts for at least the next year. There has been some slow down after the economy tanked, but from everything I've seen, the cash flow spigots are opening up.

In my own experience I just got a new job six months ago and it has been non-stop, balls to the walls busy since I walked through the door. We're hiring new people and spending millions on hardware. Of course, we are an IT business. Our SaaS environment is what allows our part of the organization to make money. The spending priorities might be different in other sectors.

Re:How is this newsworthy? It's just common sense.

[KublaiKhan]

Oh, is -that- the way to get your boss to authorize expenditures?

I need to make some friends in the news media.

Re:How is this newsworthy? It's just common sense.

[arivanov]

Depends on the level of "bespoke" in your house.

Scavenging for desktop parts is the "little devil". Scavenging for people who know how the bloody things work more than 3 years after for IT systems is the real nightmare.

If a system has been in the field for 3+ year nobody knows what are its real dependencies and what does it really take to augment, add capacity or do any changes. The people who knew have left, gone to pastures new or have forgotten what the problems used to be and no documentation can help you here (even if there is any suriviving docs on the design of the system in question). This is valid for almost all classes of IT and telecoms systems and is the real cost factor in IT "maintenance debt". If we use a real-life analogy IT maintenance debt is like a discounted mortgage. You pay virtually nothing for 2-3 years and after that the lender skins your hide.

Cost savings is really expensive.

[Script Cat]

Cost savings is the biggest expense to any large organization that does it.

Re:Cost savings is really expensive.

[korgitser]

The english have an old saying: I am not that rich to afford something so cheap.

Re:How is this newsworthy? It's just common sense.

[puto]

I live and work in Latin America. My residence is in Colombia, but for work I am in most LATAM countries, trust me, the US has wonderful infrastructure compared to us. And any latin country i have been in.

Reminds me of a few years ago...

[Anonymous Coward]

When engineering section of the network would go down on Thursday, EVERY WEEK. It's not on any specific time of the day, but it would abruptly interrupt Ethernet connection for everyone in the area for a few seconds. Unfortunately the Windows boxes we hang off Linux samba shares don't react well to that and would cause everyone to lose whatever source files they happen to be editing at the time. This went on for three or four years.

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

I don't know, but you seem to think it is comment worthy or you wouldn't have commented.

Re:How is this newsworthy? It's just common sense.

[I8TheWorm]

Careful what you wish for... the CIO at the Fortune 100 company I just left still thinks DLink routers without redundancy are the way to go. He still approves purchases for replacements and new ones.

You could be having fun with that.

hmm

[buddyglass]

I'm sensing many organizations may defer it until 2014. No special reason.

Re:How is this newsworthy? It's just common sense.

[c0lo]

Deferring any maintenance can have calamitous effects.

I fail to see why this is newsworthy?

Probably the estimated amount of money it would be needed to catch-up with the backlog (the so called "IT debt") and a few words on how the estimation has been made? Also, where these money would be needed? TFA:

$500 billion -- it's a number so big you'd assume it's a component of the national debt. It isn't. Instead, it's what Gartner analyst Andy Kyte calls the IT debt. "

The "debt" really has two major components: One is underfunding and even neglect of routine but important hardware replacement purchases and software upgrades. The other is the slow degradation of enterprise applications.

...

Is that $500 billion number too high? Kyte says he derived it by analyzing several large Gartner clients that generally do a good job of keeping applications up to date. That led him to estimate that a typical Fortune 2000 company would require upgrades costing more than $200 million each.

I admit, these as scraps of "meat", but they are nevertheless still meat.
I imagine that this may be the first step (as an argument) in a push the big IT houses (hardware/software/services) will make to get so more revenue... Something on the line of "well... Apple and Android tablets and phones (that is, the consumer space) are all well and good... But, guys, what are we going to do with the enterprise?".

Re:How is this newsworthy? It's just common sense.

[puto]

I agree wholeheartedly with you. I think another large part of the equation which our fellow IT workers fail to admit is that our ilk are incredibly stubborn about replacing and fixing things. IT workers are notorious for telling management that they can make things work with a hodge podge of coathangers and toenails, not because it is the best solution, but because they can. The problem lies on both sides of the coin. Management not wanting to spend money and IT workers not setting a realistic expectation.

life of a sysadmin

[br00tus]

I have been a UNIX sysadmin for years, and have seen this at many companies. For a long time. Back in 1997 I worked at a company that had a lot of critical applications on an old, old Sun Netra. Lots of organizations have a few of these machines around - they are out of maintenance, the people who built it have left the company, it is out of warranty, and people generally don't touch them and hope they keep running. After years of meetings, presentations, budgets etc. we finally got a new machine shipped - and then a more politically connected department heard the machine had arrived, and had it stolen for their department on the day it arrived. I guess everyone thought it would be cheaper to page me in the middle of the night if the old machine failed and blame me the next day. The replacement machine being pulled is one of the main reasons I quit the company.

I worked for another company that had a lot of money, but one thing we had to deal with was printing. Print jobs would come into our machines from strange places (IBM mainframe machines, from programs that were written 40 years ago) and go out to strange places (old dot matrix printers in a field office out in some obscure city in India). Thus I was sometimes left to puzzle why some program written in PL/I, coming from a mainframe which I don't have access to, is not printing to some ancient printer in Bangalore which is hooked to some ancient PC's parallel port.

My former company from 2009 had some machines like this. Two very old Ultras running StoryServer and who knows what else. The StoryServer license had long fallen out of use, the machine firmware and Solaris OS had not been upgraded or patched for years. It sent e-mail through, for some reason, four Macintoshes. The Macs did not even run MacOS X, they were previous MacOS versions. E-mail starting with the letters A-F went to Mac1, G-M went through Mac2 etc., if a Mac crashed, mail to those letters would stop going through. The developers did not want to spend the time migrating to a new system, and I don't blame them, the oldest long-time developer there who dealt with such arcana was laid off, while the people building the latest new and shiny that the business wanted had the most secure jobs. Aside from this, we did not ever patch or upgrade our Red Hat Dell servers or firmware, we had no scheduled system downtimes etc. Our major Java application server had had its license run out. As I was leaving, the operations boss (soon to be fired) was considering not re-upping our Red Hat licenses.

If a sysadmin goes on a job interview, and is not desperate, these are the types of questions they should ask, at least on the second round of interviews. Are all of the machines, OSs and applications I'll be responsible for under license? Are they all fully patched and upgraded for firmware, OS and application on a regular basis? What is the oldest machine still under responsibility - is it older than three years? Because all servers should be phased out every three years - at the very least. Try getting Dell/HP to support a 7 year old server decently. Also, do you have scheduled downtime once a week? Meaning do you have the option of rebooting and patching your main database machine, even if it is early Sunday morning? If they want 100% uptime it would necessitate paying for the infrastructure for high availability.

Why should they spend the money when they can just call you in the middle of the night, to continue keeping it running with duct tape? Then they can blame you the next day after it broke. And you get no credit for it continually running either - the time you spend keeping it running is not counted, only time you devote to the latest shiny they want to implement. In fact, too much time devoted to keeping the machines they decided not to spend money on keeping up can cost you your job - if there's a choice between laying off the guy maintaining legacy stuff, and the guy who makes the new shiny for the business group and management and who deals with the

Re:life of a sysadmin

[CAIMLAS]

My GOD, man! Get out of my head!

Thank you for your knowledgeable and reflective post. You made my a little warmer inside (maybe it was the whiskey?). It sure has been hell, lately, with archaic

Note: the newest machines I've got are in that "3 year" ballpark. They're all critical infrastructure hosts. I can't get replacements.

Re:life of a sysadmin

[Tweezer]

I hate to say this, but recycling server hardware every three years isn't the right way to go. I have Dell servers that are 7 years old and working fine. They aren't close to using all of the resources so it makes no sence to replace them with anything newer. Resourse intensive apps get server upgrades every 2 or 3 years, but you don't need to upgrade for the sake of upgrading. I don't even bother carrying support on servers beyond the initial 3 years on 90% of my servers, but I do have a third party support some specialized servers that would be a huge pain to change. The trick is you need to know how to rebuild any app you support from scratch or with backups etc. If one of my 7 year old Dells were to fail today, I'd immediately fire up a new box and install everything that's needed on the new server and move on. Really not a big deal. The problem is you have to know how to support what's installed on the servers... every single one of them. If you can't rebuild something from scratch you'll be spending all night learning at some point anyway. Keeping a spare server around is much cheaper than paying for support on a bunch of old servers that rarely have failures other than the occasional HDD.

Re:life of a sysadmin

[Anonymous Coward]

"If a sysadmin goes on a job interview, and is not desperate, these are the types of questions they should ask, at least on the second round of interviews. Are all of the machines, OSs and applications I'll be responsible for under license? Are they all fully patched and upgraded for firmware, OS and application on a regular basis? What is the oldest machine still under responsibility - is it older than three years?"

  Man, those questions are gold. I wish I'd asked them before I took my current job.

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

Well, it is sort of a "duh" story the way it is written, but OTOH the subject is not without merit.

I have been involved with infrastructure assessment of companies prior to acquisition and some stuff is just shocking. Publicly owned companies are driven by return to the shareholders; one way to keep the dividends flowing when the economy is in a downturn or when the business plan isn't working is to reduce operational expense.

Releasing employees is very effective to reduce the spend side but usually that means there is less available effort to work on maintenance. It looks good to have all employee time capitalized on projects but who is keeping stuff working? Also, each person out the door takes expertise with them that is lost to the company. After a while, the company may not even have enough knowledge internally to understand that their boat has holes in it and that patching isn't happening.

This isn't smoke; I've seen it. Data centers with overheating problems and with inadequate standby generators. Power is distributed unwittingly to cause a cascading failure if one breaker trips. Leaking roofs over financial servers (plastic tarp and bucket gave that away). Licensing that has not been kept up to date because no one has a good inventory and no one wants to look-see. So... Oracle enterprise instances running in non-secure network zones and without proper licensing ( potentially million$ in back costs). A database server being used as a network monitoring node and firewall because funds were not available to separate the functions.

Deferred infrastructure investment and maintenance investment happens and it is a ghastly mess to clean up. I am not surprised that more of this is happening.

I love outdated equipment and code

[sirwired]

I work in enterprise tech support, and finding a setup with unsupported hardware or code is the highlight of my day. No problem is easier for me to close than one where the customer has let equipment drop out of warranty and/or software support. Just did one today... box has been in continuous service (without poweroff) for six years; the model was discontinued in '05 and dropped out of support on 12/31/09. They have a difficult problem that likely would have taken me a day or two to solve... instead I made it go away in five minutes of pulling up their logfiles. Now, it's entirely possible that removing the equipment won't fix the issue, but then I have access to better troubleshooting tools, developers that give a *bleep!*, and more comprehensive log files when the put in newer gear.

All this is good for me, but it sucks to be you if you have some really important stuff on that gear...

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

It's not just IT. It's roads, bridges, state governments, municipalities, houses, businesses---

---Detroit....

Connections

[SuperKendall]

Maxo gave a great answer, but there are matters of internal compatibility also. Generally an IT app relies on lots of other IT apps to function, all of them are changing, and if YOUR app does not change you are retarding what can be done in other applications, or forcing maintenance of a backwards compatibility layer.

There are some systems that can really just sit there and don't interface with much, really core systems... but then you run into that systems support issue and there is nothing corporate IT people fear more than unsupported anything, even if it's incredibly easy to care for.

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

D-Link routers are just fine...

For my piss-poor (read: AT&T, and you'll like it or else!) home DSL connection at 384/128.

Re:How is this newsworthy? It's just common sense.

[Desler]

D-Link routers are just fine...

FOR ME TO POOP ON!

FTFY

Have you ever?

[swb]

...worked anywhere where someone actually swapped CPUs in a server from a real vendor (ie, not some BS whitebox)?

I *added* a CPU to an HP server once (single to dual CPUs) and it was super expensive and not all that easy to get the part. From HP it was OMFG-are-we-really-spending-this-kind-of-money expensive, a "re-certified" part from a third party was still way more than the $300 you'd spend for your home system as the part and its corresponding VRM were proprietary.

And the only reason it was done at all was a branch office needed a dual-CPU system for some application or other and management had one of their periodic "spending freezes" (which never seemed to apply to executive office space remodels or furniture...) that kept the "right" solution, a new box, from being purchased. We did buy the parts on individual invoices to avoid capex.

Anyway, nobody upgrades CPUs in major-vendor products. Too expensive relative to the benefit, especially on systems that are old enough to seem slow or considered lacking in power for some kind of repurposing.

Re:Have you ever?

[magarity]

...worked anywhere where someone actually swapped CPUs in a server from a real vendor (ie, not some BS whitebox)?

I *added* a CPU to an HP server once (single to dual CPUs) and it was super expensive and not all that easy to get the part..

I did and while the parts from HP were outrageously expensive we ordered it from a third party. HP does not have an exclusive market on Xeon chips or the plug-in voltage regulators they used to use (maybe still do?). The tricky part was the heatsink which needed to be a funky size to squeeze in the space allotted but that too was solved from a third party vendor. Total cost was about a third what HP wanted. We did it as a test case to make a single CPU machine with 6 total slots into a dual. It was eventually upgraded to all 6 and since HP's prices never changed over time except to go up a little, the third parties dropped prices so we saved even more.

Re:Have you ever?

[petes_PoV]

Yes, I've personally seen it done on two seperate occasions. In both the situation was the same. A software vendor that licenced on a per-processor basis (Oracle). One box was a 12 processor macine, the other was a 16 processor job. In both cases it was far, far cheaper to swap the CPUs for faster ones than to add more of the same speed. In both cases that was my recommendation - based purely on cost savings and in both cases it worked very well.

I'm confused.

[wcrowe]

...'scrambling to keep systems up and running with budgets that barely cover the basics'...

How is this different than it's ever been?

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

The newsworthy part is the developing trend of many companies deferring maintenance and what that means across industries, not the well known consequences of deferring maintenance as a concept.

Re:How is this newsworthy? It's just common sense.

[Lonewolf666]

The software can bite you on the desktop too. A likely scenario:

Three years ago your company introduced some software that was programmed in happy ignorance of good programming practices, especially where access permissions are concerned.

Now you buy new computers. XP is no longer available, so they come with Windows 7. Not really a bad system, but it is a bit more strict where writing to the program directory and such is concerned. Also, it tries to hide this by "virtualizing" the files in question and giving every user his own copy. Very clever if said files are meant to serve to exchange data between users.

Have fun finding out where these strange side effects are coming from ;-)

Re:How is this newsworthy? It's just common sense.

[hawguy]

Sometimes cheap routers are the way to go -- I replaced 2 Cisco routers at a remote site in a seaside warehouse (one due to a power surge when a generator failed, one due to water from a leaky roof) before switching to cheap Netgear routers at about 1/10 the cost. Redundancy? We had a spare configured and ready to go the foreman's truck toolbox and another at his house.

One of the Netgears even survived a similar water deluge to the one that took out the Cisco (but then the Netgear didn't have a fan to suck the water inside).

(before you ask why I didn't put them in a waterproof box, that apparently was not allowed under our lease - no permanent equipment was allowed and apparently a metal box on a shelf was "permanent" but a bare router was not)

Debunking the new adage

[GaryOlson]

Thus

Do more with less

becomes

Doing nothing with nothing

as all systems fail and bankruptcy court takes your now worthless assets.

Re:Low cost strategy

[Lonewolf666]

Low cost needs to be balanced against getting the job done, and reliably done.

Because if your IT starts to have frequent outages or lose valuable data, it can be more expensive than investing in decent equipment and competent employees.
Since /. likes car analogies:
In the 90s Opel, a German branch of General motors, was a bit too aggressive in cutting manufacturing costs. The resulting quality problems were quite damaging to the brand and customers started to look elsewhere for their next car.

yep..

[Chardansearavitriol]

We all saw this one coming. My old school is still using ORIGINAL IMACS for gods sake. This is a high school. one that promotes "Computer Science" despite macs being closer to potpourii than they are to science. It really is bad cause they're like 13 years old. Way to set people up for humiliation once they get out of their colorful plastic bubbles and actually need a computer that does something besides act as a really big ipod.

Re:yep..

[Lifyre]

Just because iMacs made better aquariums than computers doesn't mean all Macs have that problem. A few of them actually have uses and the new MacBooks make really good stands for my iPad.

Printing, the bane of every IT worker

[2bfree]

I think anyone who works in IT long enough comes to think of printing as the biggest waste of money in corporate America. How many forests have ended up as paper jams in a printer because a manager wanted to print his email.

Re:Printing, the bane of every IT worker

[Lifyre]

So true. It gets even more fun when you're in an environment that is unfriendly the many small moving parts of a printer. Such as areas with large amounts of dust or sand in the air. We used to have printer parties . We'd take a bunch of old (mostly broken) printers out back and find various creative and entertaining ways to disassemle them so they would fit in the disposal barrels.

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

Hey hey now. That coat hanger is mission critical. Don't touch it.

We stopped using toenails though. They worked fine but degraded too quickly.

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

Damn right! Since government is evil, we shouldn't have to pay taxes, and our libertarian benefactors will pave the roads according to their self interest!

it's easy to have security slaves killing people

[Anonymous Coward]

it's easy to have security when you work people like slaves and you kill people and make it look like suicide.

Re:How is this newsworthy? It's just common sense.

[hairyfeet]

Oh Please! That is SUCH an easy one to fix! You either run XP Mode in Pro or just load up XP VMs. No you want to talk about "IT debt" try some of the places I walk into, where there is ALWAYS a "mission critical app" that is this horribly mangled piece of badly coded VB+Access mess of no comments anywhere junk, and then they expect YOU to deal with it! Hell one place I walked into in mid 09 had a NT 4 box running a VB3 "app" because each guy they brought in took one look at that beast and said "fuck that!".

Man I can hear the real programmers right now screaming out in pain just at the thought! You want to watch a "real" programmer wet his pants in fear you hand him a huge 14 page VB mess written by a half a dozen guys over the years, NONE of whom ever heard of a comment, with shit all over the place and nothing indented or even calling in a logical order, unless "insane band aid" is considered logic.

You want to know why there is an ever increasing IT debt I'd say that is a BIG part of it. All across the country you have this huge mess of apps written by some Joe Schmo that was bought ages ago and nobody knows how to live without and it DON'T run on anything but what it was written for and even then it is fussy as hell. And that of course don't even take into account the lovely crap like that ISA C&C controller written for DOS 3 that runs a $75,000 piece of machinery made by a company that has been DOA for a decade plus! I have stared into the abyss pal, and not only did it stare back it gave me the finger to boot!

Same holds true for *all* maintenance

[GodfatherofSoul]

In the age of BS corporate leadership, who *doesn't* want to be the guy who cuts costs by 25%, gets promoted up into the suites, then lets one of his successors take the fall when the shit hits the fan? I'm more concerned with our public infrastructure BTW.

Re:Urgent announcement for all of slashdot!!

[cinderellamanson]

"1. The Heretic, who convincingly builds a case that the company is hopeless and run by a bunch of morons;"

that is all

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

> (before you ask why I didn't put them in a waterproof box, that apparently was not allowed under our lease - no permanent equipment was allowed and apparently a metal box on a shelf was "permanent" but a bare router was not)

The rules are obviously bogus. My guess would be that they don't want you setting up anything they'd have a hard time removing. A router they can just unplug, but if you screw a waterproof box to the wall, it might cause more issues.

I'd have gone with a plastic storage box sealed with tape. Crappy, yes, but better than nothing and cheap.

Re:How is this newsworthy? It's just common sense.

[drcheap]

If you are in the US---just look around. Infrastructure systems are crumbling away because of "deferred maintenance". It's not just IT. It's roads, bridges, state governments, municipalities, houses, businesses---it'severything!

Cars, don't forget cars!

(insert obligatory car analogy here)

Re:How is this newsworthy? It's just common sense.

[hawguy]

Sure, the rule may have been bogus, but we had to obey nonetheless. That's one of the joys of dealing with a government bureaucracy where the guy who manages the facility gets to interpret the rules no matter how capricious and arbitrary they may appear. Things were a lot smoother at that facility after we found out that the harbor master liked jelly donuts.

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

Hint: Detroit's problems are not the infrastructure
References: see Africa

We cant even get a patching window

[wesleyjconnor]

Our company has been going for 8 months now and every scheduled maintenance window we have had has been pushed back or cancelled by the higher up's who cant see the need, well its working now isn't it? And/Or cant bear the downtime necessary. Until of course everything falls down around them.

Re:We cant even get a patching window

[niks42]

Our service management team haven't applied a single patch to antivirus in over two years. Firmware on storage silos is pre 2007 level. OS levels and middleware are all out of support, and the application software will probably fall out of support in March unless someone has the balls to sign an extension before the end of the financial year. I won't bore you with the details on ancient client computers, which can't be upgraded to supported levels of software. When I point out the obvious, rising risks and real-world examples of catastrophic failures caused by neglect at all levels, I am told I am guilty of scare-mongery and to STFU, if I want to keep my job. wat do, /.? Well, I tell you what I do. I keep myself employed, run around with the duct tape, make sure I capture every bit of documentation from any old fart who is leaving to go fishing instead, and generally have fun keeping this mangled, messed up thing going as long as I can. Oh, and I constantly advocate open source solutions that could be maintained without incremental licence costs that scare the CIO, and continue to run well on the old, crappy junk hardware around here.

Re:How is this newsworthy? It's just common sense.

[Minwee]

FOR ME TO POOP ON!

FTFY

He already said he was using AT&T. Your fix was redundant.

Re:How is this newsworthy? It's just common sense.

[rubi]

Easy! New stuff is useful to show "upper" management levels that you are "innovating", "aligning IT" and all those pretty buzzwords "CIOs" (or applicable title) like to throw around to justify spending. There isnt much chance to "be seen as a profit center instead of a cost center" in maintenance, upgrades or other such upkeep activities. This is just seen when you have big trouble that can't be patched up and need to upgrade to current hardware or software.

Who cares? I am working on my sauces this year

[Anonymous Coward]

And my putting. I am still finding fish is a bit difficult to grill just right. It tastes great, but getting the grill marks without sticking to the grill, well, I haven't quite got the fung sway of it yet.

Ooops there goes the phone. It's my buddy. I am going to do this thing called FUN now.

Re:How is this newsworthy? It's just common sense.

[scottv67]

like that ISA C&C controller written for DOS 3 that runs a $75,000 piece of machinery

I'm pretty sure you meant to say "CNC"
http://en.wikipedia.org/wiki/Cnc

The chickens will eventually come home to roost.

[Anonymous Coward]

Look, the higher ups don't give a rat's you know what about you or what you think. All they care about is that things continue to run smoothly. Or smooth enough until they can load the two magic silver bullets - outsourcing and the cloud - and eliminate your position. They fear you because you are technologically smarter than they are. In their fantasy land, they think they can move to the cloud and it will be free like Twitter and Facebook.
At first, it will be cheaper, or it will seem to be. And then they'll try to pull the crap on the outsourcer that they pulled on you, like - we want 24x7, or no you can't take it down for patching. The vendor will smile and say, of course, we are glad to accommodate, btw here is your extra-charge invoice. Ha ha ha ha. By then, most of us will have moved on (probably to decent countries with health care and retirement). They think the overseas guys will save them. Ha ha ha ha. My only regret is that I can be a fly on the wall in their corner office to watch it all play out.
Stop trying to be logical at work. If you want to get filthy rich, go be a consultant, charge exorbitant fees while delivering meaningless thick reports that state the obvious. You'll be much happier.

Re:How is this newsworthy? It's just common sense.

[nabsltd]

Things were a lot smoother at that facility after we found out that the harbor master liked jelly donuts.

I have to wonder why this wasn't your first strategy.

Or, am I the only one where the phrase "harbor master" conjures up an image of someone like Homer Simpson or Louie DePalma?

Re:How is this newsworthy? It's just common sense.

[hawguy]

I have to wonder why this wasn't your first strategy.

New York management. It wasn't until they hired a local guy to manage the facility that he was able to get things done.

Simple

[shentino]

Geeks are handicapped in social areas, they have trouble advocating for bigger budgets, they get shafted by their bosses on funding, something goes wrong, they get blamed for it.

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

Oh, you've noticed we can no longer build bridges (and when we try to fix them it takes 4x as long as it took to build them), keep highways maintained, keep our electric grid running, our sewers working, or even our streets plowed (and not just NYC). Face it folks, we're living on what our parents and grandparents bequeethed us - we have become a 3rd world society

Re:How is this newsworthy? It's just common sense.

[Z34107]

You want to watch a "real" programmer wet his pants in fear you hand him a huge 14 page VB mess written by a half a dozen guys over the years ...

Do you want to see real bedwetting? If you're most anywhere in America, your healthcare depends on a few gigabytes of VB6. That it works speaks to the value of good development practices.

Re:How is this newsworthy? It's just common sense.

[aztracker1]

Try a pelican case, a drill and some rubber grommets.

Re:How is this newsworthy? It's just common sense.

[aztracker1]

IE6 based intranet apps for the win!

Re:How is this newsworthy? It's just common sense.

[tqk]

AC (sorry):

Oh, you've noticed we can no longer build bridges (and when we try to fix them it takes 4x as long as it took to build them)

Meanwhile, Cairo, Rome, and London's sewers and garbage systems continue to work. Those cities are a thousand years old! They continue to work because geeks TRY to find a way, despite the challenges and their Princes' inclinations. Think William Penn. "We need to start over, and it's going to be expensive, but ya gotta suck it up, 'cause it needs to be done!"

Cf. "Man up" and related (silly) platitudes. Tell the Boss what needs to be done, and damn the consequences! Then it's his problem to explain. I know, the shareholders are too greedy and The Board just wants to retain control and keep their jobs and ludicrous bonuses ...

Just saying, some of us manage to make a difference, but you're just giving up? Too easy. I hope you're not Irish, with drinking water pipes bursting in the streets from poor maintenance.

Re:How is this newsworthy? It's just common sense.

[RubberDogBone]

Hey now, don't know Y2K vintage hardware. Place I worked spent a lot on some beige and white Dell workstations that year. They were spiffy at the time. The boss/owner was ultra proud of his Dells and made sure they sat on TOP of the desks so clients and whatnot would see them and be impressed. Woooh he spent money on Dell.

Boring beige Dell was actually an improvement. Prior to this, the place was run off an old infected Vision PC they got from a radio ad. I upgraded them with almost no budget to DIY systems I built in the back room. They didn't want to spend money on parts for that. I had to share one CD Rom drive among all of them. But they worked well.

They went to Dell later when they had more employees than PCs. My DIYs were too much trouble, I was told.

Left there in 2002 and the Dells were still front and center. Looking already a little dated because by then Dell had gone to a black motif.

The place closed down in 2007, same Dells front and center and horribly obsolete. Why still there? Well the boss/owner had spent all that money and he wasn't about to let anyone forget it, much less spend a dime more on new PCs.

This is a place that had the entire office tied into one 24-port hub that wasn't even a switch. Files crawled around with terrible packet collisions because the owner didn't want to buy a switch or anything that could do vlans. It was nuts.

incorrect on Magnitogorsk

[Anonymous Coward]

the point was not tractors but tanks (almost the same production line). No one cared much about the peasants starving, and indeed they starved, by millions on a couple of occasions (see "golodomor"). But since they were a "regressive" class, that was OK to the Marxists running the show. Also, with or without tractors, Russia never managed to reproduce its agricultural outputs prior to 1913 when Russian grain prices drove European markets.

In other words, it did not make sense for efficiency, only for control.

Slashdot Writers Don't Give Me Hope...

[BoRegardless]

that a lot of businesses we work with every day control their own "fortunes".

It seems like the companies that fall into this "duct tape" IT maintenance mentality are just buying time until a stroke occurs and the company essentially dies.

And companies want me to believe that I should "trust" them with my data, my orders and my trust.

Sheesh.

Re:How is this newsworthy? It's just common sense.

[CAIMLAS]

I'm guessing it has something to do with the fact that IT managers read InfoWorld. Some read slashdot, too. Many of them are dropping the ball on this, big time. "We don't have the money this quarter" comes up time and time again, and they don't realize that the decreased maintenance is going

The only people who really realize this kind of thing are sysadmins. Unless the sysadmins have a huge amount of power within the organization (including their own ability to prioritize the budget and 'vision' of the organization, as it relates to IT), this kind of thing doesn't even get looked at. People think of IT infrastructure as a cost to be avoided, not realizing that it's analogous to their phone, power, and data services which keep their company alive.

Re:How is this newsworthy? It's just common sense.

[CAIMLAS]

I can't say I particularly care for DLink, but for the cost of a (used) Cisco, I can pick up two Procurves with better specs and a management interface which anyone with basic networking comprehension can master in a couple hours. I'm not hearing anyone say the Procurves are shit (because they aren't). With d-link, I'm sure the same is true - and I can have a stack of hot spares for the same cost, too - Just In Case.

Re:How is this newsworthy? It's just common sense.

[CAIMLAS]

Silly young grasshoppah. There is no "just do..." in IT. The mythical solution you're referring to is a cruel joke told by vendors.

huge 14 page

So, a little script, then?

Something on my table right now: 15k (in-file - and probably significantly more on the printed page) lines of PHP3 with nasty embedded SQL up the wazoo. It ties into half a dozen (literally, 6) other 'mission critical' applications and is customer facing as well as providing internal network management functionality. And this is small fry compared to some of the crap out there.

You want to know why there is an ever increasing IT debt I'd say that is a BIG part of it. All across the country you have this huge mess of apps written by some Joe Schmo that was bought ages ago and nobody knows how to live without and it DON'T run on anything but what it was written for and even then it is fussy as hell

I couldn't agree more. We've had entirely too many Boy Geniuses in decision making places in IT who think they've got something special and unique which will have Totally Awesome Results. They don't bother to think through their decisions.

The proper approach to something like this isn't to fix it. It's to replace it outright with something that does 90% of the task, better, with 50% fewer inter-dependencies by modularizing things as much as possible. Re-implementing, feature for feature, is quite often quicker. Just make sure you don't make the mistake of so many before you and re-implement it - poorly. If you can't do it, find someone who can.

IMO, the key to a successfully maintainable software infrastructure is to KISS and leave things as White Box as possible. When you can't keep things generic, you keep things isolated and modular. When you need something custom purpose that your users rely upon, you make damn sure it's standards based and that there are alternatives available.

(I don't even want to THINK about where we will be with things like Sharepoint in 3-5 years. Likely, another lengthy, drawn out, and costly migration project. This time, maybe back to something like, oh, NFSv5.)

Re:How is this newsworthy? It's just common sense.

[hairyfeet]

Ohhhh...sounds like the abyss has given you the finger as well. And to be fair we are talking about 14 pages of badly written VB4 with about half of the pages nothing but fricking GOTOs bouncing all over the place.Now I'll admit when I've had to do something quick and dirty I've thrown in the occasional GOTO but the GOTOs all went to the same place whereas this massive pile of shite had more twists and turns than a bad detective novel!

But since you have seen the "true horror" of IT I'm sure you can see why I sometimes want to bash my head on the desk when dealing with FOSSies. They always seem to think "All you have to do is replace Windows and Office, and all will be hearts and flowers!" when it is NEVER Windows and Office that is the problem. Hell most companies can't even upgrade to the latest windows version for fear of that giant reeking mess of garbage code they've come to depend on will come falling down like a house of cards in a hurricane.

And while I agree 110% that the goal should ALWAYS be KISS, the problem is the PHBs at these places will never ever in a million years shell out what it costs to actually get all the data out and build a REAL solution, not until they have the crap they are depending on fall apart like some giant train wreck from hell. Like that VB4 app I mentioned early on, I ended up just jamming the thing in a Win2K VM and letting it rot for the next guy. Not because that is what I wanted to do, but because you couldn't pay a college kid what they wanted to pay to have it rewritten, much less get a REAL programmer of any skill.

And that to me is the problem in a nutshell and why the shit isn't gonna get any better for the foreseeable future. It is because businesses in the USA can't pull their heads out of the stock page to realize that working long term solutions cost money no different that decent roads or schools, and look how well THOSE are faring here in the US. IT has it even worse because as long as the thing works that day they don't care if it is a single power surge away from taking down the whole company. To them ANY expense related to IT is just a "waste" unless it is something like an iShiny for themselves or the CEO. That is why I ended up getting out of dealing with corporate, because I frankly got tired of people with impossible problems that wanted to pay pennies to fix years worth of neglect. I mean with the corporate attitude in the USA, is it any wonder nobody young is going into IT anymore? You'd have to be nuts!

Re:How is this newsworthy? It's just common sense.

[TheLink]

The local guy has the edge on figuring out what donuts the local people like? :)

Re:How is this newsworthy? It's just common sense.

[dbIII]

It's not just bespoke unfortunately. If, for example,the vendor decided some time after 2003 to lay off their programmers, support staff etc and get a huge suite of software rewritten in India in java but have not finished yet then you just have to have the in-house expertise that the vendor can no longer provide. It sucks having to edit install scripts of expensive commercial software just to get it onto the machines and then alter undocumented scripts to get it to do plot outputs or do remote tape access. An entire day of debugging very expensive software just to add a single LTO5 tape drive to the backup pool.

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

buy a small pelican case and make appropriate waterproofed holes in it, it will float. we do this since 2007 and it does work. the case is a bit expensive though. an added benefit is that people can even sit on it and it won't break.

Re:How is this newsworthy? It's just common sense.

[Confusador]

Touche, but to be fair that doesn't mean we shouldn't still inspect our bridges.

Re:How is this newsworthy? It's just common sense.

[jellomizer]

Now how is pushing services to a cloud exsaserbating the problem? Sure going to the cloud has it pros and conns. But if you are retiring your old servers to a cloud in terms you are Upgrading your infrastructure and removing your old stuff.

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

Do you want to see real bedwetting? If you're most anywhere in America, your healthcare depends on a few gigabytes of VB6. That it works speaks to the value of good development practices.

I doubt it.

My health care depends on an IBM mainframe app written in the 1970s. I know this because I work for an insurance company, but not on the mainframes :)

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

I also have been told to look after a "bet your business" application that runs hardware via a 25 MHz 386 PC with a 16 bit ISA card under Windows3.1

The upgrade package is more than the company can ever afford. But we can't afford to lose this application either.

If the VP Research had been paying for upgrades over the past 20 years as he was continually reminded then we wouldn't be in this mess.

Re:How is this newsworthy? It's just common sense.

[AmiMoJo]

Some admins are just scared of things going wrong during maintenance do don't do it. My boss at the last place I worked was like that. He never did updates on servers if he could possibly avoid it. The fear that the server might not reboot and he would have to drive all the way out and spend part of his weekend fixing it stopped a lot of preventative maintenance getting done. It was easier to blame users when the RDP server was hit with a 2 year old virus using a vulnerability that was long since patched. Ditto capacity upgrades.

In some ways frequent virus attacks are useful for an IT admin. They can easily be blamed on users and anti-virus vendors, take time to fix but don't cost much (at least not for the IT department) and justify the admin's existence. Similarly the best way to get an expensive new backup system is to delete some of the boss' email and blame it on a failed HDD. That's the problem with all maintenance though: it just looks like a bottomless pit into which you poor money, at least until the day you fall in yourself and land on a nice soft pile of bank notes (unless you put pennies in, in which case you are fucked.)

Re:How is this newsworthy? It's just common sense.

[Z00L00K]

Looks familiar.

And the servers are often placed in locations sensitive to flooding too. Office near a major river where the computer hall is actually below the water surface. Only takes a drainage failure or riverbed overflow to take out the center.

But it's cheap... As long as the servers aren't submerged.

Re:How is this newsworthy? It's just common sense.

[I8TheWorm]

Want a punchline? That company recently bought another in a different town. The head of the NSS group imaged their machines over the network during peak hours through that DLink crap. When the new images didn't work, they were unable to revert them to their old images because, you guessed it, the packetstorm created during the imaging caused the files to become corrupt.

The accounting group at the new company was down for a week.

TO be fair, I use a DLink router at home, and it's been fine. Except for the limit on MAC addresses it will store for network filtering... dadgummit.

Re:How is this newsworthy? It's just common sense.

[I8TheWorm]

That's fantastic when you 1) set up redundant switches/routers, and 2) hadn't decided to remove support at your remote locations and try to handle it all from one distant office.

Re:How is this newsworthy? It's just common sense.

[esimone928]

Common sense to anyone that been working on systems for over a few years. It's not common sense to 90% of most business executives. What amazes me are the comments surrounding Windows &VB. Those applications are just "the peak of the iceberg" . While I have seen older client/server applications in dire need of modernization (VB, PowerBuilder, Smalltalk, C++, etc) the REAL problem exists on mainframe & midrange systems where 20+ year legacy (COBOL, Adabas/Natural, RPG, PL/I, Assembler etc.) applications are still running. This is the real problem...the systems running our governments (local, state, federal) and large corporations (banking, insurance, healthcare). The Y2K problem was a nit compared to this. To complicate matters, most organizations have no idea how these applications are architected. The the people who even have an inkling are gone...most of them retired. Who's going to maintain/modernize/replace these systems? Conclusion, this is very a conservative estimate on the problem and therefore newsworthy.

Re:How is this newsworthy? It's just common sense.

[DarthVain]

Ya, well I'm the Joe Schmo that gets tasked to fix this sort of stuff.

The conversation usually goes like this:

Me: "To fix this properly you will need to do this, buy this software, redesign this system, it will take X amount of dollars and Y amount of time."
Boss: "We don't have the money. You need to fix it as best as possible using the materials at hand with no budget, and a limited time."

Thus, slaps together whatever I have to get it working, cheaply and likely crudely. Forget documentation, you don't have time for that, besides that's like evidence that I was involved, which means the next time this stupid system breaks (and oh by god it will) you will be called in to help try and fix it, again with the restrictions like above.

Believe me, I know how to do it right, I will even offer up my opinion that proceeding like this is a "very bad idea", but when a manager makes the decision, you do it, and sadly hope it is some other "Joe Schmo" that gets stuck with it next time around... :)

Re:How is this newsworthy? It's just common sense.

[doomicon]

"IT workers are notorious for telling management that they can make things work with a hodge podge of coathangers and toenails, not because it is the best solution, but because they can"

Take it another step further, the disasterous effects of those hodge podge implementations are typically not seen immediately, and may take a few years to eventually explode.

So those with the knowledge and experience for disagreeing with the hodge podge, are seen as, dissenters and "Not part of the team". While those workers and managers who supported the hairbrained idea move up the ladder for implementing a solutions so quickly.

Then someone else comes in and inherits the mess, those with experience that disagreed in the first place, have moved on either let go or no chance of upward mobility (don't promte those who aren't part of the "team"), and end up working somewhere else, and eventually it comes to a head and all comes crashing down. Your bought, out of business, etc.

Years in the industry has taught me, there are alot of unqualified and inexperienced people providing solutions, and even less qualified and less experienced leadership making decisions.

IT Industry, where the less you know the higher up the ladder you go.

Why is the cloud better?

[mu51c10rd]

Why is the assumption around the cloud such that people assume companies running cloud services are in much better shape than any other IT company? The datacenters and systems running the cloud services could also be suffering under the threat of this "maintenance debt".

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

>

Something on my table right now: 15k (in-file - and probably significantly more on the printed page) lines of PHP3 with nasty embedded SQL up the wazoo. It ties into half a dozen (literally, 6) other 'mission critical' applications and is customer facing as well as providing internal network management functionality. And this is small fry compared to some of the crap out there.

You just described my workplace. Our stuff is mostly sensible, but we inherited a piece of crap just like this. I've spent three years gradually getting it more maintainable, but it is frightening to think how much more it has costed us to maintain this thing rather than do it right the first time.

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

hand him a huge 14 page VB mess written by a half a dozen guys over the years, NONE of whom ever heard of a comment, with shit all over the place and nothing indented or even calling in a logical order, unless "insane band aid" is considered logic.

I was handed a VB5 disaster that was split up into about 40 modules, semi-random indentation, relied on obsolete proprietary objects (not useful ones, just to make it look "pretty"), and all the function names, the "descriptive" variable names, and the few comments they actually bothered to write in the code were in French. I'm still bitter.

Re:How is this newsworthy? It's just common sense.

[hairyfeet]

Hi MR AC! Word of advice? Buy a shitload of old boxes and have them ready, because machines THAT old WILL DIE HARD. That customer I had with the $75,000 CNC controller that requires DOS 3! (Yeah no shit, fricking DOS 3. Not 4, not 5, won't run on anything but DOS 3 and bare metal) I sold two boxes upfront and got another half a dozen, all with images of the "OS" and software loaded up which he picked up the tab for.

That way when the box this "mission critical" app is running on bites the farm you can just fire up one of the boxes in storage and switch it out. Boxes that old can be had for cheap but are getting harder to find, so do so now. Use the local paper or Craigslist if you have trouble finding any. I would also see if I could find any more of those ISA cards and buy them up. Sadly I looked for nearly a year and couldn't find any, so if that card goes tits up they will be SOL if my engineer buddy can't make them some sort of new interface. But like your sitch this little company could never afford to replace the machine, which is a computer controlled lathe that makes custom columns. Not exactly the kind of thing one finds on eBay cheap.