Deferred IT Maintenance Is a Ticking Time Bomb

snydeq writes "The underfunding of routine hardware replacement purchases and the degradation of aging enterprise apps pose systemic risk for many IT organizations, thanks to a ballooning 'deferred IT maintenance debt' in the decade since Y2K fears pushed enterprises to invest heavily in essential system upgrades, InfoWorld's Bill Snyder reports. And with sysadmins 'scrambling to keep systems up and running with budgets that barely cover the basics,' this 'IT debt' promises only to increase in the coming years, especially as IT continues to defer routine maintenance in favor of new 'cost-saving' initiatives, particularly around the cloud."

Comments

How is this newsworthy? It's just common sense.

[mschaffer]

Deferring any maintenance can have calamitous effects.

I fail to see why this is newsworthy? Is it just because IT people whine louder?
If you are in the US---just look around. Infrastructure systems are crumbling away because of "deferred maintenance". It's not just IT. It's roads, bridges, state governments, municipalities, houses, businesses---it'severything!

Re:How is this newsworthy? It's just common sense.

[I8TheWorm]

I think it's a setup for the "IT Industry Invaded by Incompetent Idiots" and "CIOs Found Replacing Working Systems with Crap Made By Their Hunting Buddies" articles.

A large portion of /. readers are in IT and already knew this. However, seeing it in "print" in a newsrag you might find in a CIOs office is a little noteworthy. It means it's only a matter of time before someone comes rushing to your desk to say "Our CIO just read an article about infrastructure and we need an ans..."

Hang on, someone's at my desk.

Re:How does that work again?

[sycodon]

CIOs and organizations blissfully march towards disaster while quietly chanting to themselves, "The Cloud will save us all".

It's a governance issue - plan and simple

[GPLDAN]

Too many CIOs of too many western corporations report to the CFO, not the CEO. There are WAY too many CIOs who come into organizations with an eye, or a reputation, for cost cutting instead of tech innovation. Pick up any copy of CIO magazine and look at the toadies who make the top CIOs in the nation, and ask yourself - what innovation did they bring to make that list? What business process did they improve with tech? Only a handful make the cut. Most are there because they are good at pinching out costs, kicking out the older IT workers and either outsourcing or bringing in college grads.

I routinely see job ads for experienced Java developers, people with hard core experience in integration, esp. with telephony or security technologies, need 5-10 good years, offering $70k tops. Good luck with that, but again it is the CIOs who get the jobs telling people they can staff cheaper, run leaner, cut the corners - that get the job because it is the CFO who is doing the hiring and the performance reviews.

The big corporation IT C-level execs are a fear driven lot, there are no Gates or Zuckerburgs in their midsts. The action is being with the cloud providers, or the web service providers themselves. Enterprise IT is really a shit place to be outside China. It's a world full of EDS consultants and chickenshit CIOs who won't think how a business could use IT to expand. And the social media space is going to tear a bunch of them new assholes, because none of them know how to leverage it. The startups do.

Re:It's a governance issue - plan and simple

[mlts]

Nail, head hit. I have worked for people who had bosses who had zero interest in anything other than cost. The same people that I lambast for putting basic security precautions as an extreme low priority, due to their attitude of "security has no ROI."

What is ironic is that even though this makes the quarterly figures look good, it is killing competitiveness long term. Another example is R&D. Not product research, but true R&D where people discover something, shrug, put it on a shelf for 20 years, then it becomes immensely marketable (think Corning and Gorilla Glass as the prime example of this.) Instead any groundbreaking research ends up being done overseas or by small firms which are bought out, or have their IP infringed upon. If research is done, it is for a product to cough up this quarter or perhaps this FI, and usually it is how to add a gewgaw to something existing and palm that off on the market.

You mention China. Chinese companies know how to lock their doors down. They know what happens if you run your company with your fly open, and most of the companies over there wouldn't even be in business had it not been for "borrowed" IP from the West. Take Foxconn as an example. For a company that size making so many Apple products (including ramping up production on unannounced items), they are quite airtight about what their factories produce even with the hordes of workers they have. Had this been a company run by the typical PHB here in the US (with their usual lack of interest in security), everyone and their brother would know what the iPhone 5 looks like, and perhaps even have the source code for that rendition of iOS.

I wish the US would start borrowing from China in this regard. Even with security aside, just because a company's IT infrastructure works today doesn't mean it will work 5 years, or perhaps even six months from now without major issues. Taking a charge off quarterly earnings to fix problems now means a lot of less wasted money when the upgrades have to be done post-haste.

Re:It's a governance issue - plan and simple

[hedwards]

That's not entirely the CFO's fault, a lot of that has to do with the US tax code. If the capital gains rate for investors didn't start to kick in until 2 years down the road and took another year to fully kick in, you'd see longer holding periods. Also, if the short term tax rate was higher for individuals holding for less than a month.

As it is there's very little reason for corporate executives to think down the road by more than a couple quarters, as chances are the people who own shares now won't still own any much beyond that.

Re:It's a governance issue - plan and simple

[hedwards]

It's not until they loser your records like TD Ameritrade did mine, or are unable to find a file vital for handling an account.

Re:It's a governance issue - plan and simple

[Lifyre]

It's amazing how right this is. The problem also stems from companies that could or should be using IT as a way to improve their core competencies and improve their competitive position aren't because of the recent economic issues. Many people are getting power and influence by riding the penny pinching wave instead of making good long term decisions. We're going to be facing the aftermath of having these people over promoted for a long time to come.

Re:It's a governance issue - plan and simple

[Archangel Michael]

Not only is the nail on the head, it is worse. The CIO reports to the CFO but doesn't actually know any of the terms that the CFO uses, so all they have are "saved X by not doing Y". A REAL CIO would have the CFO terms down and be able to say "by doing T we can do S things more productively".

I remember when this hit me about a dozen or so years ago, when I was the business manager for a small regional ISP (when they still existed), and had to process some logs for billing purposes. I had one computer that I used, bought when we started the ISP, and its processing time had slowly increased to 3 to 4 hours to process the logs the way I wanted. I bought a new computer and the process no longer took the 3-4 hours, but ran in less than 15 minutes.

The way it hit me was I was considering not running the processing because of the time it took my computer to process the logs, leaving that computer unavailable for other usage. The result is instead of NOT doing something, because it took so long, I was able to use a new computer to do it more efficiently. Cost savings on the books (-$1500), productivity gains 3-4 hours week. Those hours do NOT show up on the balance sheet anywhere.

Computers allow us to do more in less time, and time is money, except to bean counters. All of the technology cost savings I've ever seen end up cutting productivity, not just in IT staff, but across the board. Until CIOs can speak to the CFO in terms of productivity and opportunity costs associated with other non-financial costs. Yes, while it is "cheaper" on the books to run computers to 4 and 5 years of use, the cost is often more than just hardware. People need to do more with the computers they have, and tech time fixing equipment failing because it is beyond support life is a cost, it just doesn't show up on the books in obvious ways.

Re:How is this newsworthy? It's just common sense.

[fuzzyfuzzyfungus]

I suspect two reasons: 1)(and most important): This is being published by Infoworld, ergo it focuses on IT stuff. 2) Much of the worst rot in IT is largely invisible to the layman.

Slow computers with styles that were pretty neato back in 2000 are obvious to the poor office drones who have to endure them; but anything that new can, largely, be forklift upgraded for the cost of the new systems and some grunt labor. Turning a 3 year desktop refresh cycle into a 5 year(or 7 year, *cough* *cough*) desktop refresh cycle doesn't make anybody happy(particularly once warranties run out, the scavenging and improvising begins); but is architecturally a small problem. You don't really accrue much "debt" over time. The cost will be "1 forklift upgrade to present day PCs" whether that upgrade takes you one generation ahead or three.

It's the complex software, the highly specialized proprietary industrial controller cards, and suchlike widgetry where there is real hell to pay, and most of that is invisible...

Re:How is this newsworthy? It's just common sense.

[Toe, The]

Because much of this IT is stuff that affects individuals who have no influence over it.

When a company puts off investing in security, for example, and when they also collect and store my credit card info / medical info / personal demographics / shopping history / etc., they are putting me at risk.

I have to trust that their IT department is on the ball. Something I am beginning to think is never a good idea. But it's impossible to not give companies some info on me and still be a normal modern human, and thus I am forced to trust them all the time.

So if they're further neglecting their IT, it means my data is more vulnerable. Not that's there's a damn thing I can do about it.

I read it like this:

[TheRedDuke]

"The underfunding of routine exercise programs and the degradation of aging overweight sysadmins poses systemic risk for many IT organizations, thanks to a ballooning 'deferred weight loss program' in the decade since Y2K fears pushed enterprises to invest heavily in dudes who live in their parents' basements", InfoWorld's Bill Snyder reports. And with sysadmins 'scrambling to keep their bodies up and running with foods that barely cover the nutiritional basics',' this 'IT chub' promises only to increase in the coming years, especially as IT continues to defer routine workouts in favor of new 'cost-saving' initiatives, particularly around the refrigerator."

I didn't have my glasses on, though.

Re:How does that work again?

[jeffmeden]

Don't blame the cloud... If it weren't around they would simply chant about outsourcing, virtualization, or right-sizing whilst marching to their doom.

How about weeding out enterprise standards?

[hsthompson69]

Just one minor gripe with the parent - a lot of times, what should be weeded out isn't the "one-offs" (which are often times built way under budget with way more capacity and way less maintenance cost), but the actual official enterprise standard that got put in because some CIO was buddies with some sales rep. "One-offs" are a signal that the current standards (either of technology, or product development), are having problems. While not all "one-offs" may be worthy of keeping, when going through the weeds, don't assume the enterprise standard is perfect, and don't assume the one-offs don't have something to teach you.

Examples of enterprise standards that should be weeded out where I work -> Lotus Notes, StarTeam, Windows XP.

Re:How about weeding out enterprise standards?

[hawguy]

I think the one-off's he's talking about are things like the purchase tracking system that Finance installed on a workstation under someone's desk. IT was not aware of it being purchased and of course it has no way to export data into any of our other systems so we can integrate it with the rest of our financial systems.

Eventually we sucked it into our VMWare infrastructure to get it off of the desktop machine (no RAID disks, and the backup consisted of someone in accounting copying the database to a flash drive "periodically", which was anywhere from weekly to bi-monthly).

We're going to spend a bundle of money to get the data out of this system to import into our ERP system we're working on - had Finance come to us for this system in the first place, we would have pointed them to the add-on module for the accounting system they were already using. Oh, and it would have cost less than the system they purchased behind IT's back and would have saved them a lot of labor in double entering data. But the CFO swore by this system in his previous company so that's what they used.

Re:How does that work again?

[DragonWriter]

Theoretically, taking advantage of the cloud where it fits your organization will offset the "maintenance debt" problem.

"Cloud" (as in, dynamic server provisioning) has very little to do with it.

Outsourcing IT functions to a firm that is contracted to actually perform the maintenance that was being deferred on the in-house systems (whether hardware, infrastructure software, application software, etc.) obviously can address problems related to deferred maintenance, not because of the outsourcing itself, or because the vendor to whom the operations are outsourced happens to use "cloud" technology to power its offerings, but because the maintenance is actually happening.

OTOH, its not a magic wand to deal with maintenance debt with regard to information systems. You still need to conduct ongoing evaluation and updates of business processes and the supporting applications not merely to meet generic needs but to meet the particular needs of your business. If you are using generic apps provided by a "cloud" vendor, your flexibility to keep them up to date with your processes may be limited (the same is true of locally hosted COTS software, of course.) If you are using custom apps -- or scripted customization of off-the-shelf apps -- hasted by a cloud vendor then, just as with similar local-hosted apps, you have to maintain the software as part of that continuous maintenance of the business operations.

Keeping operations -- whether implemented in hardware, in software, or with organized groups of people -- in tune with the changing needs of the business is a fundamental need of business which is largely technology-independent. Using vendor-provided, cloud-hosted services may be a way to outsource some of the more generic parts of that (e.g., someone else gets paid to, among other things, apply basic OS patches and patches to software shared with other users) and may provide tools that simplify some of the rest (if all your key apps are cloud-hosted web apps, the mechanics of rolling out updates may be trivial), but it doesn't eliminate the basic need or make it so internal staff don't have to do anything to address it.

degradation of aging enterprise apps?

[jolyonr]

How, exactly, do enterprise apps degrade?

Do they suffer from bit-rot, and have some kind of half-life?

I understand that eventually apps will fail to be supported by the developers, won't potentially work on more modern operating systems, and in some cases require updating in order to be able to work correctly with the rest of the world.

But it's a bit disingenuous to call this "degradation". The app continues to do what it always did. You're just wanting more out of it than you did before. The app didn't change, you did.

Re:degradation of aging enterprise apps?

[Maxo-Texas]

1) The languages, special hardware, libraries and controls become unsupported on new hardware.
2) The languages, special hardware, libraries and controls become unsupported on new versions of the operating system.
3) The operating system becomes unsupported.
4) The hardware becomes unsupported.

Example: VB6 program uses bar code scanners.

2004? VB6 unsupported as a language.

2008, VB6 unsupported for security patches (so any required security patch could kill VB6 program)

2009 bar code scanners unsupported (change to optical recognition with new software interface)

2009 VB6 Outlook/Word integration fails.

2010 Hardware and operating systems to support VB6 start becoming unavailable. All are unsupported by vendors.

Cost to redevelop VB6 program-- about 1.6 million dollars.

At some point- basically find a new packaged product (cost $100k + $500k user licensing & support + loss of ability to differentiate business) which provides 80% of functionality of the VB6 program and toss it. Can't be changed to match your business - you must change business to match it.

Re:degradation of aging enterprise apps?

[Red Flayer]

At some point- basically find a new packaged product (cost $100k + $500k user licensing & support + loss of ability to differentiate business) which provides 80% of functionality of the VB6 program and toss it. Can't be changed to match your business - you must change business to match it.

Larry Ellison? Is that you? That's been standard Oracle-speak for nearly a decade, now.

I think you may have meant it sarcastically... but I think it's a good idea.

How much benefit do you really derive from your unique business practices? Why not standardize? Besides the customization and integration savings, it's cheaper to train employees (since many applicants will have been using the same workflows at other employers).

Larry might have his flaws, but this is one thing that I agree with him 100% on.

Re:How is this newsworthy? It's just common sense.

[KublaiKhan]

Oh, is -that- the way to get your boss to authorize expenditures?

I need to make some friends in the news media.

Re:How is this newsworthy? It's just common sense.

[arivanov]

Depends on the level of "bespoke" in your house.

Scavenging for desktop parts is the "little devil". Scavenging for people who know how the bloody things work more than 3 years after for IT systems is the real nightmare.

If a system has been in the field for 3+ year nobody knows what are its real dependencies and what does it really take to augment, add capacity or do any changes. The people who knew have left, gone to pastures new or have forgotten what the problems used to be and no documentation can help you here (even if there is any suriviving docs on the design of the system in question). This is valid for almost all classes of IT and telecoms systems and is the real cost factor in IT "maintenance debt". If we use a real-life analogy IT maintenance debt is like a discounted mortgage. You pay virtually nothing for 2-3 years and after that the lender skins your hide.

Cost savings is really expensive.

[Script Cat]

Cost savings is the biggest expense to any large organization that does it.

Re:How is this newsworthy? It's just common sense.

[I8TheWorm]

Careful what you wish for... the CIO at the Fortune 100 company I just left still thinks DLink routers without redundancy are the way to go. He still approves purchases for replacements and new ones.

You could be having fun with that.

Re:How is this newsworthy? It's just common sense.

[puto]

I agree wholeheartedly with you. I think another large part of the equation which our fellow IT workers fail to admit is that our ilk are incredibly stubborn about replacing and fixing things. IT workers are notorious for telling management that they can make things work with a hodge podge of coathangers and toenails, not because it is the best solution, but because they can. The problem lies on both sides of the coin. Management not wanting to spend money and IT workers not setting a realistic expectation.

life of a sysadmin

[br00tus]

I have been a UNIX sysadmin for years, and have seen this at many companies. For a long time. Back in 1997 I worked at a company that had a lot of critical applications on an old, old Sun Netra. Lots of organizations have a few of these machines around - they are out of maintenance, the people who built it have left the company, it is out of warranty, and people generally don't touch them and hope they keep running. After years of meetings, presentations, budgets etc. we finally got a new machine shipped - and then a more politically connected department heard the machine had arrived, and had it stolen for their department on the day it arrived. I guess everyone thought it would be cheaper to page me in the middle of the night if the old machine failed and blame me the next day. The replacement machine being pulled is one of the main reasons I quit the company.

I worked for another company that had a lot of money, but one thing we had to deal with was printing. Print jobs would come into our machines from strange places (IBM mainframe machines, from programs that were written 40 years ago) and go out to strange places (old dot matrix printers in a field office out in some obscure city in India). Thus I was sometimes left to puzzle why some program written in PL/I, coming from a mainframe which I don't have access to, is not printing to some ancient printer in Bangalore which is hooked to some ancient PC's parallel port.

My former company from 2009 had some machines like this. Two very old Ultras running StoryServer and who knows what else. The StoryServer license had long fallen out of use, the machine firmware and Solaris OS had not been upgraded or patched for years. It sent e-mail through, for some reason, four Macintoshes. The Macs did not even run MacOS X, they were previous MacOS versions. E-mail starting with the letters A-F went to Mac1, G-M went through Mac2 etc., if a Mac crashed, mail to those letters would stop going through. The developers did not want to spend the time migrating to a new system, and I don't blame them, the oldest long-time developer there who dealt with such arcana was laid off, while the people building the latest new and shiny that the business wanted had the most secure jobs. Aside from this, we did not ever patch or upgrade our Red Hat Dell servers or firmware, we had no scheduled system downtimes etc. Our major Java application server had had its license run out. As I was leaving, the operations boss (soon to be fired) was considering not re-upping our Red Hat licenses.

If a sysadmin goes on a job interview, and is not desperate, these are the types of questions they should ask, at least on the second round of interviews. Are all of the machines, OSs and applications I'll be responsible for under license? Are they all fully patched and upgraded for firmware, OS and application on a regular basis? What is the oldest machine still under responsibility - is it older than three years? Because all servers should be phased out every three years - at the very least. Try getting Dell/HP to support a 7 year old server decently. Also, do you have scheduled downtime once a week? Meaning do you have the option of rebooting and patching your main database machine, even if it is early Sunday morning? If they want 100% uptime it would necessitate paying for the infrastructure for high availability.

Why should they spend the money when they can just call you in the middle of the night, to continue keeping it running with duct tape? Then they can blame you the next day after it broke. And you get no credit for it continually running either - the time you spend keeping it running is not counted, only time you devote to the latest shiny they want to implement. In fact, too much time devoted to keeping the machines they decided not to spend money on keeping up can cost you your job - if there's a choice between laying off the guy maintaining legacy stuff, and the guy who makes the new shiny for the business group and management and who deals with the

Re:How is this newsworthy? It's just common sense.

[Anonymous Coward]

Well, it is sort of a "duh" story the way it is written, but OTOH the subject is not without merit.

I have been involved with infrastructure assessment of companies prior to acquisition and some stuff is just shocking. Publicly owned companies are driven by return to the shareholders; one way to keep the dividends flowing when the economy is in a downturn or when the business plan isn't working is to reduce operational expense.

Releasing employees is very effective to reduce the spend side but usually that means there is less available effort to work on maintenance. It looks good to have all employee time capitalized on projects but who is keeping stuff working? Also, each person out the door takes expertise with them that is lost to the company. After a while, the company may not even have enough knowledge internally to understand that their boat has holes in it and that patching isn't happening.

This isn't smoke; I've seen it. Data centers with overheating problems and with inadequate standby generators. Power is distributed unwittingly to cause a cascading failure if one breaker trips. Leaking roofs over financial servers (plastic tarp and bucket gave that away). Licensing that has not been kept up to date because no one has a good inventory and no one wants to look-see. So... Oracle enterprise instances running in non-secure network zones and without proper licensing ( potentially million$ in back costs). A database server being used as a network monitoring node and firewall because funds were not available to separate the functions.

Deferred infrastructure investment and maintenance investment happens and it is a ghastly mess to clean up. I am not surprised that more of this is happening.

Re:I feel the pain...in state Gov...

[c0lo]

I am having trouble getting basic hardware replaced - I can't get a 500-750 dollars to replace some network switches let alone enough scratch to update my primary DC. Our Budget Analyst does not see the need to plan for future needs, or periodic replacement of vital equipment as warranty cycles expire. I have documented our needs, but my boss the CIO is afraid to push the issue.

My advice: add some risk analysis argumentation. You know? Something on the line of:
1. probability of equipment failure over time - use the "cumulative hazard function" not the "probability distribution function".
2. impact the server crash will have on the business (make sure you slip-in some "lost face" apropos - after all it would be the manager's face to be lost). If you can express the impact in $$$ and plot the "risk x impact", chances are the budget analyst will "get the picture" easier.

Re:How is this newsworthy? It's just common sense.

[hawguy]

Sometimes cheap routers are the way to go -- I replaced 2 Cisco routers at a remote site in a seaside warehouse (one due to a power surge when a generator failed, one due to water from a leaky roof) before switching to cheap Netgear routers at about 1/10 the cost. Redundancy? We had a spare configured and ready to go the foreman's truck toolbox and another at his house.

One of the Netgears even survived a similar water deluge to the one that took out the Cisco (but then the Netgear didn't have a fan to suck the water inside).

(before you ask why I didn't put them in a waterproof box, that apparently was not allowed under our lease - no permanent equipment was allowed and apparently a metal box on a shelf was "permanent" but a bare router was not)

Re:Low cost strategy

[Lonewolf666]

Low cost needs to be balanced against getting the job done, and reliably done.

Because if your IT starts to have frequent outages or lose valuable data, it can be more expensive than investing in decent equipment and competent employees.
Since /. likes car analogies:
In the 90s Opel, a German branch of General motors, was a bit too aggressive in cutting manufacturing costs. The resulting quality problems were quite damaging to the brand and customers started to look elsewhere for their next car.

Printing, the bane of every IT worker

[2bfree]

I think anyone who works in IT long enough comes to think of printing as the biggest waste of money in corporate America. How many forests have ended up as paper jams in a printer because a manager wanted to print his email.

Re:How does that work again?

[turbidostato]

"The point of using "the cloud" (a hollow buzzword, I admit) is that you can offload the servers, software, and maintenance to a firm that specializes in such things."

Yes, because it's a demonstrated hard fact that those companies providing infrastructure for the cloud can't lower their operational costs by neglecting maintenance; of course they wouldn't do that anyway since it's those infrastructure companies' very valuable data what is at risk if maintenance is neglected instead of their customers'.

Oh, wait!

Re:I feel the pain...in state Gov...

[turbidostato]

"Assuming you are doing backups"

And that's exactly the problem, my friend. Too many people round here seem to imply that "deferred it maintenance" means not replacing servers when the guarantee period ends up. But maintenance means having two sysadmins when you formerly had three or maintaining the three sysadmins when capacity has grown 50%.
Lowering maintenance costs means that it has been a year that nobody has the time for a test restore so nobody has noticed that the backups are failing since six months ago because a minor glitch in the tape reader. Lowering maintenance means that your sysadmins have no time to "play" with new agile or devops concepts and tools that would allow for safer and more effective practices and that their knowledge is rusting with time so you are more dependant on external consultors that will squeeze money out your nose.

Re:How is this newsworthy? It's just common sense.

[hairyfeet]

Oh Please! That is SUCH an easy one to fix! You either run XP Mode in Pro or just load up XP VMs. No you want to talk about "IT debt" try some of the places I walk into, where there is ALWAYS a "mission critical app" that is this horribly mangled piece of badly coded VB+Access mess of no comments anywhere junk, and then they expect YOU to deal with it! Hell one place I walked into in mid 09 had a NT 4 box running a VB3 "app" because each guy they brought in took one look at that beast and said "fuck that!".

Man I can hear the real programmers right now screaming out in pain just at the thought! You want to watch a "real" programmer wet his pants in fear you hand him a huge 14 page VB mess written by a half a dozen guys over the years, NONE of whom ever heard of a comment, with shit all over the place and nothing indented or even calling in a logical order, unless "insane band aid" is considered logic.

You want to know why there is an ever increasing IT debt I'd say that is a BIG part of it. All across the country you have this huge mess of apps written by some Joe Schmo that was bought ages ago and nobody knows how to live without and it DON'T run on anything but what it was written for and even then it is fussy as hell. And that of course don't even take into account the lovely crap like that ISA C&C controller written for DOS 3 that runs a $75,000 piece of machinery made by a company that has been DOA for a decade plus! I have stared into the abyss pal, and not only did it stare back it gave me the finger to boot!

Same holds true for *all* maintenance

[GodfatherofSoul]

In the age of BS corporate leadership, who *doesn't* want to be the guy who cuts costs by 25%, gets promoted up into the suites, then lets one of his successors take the fall when the shit hits the fan? I'm more concerned with our public infrastructure BTW.

Re:How is this newsworthy? It's just common sense.

[Z34107]

You want to watch a "real" programmer wet his pants in fear you hand him a huge 14 page VB mess written by a half a dozen guys over the years ...

Do you want to see real bedwetting? If you're most anywhere in America, your healthcare depends on a few gigabytes of VB6. That it works speaks to the value of good development practices.

Re:How is this newsworthy? It's just common sense.

[RubberDogBone]

Hey now, don't know Y2K vintage hardware. Place I worked spent a lot on some beige and white Dell workstations that year. They were spiffy at the time. The boss/owner was ultra proud of his Dells and made sure they sat on TOP of the desks so clients and whatnot would see them and be impressed. Woooh he spent money on Dell.

Boring beige Dell was actually an improvement. Prior to this, the place was run off an old infected Vision PC they got from a radio ad. I upgraded them with almost no budget to DIY systems I built in the back room. They didn't want to spend money on parts for that. I had to share one CD Rom drive among all of them. But they worked well.

They went to Dell later when they had more employees than PCs. My DIYs were too much trouble, I was told.

Left there in 2002 and the Dells were still front and center. Looking already a little dated because by then Dell had gone to a black motif.

The place closed down in 2007, same Dells front and center and horribly obsolete. Why still there? Well the boss/owner had spent all that money and he wasn't about to let anyone forget it, much less spend a dime more on new PCs.

This is a place that had the entire office tied into one 24-port hub that wasn't even a switch. Files crawled around with terrible packet collisions because the owner didn't want to buy a switch or anything that could do vlans. It was nuts.

Re:How is this newsworthy? It's just common sense.

[CAIMLAS]

Silly young grasshoppah. There is no "just do..." in IT. The mythical solution you're referring to is a cruel joke told by vendors.

huge 14 page

So, a little script, then?

Something on my table right now: 15k (in-file - and probably significantly more on the printed page) lines of PHP3 with nasty embedded SQL up the wazoo. It ties into half a dozen (literally, 6) other 'mission critical' applications and is customer facing as well as providing internal network management functionality. And this is small fry compared to some of the crap out there.

You want to know why there is an ever increasing IT debt I'd say that is a BIG part of it. All across the country you have this huge mess of apps written by some Joe Schmo that was bought ages ago and nobody knows how to live without and it DON'T run on anything but what it was written for and even then it is fussy as hell

I couldn't agree more. We've had entirely too many Boy Geniuses in decision making places in IT who think they've got something special and unique which will have Totally Awesome Results. They don't bother to think through their decisions.

The proper approach to something like this isn't to fix it. It's to replace it outright with something that does 90% of the task, better, with 50% fewer inter-dependencies by modularizing things as much as possible. Re-implementing, feature for feature, is quite often quicker. Just make sure you don't make the mistake of so many before you and re-implement it - poorly. If you can't do it, find someone who can.

IMO, the key to a successfully maintainable software infrastructure is to KISS and leave things as White Box as possible. When you can't keep things generic, you keep things isolated and modular. When you need something custom purpose that your users rely upon, you make damn sure it's standards based and that there are alternatives available.

(I don't even want to THINK about where we will be with things like Sharepoint in 3-5 years. Likely, another lengthy, drawn out, and costly migration project. This time, maybe back to something like, oh, NFSv5.)

Re:How is this newsworthy? It's just common sense.

[hairyfeet]

Ohhhh...sounds like the abyss has given you the finger as well. And to be fair we are talking about 14 pages of badly written VB4 with about half of the pages nothing but fricking GOTOs bouncing all over the place.Now I'll admit when I've had to do something quick and dirty I've thrown in the occasional GOTO but the GOTOs all went to the same place whereas this massive pile of shite had more twists and turns than a bad detective novel!

But since you have seen the "true horror" of IT I'm sure you can see why I sometimes want to bash my head on the desk when dealing with FOSSies. They always seem to think "All you have to do is replace Windows and Office, and all will be hearts and flowers!" when it is NEVER Windows and Office that is the problem. Hell most companies can't even upgrade to the latest windows version for fear of that giant reeking mess of garbage code they've come to depend on will come falling down like a house of cards in a hurricane.

And while I agree 110% that the goal should ALWAYS be KISS, the problem is the PHBs at these places will never ever in a million years shell out what it costs to actually get all the data out and build a REAL solution, not until they have the crap they are depending on fall apart like some giant train wreck from hell. Like that VB4 app I mentioned early on, I ended up just jamming the thing in a Win2K VM and letting it rot for the next guy. Not because that is what I wanted to do, but because you couldn't pay a college kid what they wanted to pay to have it rewritten, much less get a REAL programmer of any skill.

And that to me is the problem in a nutshell and why the shit isn't gonna get any better for the foreseeable future. It is because businesses in the USA can't pull their heads out of the stock page to realize that working long term solutions cost money no different that decent roads or schools, and look how well THOSE are faring here in the US. IT has it even worse because as long as the thing works that day they don't care if it is a single power surge away from taking down the whole company. To them ANY expense related to IT is just a "waste" unless it is something like an iShiny for themselves or the CEO. That is why I ended up getting out of dealing with corporate, because I frankly got tired of people with impossible problems that wanted to pay pennies to fix years worth of neglect. I mean with the corporate attitude in the USA, is it any wonder nobody young is going into IT anymore? You'd have to be nuts!

Re:How is this newsworthy? It's just common sense.

[AmiMoJo]

Some admins are just scared of things going wrong during maintenance do don't do it. My boss at the last place I worked was like that. He never did updates on servers if he could possibly avoid it. The fear that the server might not reboot and he would have to drive all the way out and spend part of his weekend fixing it stopped a lot of preventative maintenance getting done. It was easier to blame users when the RDP server was hit with a 2 year old virus using a vulnerability that was long since patched. Ditto capacity upgrades.

In some ways frequent virus attacks are useful for an IT admin. They can easily be blamed on users and anti-virus vendors, take time to fix but don't cost much (at least not for the IT department) and justify the admin's existence. Similarly the best way to get an expensive new backup system is to delete some of the boss' email and blame it on a failed HDD. That's the problem with all maintenance though: it just looks like a bottomless pit into which you poor money, at least until the day you fall in yourself and land on a nice soft pile of bank notes (unless you put pennies in, in which case you are fucked.)

Re:How is this newsworthy? It's just common sense.

[esimone928]

Common sense to anyone that been working on systems for over a few years. It's not common sense to 90% of most business executives. What amazes me are the comments surrounding Windows &VB. Those applications are just "the peak of the iceberg" . While I have seen older client/server applications in dire need of modernization (VB, PowerBuilder, Smalltalk, C++, etc) the REAL problem exists on mainframe & midrange systems where 20+ year legacy (COBOL, Adabas/Natural, RPG, PL/I, Assembler etc.) applications are still running. This is the real problem...the systems running our governments (local, state, federal) and large corporations (banking, insurance, healthcare). The Y2K problem was a nit compared to this. To complicate matters, most organizations have no idea how these applications are architected. The the people who even have an inkling are gone...most of them retired. Who's going to maintain/modernize/replace these systems? Conclusion, this is very a conservative estimate on the problem and therefore newsworthy.

Re:How is this newsworthy? It's just common sense.

[DarthVain]

Ya, well I'm the Joe Schmo that gets tasked to fix this sort of stuff.

The conversation usually goes like this:

Me: "To fix this properly you will need to do this, buy this software, redesign this system, it will take X amount of dollars and Y amount of time."
Boss: "We don't have the money. You need to fix it as best as possible using the materials at hand with no budget, and a limited time."

Thus, slaps together whatever I have to get it working, cheaply and likely crudely. Forget documentation, you don't have time for that, besides that's like evidence that I was involved, which means the next time this stupid system breaks (and oh by god it will) you will be called in to help try and fix it, again with the restrictions like above.

Believe me, I know how to do it right, I will even offer up my opinion that proceeding like this is a "very bad idea", but when a manager makes the decision, you do it, and sadly hope it is some other "Joe Schmo" that gets stuck with it next time around... :)

Re:How does that work again?

[jeffmeden]

No one gets my jokes around here.