IT Infrastructure As a House of Cards

snydeq writes "Deep End's Paul Venezia takes up a topic many IT pros face: 'When you've attached enough Band-Aids to the corpus that it's more bandage than not, isn't it time to start over?' The constant need to apply temporary fixes that end up becoming permanent are fast pushing many IT infrastructures beyond repair. Much of the blame falls on the products IT has to deal with. 'As processors have become faster and RAM cheaper, the software vendors have opted to dress up new versions in eye candy and limited-use features rather than concentrate on the foundation of the application. To their credit, code that was written to run on a Pentium-II 300MHz CPU will fly on modern hardware, but that code was also written to interact with a completely different set of OS dependencies, problems, and libraries. Yes, it might function on modern hardware, but not without more than a few Band-Aids to attach it to modern operating systems,' Venezia writes. And yet breaking this 'vicious cycle of bad ideas and worse implementations' by wiping the slate clean is no easy task. Especially when the need for kludges isn't apparent until the software is in the process of being implemented. 'Generally it's too late to change course at that point.'"

Comments

All comes down to budget

[Admodieus]

In most organizations, the IT department is treated as pure cost instead of something that provides strategic value. These IT departments have no chance of getting a budget approved that will allow them to "start over" on any part of their implementation; hence the constant onslaught of temporary fixes and patches.

Re:All comes down to budget

[Opportunist]

Budget and the lack of ability to see ahead, on the side of the decision makers.

Far too often decision makers are not the people who also have to suffer, I mean work with the tools they bought. They are often easily swayed by a nifty presentation from a guy who doesn't know too much either but promises everything, and of course the ability to cut cost in half, if not more, so they buy. Only to find out that the solution they bought is not suitable to the problem at hand. And then the bandaids start to pop up.

Re:All comes down to budget

[Megaweapon]

They are often easily swayed by a nifty presentation from a guy who doesn't know too much either but promises everything, and of course the ability to cut cost in half, if not more, so they buy.

If you've worked in a huge shop, you know that the big software vendors send reps out to IT managers for golf outings and the like. Screw it if the software works or not, just fluff up the guy with the budget rubber stamp.

Re:All comes down to budget

[schmaustech]

If you've worked in a huge shop, you know that the big software vendors send reps out to IT managers for golf outings and the like. Screw it if the software works or not, just fluff up the guy with the budget rubber stamp.

I did not realize my coworkers posted on Slashdot.

Re:All comes down to budget

[eln]

The problem is not with kludges themselves, but with the fact that IT management does not stress documentation and proper change control procedures enough. If a kludge works, is documented, was implemented with proper change controls, and can be repeated, is it really a kludge anymore? IT has to screw around with stuff to make it work, that's what they (we) get paid for. If all we ever had to do was click on an install button and have everything work perfectly from there, what would be the purpose of an IT department at all? Off-the-shelf software and hardware can never be made to work perfectly for everyone's requirements. IT folks are paid to get non-unique components to work for unique requirements.

The problem is not with these fixes, it's that nobody ever documents what they did, and documentation is not readily available when needed. So, these kludges become tribal knowledge, and people only know about them because they were around when they were implemented or they've heard stories. When this happens, these wacky fixes can come back and bite you in the ass later when something mysteriously crashes and no one can get it to work like it did because nobody remembers what was done to make it work before. As people come and go, and institutional knowledge of older systems slowly erodes, we end up in a situation where everyone thinks the current system is crap, nobody knows why it was built that way, and everyone figures the only way out is to nuke the site from orbit and start over. The trick is keeping it from getting to that point.

Of course, nobody likes jumping through all these hoops like filing change control requests or writing (and especially maintaining!) documentation, so it gets dropped. IT management is more worried about getting things done quickly than documenting things properly, so there's no incentive for anyone to do any of it. Before long, you get a mass of crap that some people know parts of, but nobody knows all of, and nobody knows how or where to get information about any of it except by knowing that John Geek is the "network guru" and Jane Nerd is the "linux guru".

We will never get hardware and software that works together exactly the way we want them to. We will always have to tweak things to get them to work right for us. Citing lack of budgets or bug-ridden software may be perfectly valid, but those problems are never really going to be solved. Having our own house in order does not mean fixing all the bugs or being able to refresh our technology every 6 months. Having our own house in order means we know exactly what we did to make each system work right, we can repeat what we did, and everyone knows how to find information on what we did and why.

Re:All comes down to budget

[mlts]

Isn't this taught to death in ITIL 101 that every MBA must go through in order to get their certificate in an accredited college? It sort of is sad that the concepts taught in this never hit the real world in a lot of organizations. Not all. I've seen some companies actually be proactive, but it is easy for firms to fall into the "we'll cross that bridge when we come to it" trap.

Re:All comes down to budget

[BigSlowTarget]

To be blunt, most IT departments act like cost centers and don't provide any strategic value. Business units help by shorting the budgets and whining about band-aid technology instead of seeing how IT can build the business. It takes an exceptional move by IT or amazing insight from a business unit to raise IT above the slog and allow it to provide a competitive advantage to the business units. Projects that do this get firehosed with funding.

Consultants take advantage of this catch 22 situation when they sell new projects. It lets them get the new implementations and cutting edge development. This situation also causes application oriented mini-IT organizations to pop up in the business units from time to time. That, in turn, causes more headaches for central IT.

Re:All comes down to budget

[Grishnakh]

I don't think budget is a problem at all here. The problem as described by the article is with vendor-provided software being crufty and having all kinds of problems. The author even mentions that normal free-market mechanisms don't seem to work, because there's little or no competition: these are applications used by specific industries in very narrow applications, and frequently have no competition. In a case like this, it doesn't matter what your budget is; the business requirement is that you must use application X, and that's it. So IT has a mandate to support this app, but doing so is a problem because the app was apparently written for DOS or Windows 95 and has had very little updating since then.

The author's proposed solution is for Microsoft to jettison all the backwards-compatibility crap. We Linux fans have been saying this for years, but everyone says we're unrealistic and that backwards compatibility is necessary for apps like this. Well, it looks like it's starting to bite people (IT departments in particular) in the ass.

Re:All comes down to budget

[Lifyre]

You should strive for SOME backwards compatibility not honestly supporting more than a generation or two back is too far. Especially when you have a major OS revision (Win2k and Vista come to mind) to act as a breaking point.

Re:All comes down to budget

[almitchell]

That is very true. Unless you are working for a highly visible technology company or high-profile corporation, most companies simply want you to keep the mess you've got going, no matter if it meas bandaids and soldering irons. Over the course of my 20 years career at four different companies, and from talking with colleagues, it is much the same story - the steering committee says, we initially invested hundreds of thousands of dollars, but we sure as hell aren't overhauling and starting over. The best boss I ever had did what I now refer to as guerrilla network administration. We had an aging infrastructure supporting hundreds of banks that we wanted to migrate off Wintel based systems, because they were end of lifing and we were sick of holding it together with bandaids and baling wire. It kept breaking, we sysadmins were sick of sitting through post-mortem meetings, and we were sick of upper management's refusal to acknowledge that it wasn't that the sysadmins sucked, it was that we didn't have a magic wand to keep the old nag on its feet to pull the plow. We repeatedly added a new plan into the budget to change out the system, and just as repeatedly got turned down because of cost. One day we had what was a near catastrophic hardware failure on one of those systems and we had to wait three days to get the parts on it because no one supported it any more. My boss told us to let it sit, and we did, chewing Tums the whole time because it's not like he was the one who would get fired over it. When upper management asked why we were still down after 24 hours, he took a PO in to them and said that when they signed off on new and functional systems, the problem would be fixed. When they balked, he asked them was the cost of a new system really so exorbitant compared to the manhours and effort it took to nurse a sick system well beyond its years. They signed, we had it changed out in two months, and we went another four years without so much as a hiccup. If he hadn't moved out of state I would have followed that man anywhere.

Re:All comes down to budget

[Vellmont]

If a kludge works, is documented, was implemented with proper change controls, and can be repeated, is it really a kludge anymore?

Yes.

You've either don't know what a kludge is, or don't have enough ability to see how fixing things or implementing something the wrong way can really be a horrible mistake that feeds on itself and creates other mistakes. Kludges aren't something you can simply document around. The rest of your post isn't really worth responding to, since it makes the false assumption that kludges are simply poorly documented behavior. If that's the worst you've seen, you're lucky.

Re:All comes down to budget

[HangingChad]

the IT department is treated as pure cost instead of something that provides strategic value.

I can't count the times I've gone in somewhere and saw major deficiencies in their IT infrastructure. I mean really bad, O-M-G size problems. And when you point them out they act like you're trying to pad your billing. Just fix whatever isn't working that day. One of them was a doctors office.

Imagine if their patients acted that way. I don't care if I have cancer, just remove that lump in my underarm.

That's what you get when the problem is dictating the solution.

Re:All comes down to budget

[antirelic]

All comes down to bad management. And what I mean by bad management, is management that does not understand how intricate IT is to their bottom line, and does not plan accordingly. IT is just as important as accounting, or HR. If you neglect it, the problems will cost you down the road.

It doesn't matter what line of business you are in. If you do not understand the role has to play in your company, then you need to hire someone who can help you figure that out, or simply be replaced by someone who can.

Re:All comes down to budget

[Grishnakh]

To be honest, though, Linux is generally very good at backwards compatibility if you statically-link everything when you compile (as is frequently the case with commercial software). The Linux system calls never change, except to add new ones once in a while, so it should be very rare that something doesn't run.

Of course, if something is compiled with dynamic links, this isn't the case, as many of the dependencies will change over the years, but that's why static-linking is available, to avoid this problem. Dynamic linking is better for software that's distributed by the distro, as they can make sure all dependencies in place. Boxed commercial software doesn't have this luxury, so it needs to stick with static linking.

The main place where people complain a lot about Linux's backwards compatibility is with drivers, but that's a design decision. In Linux, drivers are supposed to be included with the kernel. If you don't want to do that, then you'll suffer the consequences. Application software doesn't have this problem as it doesn't link directly to the kernel.

Re:All comes down to budget

[Gorobei]

Yeah, that's why the sane firms have rules on accepting gifts. More than $50 or so once in a while, and you need sign-off before you can even show up.

Re:All comes down to budget

[Cryacin]

Yeah, that's why the sane firms have rules on accepting gifts.

Yes, and both of them have never looked back!

Re:All comes down to budget

[Anonymous Coward]

$50? Our limit's $25!

The VMware rep gave me a backpack with their logo on it and I looked at him and said "this is worth less than $25 right?" while slowly and pointedly nodding.

Note that this wasn't a nice backpack, just a typical nylon thing.

Re:All comes down to budget

[Ritchie70]

They are slowly doing this, they just have to do it slowly.

For example, 16-bit programs don't work on 64-bit Windows.

Re:All comes down to budget

[antirelic]

I wish I wouldnt have blown all my mod points today or I would have modded you up. Your dead right. Implementing something the wrong way is similar to building a mansion on the sand. Understanding the sheer destructive force of a kludge is what separates the senior engineers (and sysadmins in particular) from the rest. This is why you get a crusty veteran shooting down all your bright ideas without ever really explaining it. Because its a game of chess, kludges lead to checkmates... but only the pro's can see that far ahead.

Re:All comes down to budget

[Bing Tsher E]

In most organizations, the IT department is treated as pure cost instead of something that provides strategic value. These IT departments have no chance of getting a budget approved that will allow them to "start over" on any part of their implementation; hence the constant onslaught of temporary fixes and patches.

In most organizations, IT is an infrastructure thing, like the file cabinets, cubicle walls, and the pencil sharpeners. So management doesn't think in terms of 'strategic value' regarding IT. They're in the business of producing whatever widgets or other product their company is in business to produce.

IT people tend to think about IT for the sake of IT. What color file cabinets do you think we should order next time?

Re:All comes down to budget

[dido]

It's not so much that no one wants to do them (although this is true to a degree), but that most people, especially managers, never consider that making quality documentation and obeying procedures takes time and effort, sometimes almost as much as developing the software itself. IT professionals are almost always under unrealistic deadlines to get the project finished and working and so these "non-essentials" are among the first things to fall by the wayside when the project meets a looming deadline. This is the main reason people don't want to do them: they don't directly contribute to the "real" work that is visible to the end users of the project, and as such are considered expendable. Eventually the whole project becomes nothing but kludge piled upon kludge, and the only fix is to start over, again repeating the same mistakes that brought the system to that pass.

Re:All comes down to budget

[DrugCheese]

If he hadn't moved out of state I would have followed that man anywhere.

... except following him out of state of course ...

Re:All comes down to budget

[Anonymous Coward]

I worked for a company where the IT department indicated it needed "X" million dollars to fix the corporate engineering backup system. That was too much money. some bright person in the company discovered that if the company outsourced IT to IBM in India, we could save X-1 dollars. To IBM's credit, I believe they did indicate that the company needed to spend $ on upgrading the antiquated disk backup. But times were tough, and the money wasn't spent. IT folks were laid off en-masse. Then one of the disk server cards had a bug which corrupted the engineering server disk set. The engineering department (one entire floor of the company) was out of work for a month while the poor surviving IT person struggled to restore the files off of archived tapes. Took an entire MONTH to restore the system. Parts of engineering were back online in two weeks, but much of the company was down.
Worst IT disaster of my entire technical career. I had much of my material on my local workstation (in-house, but field support employee), however much of engineering was completely down. Reminded me of the old ancient days at Motorola when all the technical staff was dependent on the mainframe. I had thought workstations resolve all of that, but I was mistaken....

Re:All comes down to budget

[Artifex]

They are often easily swayed by a nifty presentation from a guy who doesn't know too much either but promises everything, and of course the ability to cut cost in half, if not more, so they buy.

If you've worked in a huge shop, you know that the big software vendors send reps out to IT managers for golf outings and the like. Screw it if the software works or not, just fluff up the guy with the budget rubber stamp.

Reminds me of when I heard the manager of a customer service organization I used to be attached to signed off on a new phone system costing hundreds of thousands of dollars, along with the offhand remark that he and a couple other managers were attending "training meetings" in Hawaii for a few days.

Yeah.

Re:All comes down to budget

[sjames]

The problem is a vicious circle. IT doesn't want to make any grand proposals because they fear management will tell them to just squeeze it in when they're not busy but will expect an actual delivery date as if it were a properly funded project with dedicated staff.

Re:All comes down to budget

[robot256]

Speaking from experience, one place this can actually happen is NASA. It may seem like a HORRENDOUS amount of paperwork to do for a flight project, but you better believe it makes a difference. For example, the space shuttle has so many retrofits and modifications and just plain screw-up-fixes it's a miracle the thing flies at all, but they have the procedures so well controlled that now every band-aid get applied every time like clockwork.

I'm really not surprised understaffed IT departments don't spend that much time on documenting what they've "already finished". Even at NASA, it's hard enough to twist people's arms (myself included) into coming up with real manuals and changelogs, but you get what you pay for.

Re:All comes down to budget

[TheRealMindChild]

I've seen some companies actually be proactive, but it is easy for firms to fall into the "we'll cross that bridge when we come to it" trap.

To be honest, in all of my years as a programmer eventually becoming a full software engineer (meaning I design, implement, and maintain software solutions), doing it "The Right Way" has always lead to bankruptcy. Always. Of course correlation is not causation, but for the times I've seen companies fail when "following the process" vs. "Release early and often", the latter half were the ones to stay in business.

Re:All comes down to budget

[antifoidulus]

s/we come to it/after I get my bonus and then bail out for a better paying job/

Re:All comes down to budget

[Anonymous Coward]

Let me say as a developer on a Point of Sale application, we are working hard to support Vista and Windows 7 - should be out by October. (Maybe in time for W7 SP1?)

Re:All comes down to budget

[Anonymous Coward]

This doesn't happen just in major corps with IT Departments. I work in small 300-400 people software company where the core of our flagship product hasn't changed in the better part of a decade. The problem is you can't sell a rewrite. We can't ask our salesmen to tell our clients that this new version is completely rewritten from the ground up without band-aides patches or hacks so its way better but oh by the way we didn't have time to implement any of the new features you've been asking for... I'm not saying there isn't a happy medium but it isn't always easy to rewrite the core of an application while tacking on new features at the same time. So in the interest of competitiveness it gets pushed to the back burner again and again. 10 years later to the surprise of no one whose seen it the application doesn't perform very well. Layers of hacks have gotten to the point where the simplest of requests results in a dozen database queries. Now when we rewrite the app (in progress) we can sell it as 3x faster! I guess what I'm saying is that myopic mangement isn't always the culprit; sometimes it's intentional.

Re:All comes down to budget

[hitmark]

sadly, the ones being pampered are also the ones able to wave the rules...

Re:All comes down to budget

[DrSkwid]

Not just in business :

Worse Is Better

is the foundation of every piece of software you're using, bar a few.

Re:All comes down to budget

[Some Bitch]

To be honest, in all of my years as a programmer eventually becoming a full software engineer (meaning I design, implement, and maintain software solutions), doing it "The Right Way" has always lead to bankruptcy. Always. Of course correlation is not causation, but for the times I've seen companies fail when "following the process" vs. "Release early and often", the latter half were the ones to stay in business.

You can do "release early, release often" within an ITIL framework, just because most places implement it poorly doesn't mean it can't be done well.

Re:All comes down to budget

[Opportunist]

I know. I was the guy with the budget at some points in my life. Problem: I don't play golf, and I prefer burgers to caviar. Plus I know IT. So what happened? They went further up the ladder and invited my superior, which basically meant that he went on and badgered me to buy that crap.

What do we learn from it? That you can as well be for sale. The only thing that changes is that you get fluffed instead of just having to suffer from the crap.

Re:All comes down to budget

[Opportunist]

Do you need a CTO and are you hiring?

Please?

I can't stomach this "culture" here anymore where you either get bought or some of your superiors gets and prods you 'til you cave in.

Re:All comes down to budget

[Opportunist]

Odd, ain't it? Those sales, I mean, training meetings are always in a holiday resort. When your boss is at a "business meeting" at some place near the sea or high up in the mountains (Summer and Winter, respectively), you better make some room in the next few weeks in your schedule, you're gonna get some new hard- or software.

Re:All comes down to budget

[wintermute000]

This is because IT is managed by managers, not engineers.
If all managers had coalface IT backgrounds at least (even to the point of just helpdesk) the problem would not be there.
As usual strategic and policy decisions are being made by people who don't understand the nuts and bolts.
Would you design a car by having a committee of non engineers approving every major decision. No. But that is how IT infrastructure seems to be built...

Re:All comes down to budget

[TheRaven64]

The correct strategy is to get someone whose job it is to take bribes. You pay them a small salary, which they can add to with any extra gifts that they receive. You tell all of the sales reps that they are the ones with the final purchasing authority. Then you let someone competent actually make the decision.

Alternatively, you can use a downwards delegation strategy, where the people at the top have to justify purchasing decisions to the tier below them (recursively). You're free to take as many kick-backs as you want, as long as the people implementing your decision agree with it.

Re:All comes down to budget

[TheRaven64]

The only thing surprising about TFA and your comment is the belief that this is in some way specific to IT. Every single form of infrastructure has the same trades and the same bad decisions. Investing capital in infrastructure is expensive, has small short-term benefits, often causes short-term disruption, and usually has a significant long-term payoff. Investing the minimum in infrastructure maintenance lets you redirect money to areas that have more obvious short term benefits, doesn't appear to cost anything, and leaves you with a huge mess in the future.

Exactly the same economics apply to a small business IT department and a national rail, road, or telephone system, the only difference is scale. Roads don't make money (with the exception of a few toll roads), they make it possible for other people to make money. IT is the same.

Re:All comes down to budget

[TheRaven64]

In a lot of cases, it's because IT people often don't have a good understanding of their customers' needs. If you go to a factory, the people responsible for maintaining the machines can go to management and say 'if you replace this one with a newer version, we can produce widgets 20% cheaper. It will cost this much, and result in five days of downtime for the production line while we install it. If you don't, then the cost of maintaining it will rise by 5% each year'. This is the kind of thing management wants to know. There's a capital cost, and there's a return. They do some calculations and decide whether or not to upgrade. When an IT department requests money for a new deployment, they can rarely provide this kind of information.

Most companies that have IT departments are heavily dependent on their IT infrastructure, but they tend not to be good at communicating between the users and the support staff. This isn't always the fault of the IT department. It is the job of senior management to provide strategic direction ('this is where we're trying to go') and of junior management to provide the current status ('this is where we are now, and why we're not where we are trying to go'). It's the job of IT to take this information and provide support ('this will make it easier to get from here to there'). Without the correct input, the IT department can't produce the correct results.

Re:All comes down to budget

[ckaminski]

Yeah. That only took 15 years.

Re:All comes down to budget

[wvmarle]

Indeed. And I would almost call this story a dupe... there surely are more reasons to apply band-aid instead of starting all over than just budget. Continuity comes to mind.

Re:All comes down to budget

[dekemoose]

Why would anyone who isn't able to wave the rules be getting pampered by a sales weasel?

Re:All comes down to budget

[dekemoose]

Sales weasels don't know IT, but they aren't necessarily dumb either. If they've been doing it for any period of time they're good at their task, which is finding the decision maker and geting them to choose their product. You can try and use red herrings to keep the weasels away from those with signing authority but they'll figure it out eventually. If they don't figure it out they won't get the sale and eventually they'll be out of the business, natural selection at work.

Re:All comes down to budget

[dekemoose]

This is because IT is managed by managers, not engineers.
If all managers had coalface IT backgrounds at least (even to the point of just helpdesk) the problem would not be there...

Bullshit.

I've seen organizations where the managers were IT staff promoted into management, their leve of success did not exceed that which I've typically encountered. Some of these promoted IT folks were lousy managers (for a long list of reasons that typically mirrored why other non-IT originating managers were lousy) and some of them were hobbled by a variety of forces that originated one or more steps above their pay grade. A good manager will understand the need to produce value from IT in order to get the status of the IT organization recognized as something more than a cost center. They won't single handedly come up with the solutions that do this but they will try and drive the organization in that direction.

Good managers are hard to find, period. The myth that exists among the IT labor force that all problems would be solved if management came from within their ranks is just plain wrong.

Re:All comes down to budget

[Hognoxious]

Why not have a rota? Or make it part of the prize for "employee of the month" or something.

Re:All comes down to budget

[dekemoose]

Okay, in a response to a previous comment I went off on a rant about managers not needing to be from IT to be good. I think this is a perfect example of what I was talking about. Nothing in this comment indicates that the manager was a techie (he may very well have been but you can't tell that from what's stated here). He identified a problem that truly needed to be solved and took a stand in order to get that problem addressed. He didn't do anything technically brilliant, he just did something that was right for the company and right for his people in the long term, that's leadership and integrity. Unfortunately it is in damn short supply just about everywhere.

Re:All comes down to budget

[Hognoxious]

If a kludge works, is documented, was implemented with proper change controls, and can be repeated, is it really a kludge anymore?

Yes. Doing the wrong thing right is better than doing the wrong thing badly, but not by much.

IT folks are paid to get non-unique components to work for unique requirements.

This is true, but there's still right and wrong ways of doing things. For example, most sales accounting systems have ways of dealing with returned goods. Pretending the customer is a supplier and buying stuff back from them is the wrong way.

So, these kludges become tribal knowledge, and people only know about them because they were around when they were implemented

There's a quote at the beginning of LoTR about history becoming legend, and legend becoming myth and myths being forgotten. I totally agree, I've seen this in practice.

we end up in a situation where everyone thinks the current system is crap, nobody knows why it was built that way, and everyone figures the only way out is to nuke the site from orbit and start over.

They may think the current system is crap, but that doesn't mean they want to change. Partly through fear of the unknown, and partly because if the system is replaced those people's knowledge of the old system becomes worthless. Job security.

Re:All comes down to budget

[EvilBudMan]

You think that's bad; we get looked at funny for accepting a free pen.

Re:All comes down to budget

[Anonymous Coward]

"The problem is not with kludges themselves, but with the fact that IT management does not stress documentation and proper change control procedures enough. If a kludge works, is documented, was implemented with proper change controls, and can be repeated, is it really a kludge anymore?"

Yes. It's a documented kludge, but a kludge nevertheless.

"The problem is not with these fixes, it's that nobody ever documents what they did"

You are under the "fog pov". Kludges *are* the problem. They not being documented only make them worse. You think underdocumentation is the problem because it's the first stone you find in your way and it blocks your view for everyting that lays beyond your horizon.

I'm myself I'm an anal retentive regarding documentation with a bit of power -just a bit, so documentation is a must or you're fired. Unluckly with not enough power to stablish policies that go even a bit after tomorrow, so all my power ends at the "Ye Shall Document The Kludges" point.

And d'you know what? Kludges are still kludges. We have a hill of documented kludges up to the point that when we break something I can always point to the document -even a not so difficult to find *after the fact* document, that states in big capital letters "Why Ye Should Not Do This". Of course, not sticking with the KISS and "do it the obvious way" principles connatural to kludges (hey, there *is* a very strong reason we did it in such a strange, counterintuitive, fragile way, you can be sure of that) is the reason why we wreak havoc so much often.

Just like in code, "natural", obvious, well thought out systems need almost no documentation while dirty hacks and kludges are still dirty hacks and kludges no matter how much documentation you through to it.

"Of course, nobody likes jumping through all these hoops like filing change control requests or writing (and especially maintaining!) documentation, so it gets dropped."

Of course it gets dropped! (unless you happen to suffer a maniac minion-manager like me): the net effect is a system that breaks as often, more "paperwork" to be done with less "results" and being ashamed when you fail and then you are showed that if you only took into account the proper document everything would have went OK.

"Citing lack of budgets or bug-ridden software may be perfectly valid"

But citing vision-lacking management will always be more near to reality. I have a record of trying to make management pay attention to an obvious problem... 30 months!!! in advance (when it was a project risk at early planning, when it was something to pay attention to, when it was an obvious outcome, when it was something we needed to pay immediate attention to, when it was shit hitting the fan). Only when shit was way beyond the other side of the fan and money in the five zeroes range was lost was when management started to think it was time for a solution... basically made up of unpaid extra hours from the technical staff and a ton of band-aid. *THIS* is the real problem, not lack of proper documentation.

Well, I feel better after the rant... but I think I'll post this anonimously.

Re:All comes down to budget

[david_thornley]

In most organizations, the IT department doesn't provide anything of strategic value, and exists only as a necessary support function. From the business point of view, there's no point in spending extra money to clean up something that works anyway, and wouldn't work noticeably better if cleaned up. The IT department is just another bunch of people who say they could do their jobs better if given more money.

These organizations are also going to be firmly Microsoft, since Microsoft is a known cost and is known to be good enough. Trying something that might be cheaper also carries the risk that something will break, causing more harm to the organization than could be saved.

In order to get out of that, the IT department has to come up with some way to be strategically significant, to offer some sort of competitive advantage to the organization (other than marginally reduced IT costs). Once that happens, the IT department isn't a pure cost center, and can get its priorities considered. The budget process is still going to be difficult, and IT won't get all of what it wants, but it'll at least be a player and not a pawn.

People who want to be non-business techies and still get some respect need to find an organization where software is already strategic because of what other people did.

Re:All comes down to budget

[lennier]

IT has to screw around with stuff to make it work, that's what they (we) get paid for. If all we ever had to do was click on an install button and have everything work perfectly from there, what would be the purpose of an IT department at all? Off-the-shelf software and hardware can never be made to work perfectly for everyone's requirements. IT folks are paid to get non-unique components to work for unique requirements.

Yes.

And this is ad-hoc fixing and configuration management is, in fact, programming, though software developers tend to sneer and look down their C++ noses at 'mere' IT as if it were a nasty swamp inhabited by sub-human monkeys. Our wonderful application/webservice works perfectly, so managing the configuration and integration and data must be a user issue! And users are dumb, so they don't need programming languages, they'd just hurt themselves. Give 'em a half-baked GUI with some flying animations instead. And sysadmins are just slightly less dumb users, so don't let them have real languages either.

This is madness.

Part of the problem of IT complexity management is that we do not have decent programming tools for managing systems composed of aggregations of hardware and software in the large. We don't even have decent theoretical abstractions. 'Objects' was the best we got, but unlike functional programming or relational databases, there is no consistent mathematical foundation to OOP (not even a self-consistent definition of the phrase between OOP 'experts'!), so objects are still not consistent between systems. There is no site-wide equivalent to the 'source code repository', and there needs to be.

But more and more, even 'classical' application and webn programming is becoming about stringing together modules and servers from different vendors, and then writing the appropriate glue to work around their lack of appropriate interfaces and abstractions for the task at hand. So eventually application developers are going to be in the same tarpit that Windows sysadmins have been wading through for years. Congratulations. Welcome to our nightmare, guys. Maybe now you can help us get out?

We need sensible glue languages for IT, and we need sensible enterprise-wide data structures. The tools we have at the moment are insanely ancient and decrepit, but the new ones - object oriented whizzybangs - are even worse. At least filesystems and 7-bit-ASCII control files work, can be backed up, version-controlled (to a point), edited and restored.

But we should be so much further along by now, that it's sad.

Re:All comes down to budget

[wintermute000]

I call bullshit to your bullshit.
All the best managers I've seen have some experience as a grunt and / or worked their way up.

By grunt I don't necessarily mean desktop monkey or helpdesk, just they at some point in time knew how to actually make something happen as opposed to just paperwork or management.
Then again we're just trading anecdotes eh?

Agreed many IT people make lousy managers. I'm not disputing that. But what happens if IT management don't actually understand the environment they manage nor the fundamentals of the technology? Is the guy making the decisions going to rely entirely on whatever the architects or salesmen say (in which case why is he even there? put the architect or the salesman in charge lol) or just choose the cheapest option (in which case why not just put the accountant in charge?). I've seen both extremes play out in corporation land. Neither leads to a good efficient environment.

Of course these days the trend is to pick the cheapest option wherever possible, so in that case why not just hire a primary school kid for pocket money, surely they can look at a row of spreadsheet columns and pick the cheapest.

IT skills are by no means the most important skill in a manager but understanding the job and the environment can only be beneficial. It enables better evaluation of different options, strategies and policies, better man management, to name two.

Kludges are short-time fixes and long-time problem

[Anonymous Coward]

If you don't invest the time to do something properly the first time*, you've increased the cost several times over since the wrong decisions will progress & amplify into even worse decisions & implementations meaning the cost to get back to a correct solution will only increase. And since you'll accrue dependencies on your wrong decisions, the costs will just amplify more.

* Doesn't mean you solve the problem the right way the first time. But it does mean that you are able to identify when things go beyond implementation problems into architectural & process problems & solve those before they become even bigger problems.

I don't believe in a lot of things

[Culture20]

...but I believe in Duct Tape.
As long as your backup and tertiary machines have different kludges keeping them running, there's no problem...

As a non-developer, this is what I see

[Em Emalb]

Maintaining code is boring.

Everyone wants to work on the latest and greatest stuff, no one wants to maintain or even release patches.

It sucks, especially since it isn't limited to just software development.

I've seen companies where their "core switch" was a Cisco 2548. This wasn't 10 years ago, this was last year! Unreal.

Re:As a non-developer, this is what I see

[drachenstern]

As a dev, what's the problem with a 24 port gigabit switch as the "core" on a medium sized office? Aside from the fact that 10Gb is becoming popular (has become popular?) in the datacenter? Most desktops are only at the 1Gb level (and most users at below 100Mb), and most inbound internet pipes are much smaller. I don't understand the downfall here.

Can you elaborate?

Re:As a non-developer, this is what I see

[sexconker]

I've seen companies where their "core switch" was a Cisco 2548. This wasn't 10 years ago, this was last year! Unreal.

Solid product lasts a long time and does the job.

Is this a problem in your Bizarro world?

Re:As a non-developer, this is what I see

[Em Emalb]

No redundancy, is the biggest one. No real layer 3 switching is another.

Re:As a non-developer, this is what I see

[oatworm]

Ditto this - plus, in a medium-sized office, you're probably not getting 10x24Gb/sec out of your server infrastructure anyway. Your network is only as fast as the slowest component you rely upon; at 10Gb/sec, you're starting to bump into the limits of your hard drives, especially if you have more than a handful of people hitting the same RAID enclosure simultaneously.

Re:As a non-developer, this is what I see

[JerkBoB]

As a dev, what's the problem with a 24 port gigabit switch as the "core" on a medium sized office?

If all you've got is 24 hosts (well, 23 and an uplink), then it's fine. I suspect that the reality he's alluding to is something more along the lines of multiple switches chained together off of the "core" switch. The problem is that lower-end switches don't have the fabric (interconnects between ports) to handle all those frames without introducing latency at best and dropped packets at worst. For giggles, try hooking up a $50 8-port "gigabit" switch to 8 gigabit NICs and try to run them all full tilt. Antics will ensue... The cheap switches have a shared fabric which doesn't have the bandwidth to handle traffic between all the ports simultaneously. True core switches are expensive because they have dedicated connections between all the ports (logically, if not physically... I'm no switch designer), so there's no fabric contention.

Re:As a non-developer, this is what I see

[Em Emalb]

Is this a problem in your Bizarro world?

Yes, in my "bizzaro world", most people call it the real world, using an edge switch as your core or "main" switch in your network is considered a very very bad thing.

Re:As a non-developer, this is what I see

[pilgrim23]

Its wonderful to ride the developer ship. But once the brilliant code is down on silicon, and runs into reality it must be patched. Patching is so humdrum, so tedious. No admiring fans, just plodding day after day finding that routine that seems to always call the variable from nowhere "Object not set to an instance of an object". Frank Loyd Wright was a great architect. People marvel at his design. Few know the name of the roofer who has to repair the design flaw that makes every Wright Roof Leak....

Re:As a non-developer, this is what I see

[dbIII]

Redundancy? In such a small setting if thing dies you either go out and buy a single easily available item or more likely you dust off the old dumb 100mb/s switch it replaced and pop it in the rack.
Layer 3 switching is nice but has only very recently become cheap enough to justify in many settings and it really isn't needed much. In a small office if somebody brings in something which acts as a DHCP server the sysadmin simply wanders around with a blunt instrument until the culprit is found, and unused ports are not even patched in. Security features are really more important when you can't easily see what is going on everywhere - hence something a lot bigger than what you are smirking about here.

Re:As a non-developer, this is what I see

[Anonymous Struct]

Absolutely nothing. A 24 port gigabit switch makes a great foundation for a small to medium-sized network with typical business use. It's a stretch to call it a 'core', but anybody who tells you that you need some kind of crossbar fabric chassis switch at the center of your average branch office is just trying to sell you hardware and service contracts.

Re:As a non-developer, this is what I see

[Em Emalb]

The network it was running was not a small network. Not at all. It was a travesty that this poor switch was running the network. Well over 200 devices plugged into other 2548s all bridged back to the poor "core" switch.

Re:As a non-developer, this is what I see

[Cryacin]

Few know the name of the roofer who has to repair the design flaw that makes every Wright Roof Leak....

You're not doing it Wright!

Re:As a non-developer, this is what I see

[Anonymous Coward]

In the real world for most people, IT infrastructure is taken for granted and underfunded, so using an edge switch as the core switch is all they have. If they have a decent brand that doesn't crap out after a year it will get the job done. This is especially true in small companies with 50-100 employees.

I think your idea of real world only applies to some people, not most.

Re:As a non-developer, this is what I see

[drachenstern]

Well so 200 devices still sounds like a small network to me. But this is the first reply I've read since posting earlier, let's see what else is on the thread.

Re:As a non-developer, this is what I see

[David_Hart]

First, the Cisco 2548 is a 48 x 100Mb + 2x 1Gb GBIC. Which means that it is old tech. My guess is that it may only support ISL and not Dot1q trunking, and likely only supports VTP 1.0. In other words, it's really old tech, has no support, and for which you likely can only find parts on eBay, time to upgrade. In fact, our company is in the middle of a hardware replacment cycle. Some of our guys didn't have the right cables for one site and tried to connect some 2924 C XL switches to a new 3750 stack. They just didn't want to talk. I'm guessing that a firmware upgrade would have fixed the problem (latest firmware was 8 years newer than what was on the box), but they decided not to bother and to get the right cables in the morning.

Second, it depends on what you call a medium size company. To me a medium size company would be 1000 to about 2500 users. In that case, you need a decent layer 3 switch at the core. This usually means a pair of 6500 switches for the core and 4500 switches for access.

And just to be clear, I am not a huge Cisco proponent. I believe in the right tool for the job. I also beleive that Juniper has better products in certain areas for 1/3 the price. In fact, I believe that Juniper will be eating Cisco's lunch in the small and medium size environments over the next few years. It is one of the reasons why Cisco has been working to diversify.

David

Re:As a non-developer, this is what I see

[Ultra64]

Why?

Re:As a non-developer, this is what I see

[Bing Tsher E]

Solid product lasts a long time and does the job.

Is this a problem in your Bizarro world?

Maybe he sells networking hardware for a living.

Re:As a non-developer, this is what I see

[dbIII]

Now that's the sort of thing you should have written first instead of the stupid blanket statement isn't it?
It's just like in engineering - it's very easy to stop things corroding, just coat everything in gold. In practice you determine where to put the expensive bits. Even in the example you give here we can't tell if using the device as a core switch was a bad idea or not for instance in cases where there isn't much activity since two hundred things doing almost nothing still adds up to almost nothing, but I'll take your word for it.
I'll let you get back to your "real world" of an office :)

Re:As a non-developer, this is what I see

[TheRaven64]

And did you do a user study to find out how long a typical user is spending waiting for data over the network? Did you aggregate this to provide an estimate of the total opportunity cost caused by the poor infrastructure? When you compared this to the cost of upgrading, how long would it have taken to recoup the initial infrastructure investment?

Re:As a non-developer, this is what I see

[ckaminski]

I just looked up the fabric speeds of a 6509 I used at my last company. 40 Gb/slot, 720 Gb total system throughput. At 24 port densities, this was almost fast enough to run every port at 1Gb/full duplex. Almost. At a total system cost of nearly $750,000, it wasn't a cheap investment, but aside from port density, we never had a problem with it. For that reason alone it was worth it's weight in gold... :-/

Re:As a non-developer, this is what I see

[sexconker]

A switch is a switch.
Performance and feature set were known at the time of purpose.
The switch was chosen for a reason.

If it gets the job done, then what are you complaining about?

How will a NewFangledSuperSwitch make things better?
How much will it cost?
How long will it be until you recommend NewerRangledSuperDuperSwitch?

Re:As a non-developer, this is what I see

[Em Emalb]

A switch is a switch. What? So you'd be perfectly fine with using a 3com switch at your corporate headquarters for your company with 120+ employees and 100+ servers?

Performance and feature set were known at the time of purpose. No, just like the subject that was originally being discussed, the switch was put in place a long time ago and never swapped out, even though performance and work arounds had to constantly be put in place for it. (In this case, it was daisy-chaining additional switches, including, and I don't know why it wasn't moved into place as the core switch, a 6506.

The switch was chosen for a reason. And that reason was no longer valid. It was, again, going back to the crux of the article, more of a hindrance than a helper at that point. But rather than fix the problem, people kept slapping on band-aids.

If it gets the job done, then what are you complaining about? Because when I went into that company, it WASN'T getting the job done.

How will a NewFangledSuperSwitch make things better? Uh, by not having tons of collisions, dropped packets, slowness and causing loss of production?
How much will it cost? The recommendation (which they followed) cost around $15,000 to add an additional sup II and a couple GB cards to the 6506.
How long will it be until you recommend NewerRangledSuperDuperSwitch? That all depends on how long it takes before the company runs out of space on the 6506, and also takes into consideration EOL and EOS requirements from Cisco. So...for the time being, they're good (and happy now) for the next several years.

Re:As a non-developer, this is what I see

[EvilBudMan]

You know what, we have a Cisco Pix from 2000 that's actually stopping up our fiber connection to the net and it's noticeable even if you only have 1 machine hooked up, you still don't ever get close to even half of 100 meg down but you do get more than 10 up which is the providers limit. I would bet it's that and the switch. It makes you want to hook the whole place up to 220V and be done.

Re:As a non-developer, this is what I see

[rathaven]

That doesn't sound ideal but without knowing the network and its use it could still to be a design issue. There are ways to get it to work depending on what the other switches were and what the network traffic requirements are for the network service.

The obvious solution is to distribute the network to keep the load as local to the edge as possible preventing the throughput all hitting the core, distribute the connections between switches too - more of a mesh design rather than a star, use redundant links with spanning tree (preferably better than old STP - RSTP or better) to detect loops and block looped connections (these will save your bacon when switches do fail). On many networks you have to do this anyway as there is NEVER enough bandwidth for everything that might be wanted unless you have over specified and have potentially wasted money anyway..

A network that is over specified for convenience or for future proofing has to be justified,

Re:As a non-developer, this is what I see

[sexconker]

So now instead of saying "They're using an old switch and I think it's dumb." you're saying "They're using an old switch that doesn't do what they need because of this and this and this and I think it's dumb because of this and this.".

Maybe if you had said WHY in your initial bitchfest you would have had less people laughing at you. Instead, you threw out a generalization to which I responded "Solid product lasts a long time and does the job. Is this a problem in your Bizarro world?".

In your bitching, you neglected to mention that the company has far outgrown the switch. Protip: Most companies are small and will stay small and will be well served by such a switch. And they would be hard-pressed to find one built today that will last for 10 years.

I blame you. You and lead-free solder, RoHS, etc.

Re:As a non-developer, this is what I see

[Anonymous Coward]

Having a single switch with 24 or a tenth of 6500 with 288 ports each like my company is not the point.
Problem is, they are Cisco.

Re:As a non-developer, this is what I see

[Em Emalb]

I'm glad I saw this response.

Here's a "pro-tip" for you: acting like you have the full details of the situation when you've clearly not been given all of them makes you come across as a know it all. Now, I don't know you from anyone, so that may be an unfair statement, I don't know.

I do know that I could have solved the entire frigging issue if I had said "mid-sized" instead of not specifying the size of the company at all.

But, in typical slashdot fashion, most everyone jumped on the bandwagon (even though most couldn't spell switch if you spotted them the "itch") thinking they know what the hell they're talking about.

So...to sum up: I'll attempt to do a better job of describing the network in question when I'm discussing the pros and cons of a small edge device when it's being utilized as a core device incorrectly.

Re:As a non-developer, this is what I see

[sexconker]

So, you still don't get it.

There's no difference between an "edge device" and a "core device".

There are only the requirements of the network and the capabilities of the switch.

If the requirements of the network can be met by a old, low-end model, then there's nothing wrong with using that.

Your rant is the equivalent of saying:
OMG guys did you see those monitors they're using at that office?! They're 1680 x 1050! All their HD content is going to be scaled terribly! And they're TN panels, too! And none of them have been calibrated!!!!

Vendors ... not their customers.

[Anonymous Coward]

Paul was mostly talking about the software vendors peddling completely outdated designs, where presumably the product is something they that are selling (i.e. s/b a profit center). The problem (in this case) does not reside with the users (customers) of such products, who are probably blissfully unaware of the sort of c**p that they are dealing with.

Summary

[dangitman]

There are a lot of people writing shitty software. Film at 11.

Re:Summary

[wmbetts]

Is that 11 central or eastern time?

Re:Summary

[Rob Riggs]

You obviously do not belong here.

Nerds only have one time zone: UTC

Re:Summary

[sjames]

It doesn't matter much, odds are your calendar program mis-handles timezones anyway.

Re:Summary

[ckaminski]

I think the whole fucking world should work off UTC.

Just saying.

Take responsibility and stop the magical thinking

[DragonWriter]

The constant need to apply temporary fixes that end up becoming permanent are fast pushing many IT infrastructures beyond repair. Much of the blame falls on the products IT has to deal with.

Well, sure, IT departments place the blame there. The problem, though, is not so much with the products that IT "has to deal with" as with the fact that IT departments either actively choose the penny-wise-but-pount-foolish course of action of applying band-aids rather than dealing with problems properly in the first place, or because -- when the decision is not theirs -- they simply fail to properly advise the units that are making decisions of the cost and consequence of such a short-sighted approach.

When IT units don't take responsibility for assuring the quality of the IT infrastructure, surprisingly enough, the IT infrastructure, over time, becomes an unstable house of cards, with the IT unit pointing fingers everywhere else.

And yet breaking this 'vicious cycle of bad ideas and worse implementations' by wiping the slate clean is no easy task. Especially when the need for kludges isn't apparent until the software is in the process of being implemented. 'Generally it's too late to change course at that point.'

If your process -- whether its for development or procurement -- doesn't discover holes before it is too late to do anything but apply "temporary" workarounds, then your process is broken, and you need to fix it so you catch problems when you can more effectively address them.

If your process leaves those interim workarounds fixes in place once they are established without initiating and following through on a permanent resolution, then, again, your process is broken and needs fixed.

You don't fix the problems with your infrastructure that have resulted from your broken processes by "wiping the slate clean" on your infrastructure and starting over. You fix the problems by, first, improving your processes so your attempts to address the holes you've built into your infrastructure don't create two more holes for every one you fix, then by attacking the holes themselves.

If you try to through the whole thing out because its junk -- blaming the situation on the environment and the infrastructure without addressing your process -- then:

(a) you'll waste time redoing work that has already been done, and
(b) you'll probably make just as many mistakes rebuilding the infrastructure from scratch as you made building it the first time, whether they are the same or different mistakes.

Magical thinking like "wipe the slate clean" doesn't fix problems. Problems are fixed by identifying them and attacking them directly.

Don’t patch bad code - rewrite it

[D4C5CE]

Don’t patch bad code – rewrite it.

Kernighan & Plauger
The Elements of Programming Style
2nd edition, 1974 (exemplified in FORTRAN and PL/1!)

Re:Don’t patch bad code - rewrite it

[eggoeater]

I couldn't agree more, but that's very expensive and very very dangerous. Why? Two factors:
1. Rewriting means rethinking; most legacy code is functional and is usually rebuilt in OOP. Whenever you rethink how something works it tends to change the entire behavior to say nothing of all the new bugs you'll have to hunt down. You're customers will definitely notice this.

2. Scope creep!! Rebuilding it? Why not throw in all that cool functionality we've been talking about for the past 10 years but couldn't implement because the architecture couldn't handle it. You get the idea.

Want an example? Netscape 5

Re:Don’t patch bad code - rewrite it

[Anonymous Coward]

Netscape 5
AKA Seamonkey Project
AKA Mozilla Project
AKA Firefox

Which has TEN TIMES the market share Netscape 4.7 had at the end of the first browser war

Re:Don’t patch bad code - rewrite it

[TheRaven64]

Which has TEN TIMES the market share Netscape 4.7 had at the end of the first browser war

And about half of the market share that Netscape 4 had before they decided to do a complete rewrite.

Re:Don’t patch bad code - rewrite it

[wvmarle]

We just had that discussion.

Re:Don’t patch bad code - rewrite it

[marcosdumay]

"Which has TEN TIMES the market share Netscape 4.7 had at the end of the first browser war"

And a banckrupcy to be proud at.

Re:Don’t patch bad code - rewrite it

[DragonWriter]

Rewriting means rethinking; most legacy code is functional and is usually rebuilt in OOP. Whenever you rethink how something works it tends to change the entire behavior to say nothing of all the new bugs you'll have to hunt down.

First, presumably that is intended to be "imperative" not "functional".

Second, if you change the externally-visible behavior (including introducing new bugs) as a result of rethinking the best way to acheive the existing externally-visible behavior, you are doing it wrong.

Re:Don’t patch bad code - rewrite it

[eggoeater]

Are you saying that as long as externally visible behavior stays the same that there won't be new bugs/behaviors under the hood??
You can do all the unit and end-user testing you want; when you rewrite something, some behavior is going to change and you will not be able to catch it all. This is coming from someone who's done a lot of rewrite-it-from-scratch projects.

Re:Don’t patch bad code - rewrite it

[DragonWriter]

Are you saying that as long as externally visible behavior stays the same that there won't be new bugs/behaviors under the hood??

If a change in behavior is not externally visible (that is, if it is entirely contained within the internal process of the replaced component, and has no external effect) it can't matter (and can't be a "new bug", since a bug is an undesirable, externally visible behavior.

implemented

[convolvatron]

i guess its ok that the sysadminds coopted the work 'implemented' where one would normally
say 'installed'

but that kind of leaves the actual implementors without a word now

and in this particular usage, its kind of odd, because usually the best time to
find and fix these problems is exactly when its being implemented, rather than
when its being installed

House of Tape

[Renraku]

The whole system is held together by superglue and duct tape because no one wants to pay for a replacement part. This goes for both physical and software analogies.

Written for a P-II 300Mhz?

[damn_registrars]

Wait, you mean there have been newer and faster processors released since then? So Mordac really has been hiding something from me...

Placebo effect - bandaids and kisses

[peterofoz]

With regard to band-aids and kisses, it looks to me like IT benefits from the same placebo effect as little girls do:

Adventures in Mommyville

It a similar effect as when management brings in a new product or consulting group that is the "silver bullet" that will solve all the problems.

Re:Take responsibility and stop the magical thinki

[FooAtWFU]

"they simply fail to properly advise the units that are making decisions of the cost and consequence of such a short-sighted approach."

In the defense of IT, those people they're trying to advise aren't always the best at taking advice. (But then again, neither are IT admins always the best at giving it.)

pay off your credit cards?

[Matthew Weigel]

This the essence of technical debt. Whether you're programming or deploying IT infrastructure, it's inescapable that sometimes you're going to have to include kludges to work around edge conditions, a vocal 1% of your users, or whatever. These kludges are eyesores, and fragile, but they're also as far as you could go with the time and budget you had.

Sometimes, accruing debt like this enhances your liquidity and ability to respond to change, so avoiding all kludges introduces other more obvious costs that slow you down and make you seem unresponsive to users or customers. But you can't just go on letting your debt grow all the time and not eventually come up technically bankrupt. Let it grow when you have to, but just as importantly make time to pay it down. A lot of this stuff can be paid down a little at a time, as you come across it a few months later. The pay-off if you're vigilant is that the next ridiculously urgent fix to that system can often be handled much more easily, without dipping down further... with patience and attention to maintaining this balance, you can reduce your technical debt and make the whole system hum.

The downside is that there isn't a quick fix when you find yourself deep in technical debt. You can't just spend all your time reducing it; your highest aspiration at that point should be maintaining the level of technical debt, rather than letting it grow, but it's generally been my experience that altering the curve of debt growth even a little can set you on the right path.

Re:pay off your credit cards?

[Len.Wright]

Brilliant. I wish I had said this, but in any event, I do follow it and fully believe in what you're say.

Re:pay off your credit cards?

[Rob Riggs]

The downside is that there isn't a quick fix when you find yourself deep in technical debt.

Actually, there is a very simple and quick fix when technical debt becomes overwhelming. Often it's the only fix. CTOs and CIOs do it all the time.

Quit.

Often that is the only way to catalyze the change required for companies to begin reducing their technical debt.

like bubblegum under a desk...

[Thud457]

There's nothing more permanent than a temporary fix.

Re:like bubblegum under a desk...

[Tridus]

Yeah, I saw that line and immediately thought about some of the "temporary solutions" people have proposed over the years. The statement is an oxymoron. It's either not a solution to the problem, or it's not temporary.

We've got less of those being made now, because I've taken to listing the previous "temporary solutions" every time someone proposes a new one.

Re:like bubblegum under a desk...

[Hognoxious]

There's nothing more permanent than a temporary fix.

[PHB] So you're saying that quick and dirty fixes in the past worked ... and some of them are still working? Must be good, then! [/PHB]

Re:like bubblegum under a desk...

[EvilBudMan]

Some still do. It's just the way it is. I have fixed things in a rush and came up with the best solution for then and in the future.

There are very few problems that are really hard to fix once you have fixed the problem of not wanting to get out of bed and go to work. After that it's all easy stuff.

I've seen similar issues with hardware

[peacefinder]

Long ago, at one customer, the desktops grew weak. So they hit upon the idea of using a Windows terminal server to prop up their aging desktops. By gum, that worked so well that they replaced their desktops with thin clients throughout over the next year. Now, a few years after that, their terminal server handles everything from Solidworks viewers to web browsers and it's sagging under the load... and out of warranty to boot. Time to get a new server... only the economy has hurt their business terribly and they currently can't afford one.

In short, they're screwed. Too tied to Windows to easily jump platforms, too tied to thinstations to easily return to the desktop environment, and too broke to maintain it.

Re:I've seen similar issues with hardware

[oatworm]

Ran into a similar situation with an old client of mine, only with more "hilarity" - basically, they had a consultant tell them that they could have a dirt cheap server, a bunch of thin clients, and save money. Queue one Pentium 4-powered server with software mirrored PATA(!!!) drives serving as a terminal server for over 30 people. Queue one Pentium 4-powered server quickly succumbing to the heat and wear that was generated from such a load. Queue one confused customer that was wondering why the company I worked for was suggesting they buy a server that cost three times as much (i.e. one with a SAS-driven RAID 5 and more RAM) as the one they "just got a year or two ago". Queue them telling us to get lost.

Queue them getting bought out by a competitor a year or two later, followed by the competitor rolling the former customer's IT into the competitor's IT infrastructure, since the competitor actually knew how to spec for growth and maintenance. I don't know who won on that one, but... yeah. Think about it.

Re:I've seen similar issues with hardware

[Anonymous Coward]

The only way they can really fix that is a graduated move, where the Solidworks users are moved to standalone machines, while the rest of the people remain on the creaky terminal server.

The problem is that this is a misapplication of technology. A call center where people only run an app that takes down info and updates accounts is perfect for a thin client setup. However, once people are starting to do more than something which essentially a 3270 terminal does, they need either hardware on their desktops, or dumb monitors/keyboards connected to ClearCube blades.

Re:I've seen similar issues with hardware

[timmarhy]

huh? replacing one terminal server is way cheaper then replacing a lot of desktops. you would just buy a new one, put it side by side with the old one and split the users between the 2 of them to balance the load.

plus there's nothing stopping them buying desktops and having them log into the same domain as the terminal server. your post makes no sense.

Re:I've seen similar issues with hardware

[peacefinder]

Hey, I never claimed it made sense, only that's what the client is doing. We've quoted them the most barebones terminal server addition we're willing to support, but they're still holding out for ... I dunno, a miracle? Positive cashflow? An ounce of sense?

Re:I've seen similar issues with hardware

[Mad Merlin]

Cue! Cue cue cue cue cue! Not queue.

Re:I've seen similar issues with hardware

[TheRaven64]

Too tied to Windows to easily jump platforms, too tied to thinstations to easily return to the desktop environment, and too broke to maintain it.

A good first step would be to replace some of the thin clients with hybrid clients. Install *NIX on some cheap (1GHz or so) systems and run rdesktop. They can connect to the central server and function as thin clients. They can also run a web browser, a mail client, and any other fairly generic apps that load the server down but don't need to be Windows-only. Depending on the office, you might be able to reduce the server load a lot just by moving the web browser off the server.

Re:I've seen similar issues with hardware

[oatworm]

I prefer a push-pop method of conversation myself. I was just pushing some info into the list - it's up to the reader to pop it out.

Re:I've seen similar issues with hardware

[peacefinder]

You'd think they'd be willing to do that, wouldn't you?

But no. Some customers are just determined to kill themselves off.

Re:I've seen similar issues with hardware

[drsmithy]

In short, they're screwed. Too tied to Windows to easily jump platforms, too tied to thinstations to easily return to the desktop environment, and too broke to maintain it.

If they can't afford a beefier terminal server, then - ignoring the much bigger problem of being basically bankrupt - they can't afford to do any sort of platform migration anyway, so what they're currently on is utterly irrelevant.

Re:I've seen similar issues with hardware

[peacefinder]

Yeah, that was a shock to find. Just the viewer, but still the viewer workload is not so different from the workload of the full software.

Actually

[Anonymous Coward]

It's the products that are still based on the old, supposedly 'crusty' C/native instruction code that I look for. Obviously, some products are dependent on libraries that have changed and are thus no longer compatible, but those that do still work tend to work VERY WELL on modern hw.

The modern stuff that's supposedly superior is slow and bloated as hell.

Solution is obvious - Linux

[seyfarth]

From the original message we read that the "code was also written to interact with a completely different set of OS dependencies, problems, and libraries." This seems to imply that the IT organizations are allowing outside interests to dictate the rules of the game. If there were a stable set of operating system calls and libraries to rely on, then the software vendors would have an easier time maintaining software. I recognize that Linux changes, but the operating system calls work well and API is quite stable. I have used UNIX for a long time and I have compiled programs from 25 years ago under Linux. There have been some additions since then, but the basics of Linux work like the basics of UNIX from 25 years ago.

At present there are some applications available only on Windows and some only on Windows/Mac OSX. This might be difficult to change, but going along with someone's plan for computing which is based on continued obsolescence seems inappropriate. At least those who are more or less forced by software availability to use Windows should investigate Linux and negotiate with their vendors to supply Linux solutions.

Computers are hard to manage and hard to program. It is not helpful to undergo regular major overhauls in operating systems.

Re:Solution is obvious - Linux

[Anonymous Coward]

I've been saying exactly the same thing since about 1994---since I got into linux thing. Every program I wrote since just "works" without changes (granted, I don't write many gui apps; mostly data management stuff). My Windows counterparts (same corp, doing semi-related apps) have to release a "new version" every time .net is patched---or something along those lines. Your environment shouldn't make your things break or not work right.

Re:Solution is obvious - Linux

[Grishnakh]

The article seems to be about vendor-provided apps that were written for creaky old versions of Windows. Windows, as you probably know, has a whole slew of APIs, and some work better than others. Of course, MS keeps them all around (though not all in good working order) because of their famed backwards compatibility that supposedly lets people run all their old software from 1995 (even though in practice it doesn't work well).

However, "negotiating" with vendors to supply Linux solutions (as great as that would be) isn't really viable. This is really about market sectors where the business needs some weird application which has little or no competition, so the vendor can basically name their price and do what they want. The vendor probably has done very little to update the app in the last decade, because it's a cash cow and there's no competition.

So what is the customer supposed to do? Your "negotiation" idea isn't going to go far: the vendor is going to say, "you will buy our product at whatever price we want, and put up with its crappiness. If you don't like it, go somewhere else! Oh yeah, there is nowhere else to go... hahahaha!!!" So either you deal with the problems, or you don't use the application (and you don't get your business done). That's it. The only other solution is to make your own in-house application, but that may be easier said than done, and many small and medium-size organizations don't have the resources to take on a project like that.

Basically, the IT people are complaining about a situation that has no real solution.

Re:Solution is obvious - Linux

[seyfarth]

Your point about negotiating not working makes some sense. I contend that negotiators must have an alternative. I am a DIY person, so I would always feel that doing my own app is always a possibility. Any application which stays in business would generally have a community of customers. That community could organize to prepare a collaboratively-built alternative. That might allow negotiation to work. If money is your only negotiation tool, that can also work, but it might cost more than living with the existing problems.

Re:Solution is obvious - Linux

[grcumb]

I recognize that Linux changes, but the operating system calls work well and API is quite stable. I have used UNIX for a long time and I have compiled programs from 25 years ago under Linux. There have been some additions since then, but the basics of Linux work like the basics of UNIX from 25 years ago.

I agree that open systems (and the *nix toolkit approach) can make a big difference when it comes to living with legacy code. But, as others have said elsewhere, process matters more than anything.

I speak from bitter experience. I'm currently maintaining a Linux-based infrastructure that has many strengths and is overall quite robust and remarkably efficient. However, it has all the warts that over a decade of living in a production environment can produce. There's one particular bug that I'm taking a break from right now that's causing data loss because it made naive assumptions about the way state would be maintained between two machines. For my sins, I get to dig through ~9000 lines of half-designed code, trying to find the logic behind the state maintenance mechanism (such as it is) and somehow figure out how to stop the problem from happening in the future.

To their credit, management are very understanding and supportive, but within a couple of days I have to go and tell them whether we refactor the existing application or rewrite it. The fact that it's a design flaw means that I lean toward the latter, but how do I cope with this show-stopper bug during the two months it's going to take me to get new code into production? We can't exactly shut down.

There are no easy answers to this question. The fact that I'm working with Linux and FOSS means at least that I can do something about the code, and that I can make changes in smaller increments. But when all is said and done, a lived-in house is inevitably messy. Unless you have monitoring, diagnostics, troubleshooting and maintenance routines developed as formal processes, you will inevitably stagger from crisis to crisis.

Re:Solution is obvious - Linux

[Grishnakh]

Any application which stays in business would generally have a community of customers. That community could organize to prepare a collaboratively-built alternative.

Sounds good in theory, but almost never works in practice.

I am a DIY person, so I would always feel that doing my own app is always a possibility.

Again, sounds good in theory, but almost never happens in reality.

Sure, if you're running a 1-person company or whatever, you might actually make your own app, or try to organize with some other people.

Normal businesses, however, aren't like this. Remember, the article is written from the point-of-view of an IT department head. That alone means we're talking about companies with more than a handful of employees, probably more on the order of 50 - 1000. Companies like this aren't going to go to the trouble of writing their own app (they don't have the resources available, and no one is going to do it on their spare time like you might if it's your own little 1-5 person company), and they're not going to bother trying to find the product's other customers to organize with them. It's easier for management to just tell the IT department to deal with it, and not worry about the problem since the managers aren't IT people anyway, and don't really care about the problem as long as things keep working. When they don't, they'll just blame the IT guy/people and threaten their job(s). If things get too bad, and the IT guy quits, the company will probably fold due to management's incompetence. Remember, most companies are failures in their first few years anyway, it's probably things like this that contribute to that.

Terry Childs rebuild network and look at him now!

[Joe The Dragon]

Terry Childs rebuild the network and look at him now!

If you start over you may be the only one how know how it's works and then some day the PHB will want to mess with it / replace you with a min wage guy who may not know much about how things are setup and can make it to a big mess.

Software = untouchable mentality

[Stiletto]

This happens in commercial software development, too. There's this belief (often held all the way up the management chain to the top) that software, even bad software, represents some kind of massive, utterly permanent investment that must never be thrown away and re-written.

I've worked with managers who would think nothing of throwing away a million dollar manufacturing machine to replace it because it's old, yet cling with all their might to ancient software code that represents a similar level of investment.

Re:Software = untouchable mentality

[Ichijo]

There's this belief (often held all the way up the management chain to the top) that software, even bad software, represents some kind of massive, utterly permanent investment that must never be thrown away and re-written.

Ah yes, the sunk cost fallacy.

Re:Software = untouchable mentality

[lennier]

This happens in commercial software development, too. There's this belief (often held all the way up the management chain to the top) that software, even bad software, represents some kind of massive, utterly permanent investment that must never be thrown away and re-written.

Sure, if the software is actually wrong then it needs rewriting. But just being 'ancient' shouldn't make it wrong. If it does, we have a social/cultural problem in the wider IT industry that needs fixing: why should software go 'out of date' when there are no physical parts to wear out, and the business problem it's solving hasn't fundamentally changed? Why do we expect our software environments to just naturally break? Why do we accept gratuitious change for change's sake, rather than 'do it right, set it in stone, never change it and move on to the next problem to solve?'

And then we should ask: if that 'ancient' software system actually is wrong, then why did we allow wrong software to be written in the first place? How come it tested as okay and got deployed? That would mean that our testing procedures are incapable of telling a wrong solution from a right solution, wouldn't it?

And unless we've fixed those procedures - not just 'changed' them, but can guarantee that they're correct - that means that the new shiny system we're putting in to replace that 'ancient' one most likely is also wrong, and is doing things wrong even as we're trusting it.

That's not good enough. Surely we should stop doing things wrong in the first place, rather than just deploying an endless succession of differently-wrong answers?

It's good that windows 64 kills the old 16 bit app

[Joe The Dragon]

It's good that windows 64 kills the old 16 bit apps off.

But it will be a long time before the old windows 32 bit windows 9x code base it gone in many apps.

Re:It's good that windows 64 kills the old 16 bit

[Darkness404]

How is it good? It leaves the entire internet vulnerable. It pushes people not towards Linux but towards outdated versions of Windows and more or less guarantees the future has 32 bit OSes.

Look at what is keeping people from adopting Linux: Small, niche programs.

With outdated versions of Windows already online, can we afford to push even more people to old, closed, OSes with no future of getting patches?

Re:Kludges are short-time fixes and long-time prob

[Grishnakh]

It doesn't look like "doing it right the first time" is an option here. RTFA. They're talking about vendor applications being crappy and crufty, and IT departments being required to support them. The IT department didn't pick the app, and isn't allowed to not support it. They can't switch to another app (usually apps like this have little or no competition, and they're probably locked-in anyway).

So there's really nothing they can do but complain as long as they're required to support some shitty application on the latest version of Windows, as these are the requirements set down by upper management.

"That was six months ago, Captain"

[jeko]

I'd mod you to 11 if only I had the points.

Re:"That was six months ago, Captain"

[Eli Gottlieb]

If I hadn't posted here already, I'd mod you up for the subtlest Firefly reference I've ever seen on Slashdot.

Re:"That was six months ago, Captain"

[jeko]

Shiny. :-)

You have to keep up with the times!

[bertok]

I'm a consultant that gets to see into the IT world of over a hundred organisations, and I see one major failing that companies make that cost them later:

They fail to keep up with the times.

Nowhere is this more evident than the leviathan government organisations still running Novell and Lotus Notes, on barely patched versions of Windows, with the browser restricted to IE6. Every time I see that combination, the word "expensive" comes to mind.

Solutions that used to be the best are now simply redundant, outdated, incompatible, and expensive to maintain. Inevitably, the whole thing becomes so flaky that it will crash if you apply even security hotfixes, let alone major OS upgrades. This then makes everyone nervous about making any changes at all, which just exacerbates the problem, because by moving more slowly, the organisation falls even further behind the latest generation.

If you're a developer, than the analogy would be a software development organisation that doesn't keep up with the bug fixes until the code becomes so unstable that making any change to it is a gamble. This then means that even applying bug fixes is likely to just break more things, and then the whole thing snowballs until it's best to cleanse the source server with purifying fire. Or better yet, it's like a development company that never bothers to switch to newer versions of the Java SDK, eventually falling so far behind that it's too hard to make the huge leap required to catch up, instead of the many small steps that could have been taken.

The source of the mistake is short-sighted managers always choosing the easy thing instead of the correct thing. Taking small upgrade steps to keep up is critical, because large steps are very hard. A system that doesn't change for years is a temptation for lazy developers and administrators to bake in the peculiarities of the environment, instead of being forced to develop generic and supportable solutions that won't break after an upgrade of a related component.

heres another issue

[nimbius]

companies that over-promise and assume linear growth, observe startup companies with great features and expect retooling their product to compete is a 20 minute job, and have a culture of worthless slags content to wallow in old code as a means to avoid having to learn anything new. reacting in IT to parabolic growth is difficult and patches are cheaper than extended downtime or the methodology to prevent it being instituted in a live system.

Re:Take responsibility and stop the magical thinki

[Bunny Guy]

I'm going to tackle some of the conceptual problems that are hinted at above, which is usually where the difficulties lie, usually in trying to use the wrong software and expecting to somehow "make everything better" if you just make it work "my way" - the true "Magical Thinking".

I tend to agree with your conclusions, "wipe the slate clean" is a drastic action. I disagree with some of the approach you use to arrive at them:

a.) Problems are solved by people being invested in solving them, not process. This requires the antithesis of "Units" - Ownership; Ownership in the company, Ownership of the mission, and a direct heart felt connection to the success of the company. Until you have staff, from the CEO down, that own problems, from the mess in the coffee room to server down time, you will have a "business house of cards" no matter how good the process. In fact, most of the time, fixing things involves re-writing and/or reconsidering process - usually starting with asking the question - "Do we really need that?"

b.) Sometimes you really do have a train wreck on your hands. If you have mastered a.) b follows almost effortlessly, because now, you can *talk* about this behemoth that is eating your company and everybody sees the discussion for what it is, not empire building or managerial fingerprinting.

when you run into a train wreck - assess your tech problem - is the fix easily found? Are your processes using the software at cross purposes? if so, which is cheaper to fix? No amount of bug fixing will repair using the wrong software. It won't even fix using the right software in the wrong way.

In the end, re-asses often, be frugal, not cheap, if it truly is a requirement to run your business, buy the most appropriate. If you've made the mistake of buying a Kenworth long hauler when you needed 3 old UPS trucks - admit it, sell it back, take your loss and get what you really need.

Thats not "magical thinking" it's just common sense.

The meaning of Quality

[bartwol]

More than any other type, businesses are run by salesmen. These are people whose strongest attributes are the ability to build relationships, to communicate value, and a strong inclination to increase their personal wealth.

Increasingly, the stuff salesmen sell is based on complex technologies that, really, are beyond the reach of their comprehension. They kind of understand the products they sell, but really, they don't. If the world only had salesmen, there wouldn't be any sophisticated products.

Say hello to the engineer...a person who builds products. His strongest attributes are a desire to solve problems, a willingness to absorb the tedious but essential details needed to build a complex system, and a personality that derives gratification from doing so.

We now begin the business cycle. The salesman says, "Build me something I can sell."

The engineers says, "I will build you something that works well."

And therein begins a lifetime of the two, symbiotically, talking past each other. The engineer serves the salesman, and the salesman serves himself. But make no mistake about it: the salesman is in control.

For a salesman, QUALITY means it works well enough for him to sell more, and most importantly, to make more money for himself. For an engineer, QUALITY means it works reliably and efficiently. To be sure, QUALITY is an abstract and moving target that varies according to the eyes of the beholder. But to understand why we have the predicament described in this article, we need only understand the SIGNIFICANCE OF QUALITY TO A SALESMAN.

I would continue to expound, but then, most readers here need only reflect on their already frustrated pasts to understand the mechanics of this convenient but often vacuous relationship.

It's not on purpose???

[AthleteMusicianNerd]

I thought IT guys broke shit unnecessarily to maintain job security.

Re:It's not on purpose???

[Anonymous Coward]

No, but they recommend already-broken Microsoft products. Same difference.

Re:Take responsibility and stop the magical thinki

[DragonWriter]

Problems are solved by people being invested in solving them, not process.

Both are, IMO, essential, which is while while I pointed at particular areas of process, my big picture message was about IT shops taking "responsibility for assuring the quality of the IT infrastructure."

Neglect of process is a symptom of people not being invested in solving problems that leads to bad results on its own, but even a good (nominal) process isn't going to work well if people aren't invested.

This requires the antithesis of "Units" - Ownership; Ownership in the company, Ownership of the mission, and a direct heart felt connection to the success of the company.

I prefer "responsibility"; "ownership" is, IMO, misapplied here. (Though, arguably, one of the reasons people do not take responsibility is because they don't, in fact, have ownership -- but ownership is a material relationship, and responsibility is the relevant attitude.)

But I think in substance we generally agree.

Until you have staff, from the CEO down, that own problems, from the mess in the coffee room to server down time, you will have a "business house of cards" no matter how good the process. In fact, most of the time, fixing things involves re-writing and/or reconsidering process - usually starting with asking the question - "Do we really need that?"

You kind of contradict yourself there: if fixing things usually requires changing the process, then "how good the process" is obviously has fairly direct bearing on success. The key thing is that processes aren't good (or bad) in a vacuum, they are good or bad based on the effects they have in your organization, in acheiving your mission; the same nominal process that is good for a group of people when considered against one mission is going to be bad for the same group of people when considered against different goals, and the same process that is good for one group of people with a given mission will suck for another group of people with the same mission, because people matter.

I was torn between modding this up and commenting.

[tlambert]

I was torn between modding this up and commenting.

I picked commenting.

This statement:

Everyone wants to work on the latest and greatest stuff, no one wants to maintain or even release patches.

is very, very true. We (Apple) have a hard time getting applicants who want to do anything other than work on the next iPhone/iPad/whatever. Mainline kernel people are difficult to hire, even though the same kernel is being used on the iDevices as is being used on the regular Macs. Everyone wants to work on the new sexy. For some positions, that works, but for most of them, you have to prove yourself elsewhere before you get your shot.

I think that, for the most part, we see the same thing in marketing for higher education (with the exception of one track, one of the universities I went to has become a diploma mill for Flash game programmers; sadly, I would not hire recent graduates from there unless they have an experience track record). There are video game classes at most universities, but while it might be sexy, you are most likely not going to be getting a job doing video games, 3D modeling for video games, or anything video game related, really, unless you get together with some friends and start your own company, and even then it's a 1 out of 100 chance of staying in business.

I don't really know how to address this, except by the people who think they are going to be the next great video game designer remaining unemployed.

-- Terry

Re:Kludges are short-time fixes and long-time prob

[dykmoby]

... The IT department didn't pick the app, and isn't allowed to not support it....

And they expect you to integrate it with all the other apps that other departments/teams picked without consulting IT for the past 10 years

No!

[Greyfox]

The current patchwork of duct tape and glue that works today is much better than the pie-in-the sky "lets build it from scratch" architecture that IT is pitching that will be late, over budget and eventually have its feature set scaled down until it's less functional than what you have today when it finally is delivered.

There is constant pressure to re-implement existing architecture. Most of the time, the people who want to do this do not have a clear understanding of the business process involved, don't realize that the existing frameworks represent years of bug fixes and are at least stable for that reason. They only think "Wow this sucks, a new one would HAVE to be better."

I'm not saying that you should never rebuild something from the ground up, but the scope of the project should be limited and the entire endeavor should be well documented and well understood from the beginning. And if the guy who's pushing for a rewrite can't demonstrate a deep and fundamental understanding of the business flows being automated, he should be taken out and shot (Or at least pummeled soundly.)

Re:No!

[Anonymous Coward]

Yes, it's stable, because it's impossible to make any real progress on a 10 year old quick and dirty fix over quick and dirty over,
sometimes without proper code revisioning and control...

If the business just needs that old app working on it's own XT, from a floppy disk, and does not complain about it, fine, go with it.

But that is rarely the case.

In my experience, the keep spending on fixing the XT is preferred by the business,
and on bigger companies, it goes even worse, with 40 days to make a one line change
on a good code, the prospect of changing an entire application is not good at all.

Re:No!

[scum-o]

I agree. Scripts and "patchwork" and "duct tape" is easier to maintain for an IT person than a huge program that may be more robust and more well though-out, but if something breaks in the large application, you need to re-design, and get the developer to change things, QA, deploy, etc ... With duct tape and scripts, an IT guy can make a quick change and be back up and running in no time. It's all about maintainability for me. If I can't get into the tool and get my hands dirty and mess around with it, it's a black box to me and unmaintainable. K.I.S.S. - keep things as simple as possible if you want to be able to maintain them. The more complexity that you introduce into an infrastructure the more documentation you need, the more people you need, and the slower your response will be to fix it when it breaks (and things *always* break).

Re:Take responsibility and stop the magical thinki

[Bunny Guy]

I prefer "responsibility"; "ownership" is, IMO, misapplied here. (Though, arguably, one of the reasons people do not take responsibility is because they don't, in fact, have ownership -- but ownership is a material relationship, and responsibility is the relevant attitude.)

But I think in substance we generally agree.

We do, I think the difference is that my experience has been fixing projects starting from a technical complaint to an outside organization and helping those in an IT/Technology organization drive changes up thru their organization, often into the CEO space when needed. From your choice of wording, I suspect, your experience in this might start higher up the product chain.

Until you have staff, from the CEO down, that own problems, from the mess in the coffee room to server down time, you will have a "business house of cards" no matter how good the process. In fact, most of the time, fixing things involves re-writing and/or reconsidering process - usually starting with asking the question - "Do we really need that?"

You kind of contradict yourself there: if fixing things usually requires changing the process, then "how good the process" is obviously has fairly direct bearing on success. The key thing is that processes aren't good (or bad) in a vacuum, they are good or bad based on the effects they have in your organization, in acheiving your mission; the same nominal process that is good for a group of people when considered against one mission is going to be bad for the same group of people when considered against different goals, and the same process that is good for one group of people with a given mission will suck for another group of people with the same mission, because people matter.

I probably expressed that poorly. To put it another, hopefully more correct way - For organizations you can help (there are plenty that are unreceptive to this kind of help) you have to have to start with the culture. Identify those who have true involvement, are willing to risk take, have decision power... and get all of them committed before you can fix the process, which then finally lets you help fix the bugs/tech. Lather, rinse, repeat till organization functions.

In particularly ill organizations, there is no way to separate these items (or the combatant parties :-)) long enough to obtain a fix. You have to wait till the financial realities drive some common sense into the organization. Sadly, for many businesses, this is too late

Re:Kludges are short-time fixes and long-time prob

[PRMan]

DosBox? VMWare?

Good luck

[PPH]

Been there, done that.

If you've got even a small or medium sized enterprise application (whatever that buzz word means) at a larger company (Boeing, for example), it might have its hooks into a dozen or more peer systems/hosts/databases/whatever. They are all 'owned' by different depatments, installed and upgraded over many years. Each on their own schedule and budget. When one group gets the funds together to address their legacy ball of duct tape and rubber bands, they roll the shiney new hardware in and install the spiffy new app. But everyone else is a few years away from affording new systems. And so the inter-system duct tape is simply re-wrapped.

The IT department tried selling everyone on architecture standardization. But due to the gradual pace of system upgrading, the plan was out of date before everyone got caught up to the old one. And today's 'standard' architecture wouldn't play nicely with what was state of the art a few years ago (thanks Microsoft). The whole architecture standard ploy is a salesman's pitch to get management locked into their system. Unless you've got a small enough shop that you can change out everyone's desktop and the entire contents of the server room over a holiday weekend (another salesman's wet dream), it ain't gonna work.

The solution is to bite the bullet and admit that your systems are always going to be duct-taped together. And then make a plan for maintaining the bits of duct tape. There's nothing wrong with some inter-system glue as long as you treat it with the same sort of respect and attention to detail that one would use for the individual applications.

This scales to your stack and apps as well

[Boosch]

DragonWriter sort of hit on this - and I'll leave the whole 'start over' concept to Joel and that older thread that seems to surface every few years. I don't develop software, I run an IT shop. And this issue bites me all the time - band-aiding apps and communications together until you don't know what is driving what.

You see, I'm constantly battling application creep - sometimes it's unavoidable because there really are not many options for some specialized needs. Sometimes I don't get to choose - it was purchased and the first I hear of it is to install. Although I'm morally in the clear to say 'go screw' - it's wasting money in the truest sense and I can limit that by 'getting it to work'. Now we hit on Weigel's post on technical dept. In an application - this might be easier to measure. With infrastructure, it's harder to document these future issues or temporary workarounds in some meaningful manner (open to suggestions). You of course are reminded when they bite you, but faced with time pressures of broken processes, wailing users, and the scramble to fix - the fix doesn't always get the proper attention it really needs. Again, we'll do the 'real' fix next week. (lather, rinse, repeat).

On the hardware front, sure I often get to choose the vendor and product - but it's still gotta perform some needed function - and at times I get to choose from bad, or really bad, or just won't work. I also have both inherited and purchased various servers that were available cheap or free (you keep on using that word - I don't think it means what you think it does), and then deal with odd ways of dealing with things like authentication, unique settings and variables (example = unique ways to identify 'today' or count time), and the growing skill set to maintain them. Again, not by choice or design, but by the reality of larger institutions.

I'd love to see or hear about ways to minimize this - or at least fend it off fairly well. I'm certain I'm not alone - but keep in mind the politics and realities of being in a larger institution.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Just how much documentation can you read?

[hsthompson69]

The problem with the whole idea of "if we only had enough documentation and change control" is that it becomes a non-trivial event to actually read through the documentation. Let's take an imaginary system that's been in production for 5 years...assume every last drib and drab of change has been documented...now you've got a 2000 page document and several hundred change records that tell you *everything*. Except, when it comes right down to it, mastering that 2000 pages of documentation and all the changes made afterwards is a months if not years long project - hardly effective for dealing with production problems that need to be solved in minutes or hours.

The illusion being perpetrated here is that people are interchangeable, and if you just have enough documentation, you can replace Mr. Jones with 20 years of hands on experience with the system with Mr. Vishnu living in Bangalore (or even Mr. Smith in the next cube, for that matter), with a net cost savings.

Now, I'm not saying documentation is a bad thing -> lord knows, it helps to have a knowledge base you can search...but knowing what to search for is knowledge you only get by real world experience with maintaining a production system. This is not digging ditches, boys and girls, this is skilled, if not essentially artistic labor.

Simply put, people matter more than process.

Re:Just how much documentation can you read?

[AchilleTalon]

I agree completely with this comment. I haven't seen yet any in-house documentation database that is valueable at all. Writing good documentation is not an easy task and the guy fixing problems usually just don't know how to write a simple plain non-technical letter. Imagine now he is having to write technical stuff for the next guy knowing nothing about the system or knowing too few about it to be able to ready a few short notes.

Re:Just how much documentation can you read?

[wiz_80]

A hearty "yes, but..." to that comment.

Trying to do process by hand will never work, for exactly the reasons you describe. Any non-trivial process with even a small amount of history is almost incomprehensible for any single human, and the amount of information exchange and handshaking required for a team to develop a hive-mind understanding is also significant.

The only solution is to do process automatically, bypassing the whole bottleneck of trying to get humans to follow the process by hand, and only involve the humans for approvals or for events which fall outside the process. That's where ITIL adoption falters: it's a huge overhead to do by hand, and IT departments are already under water with their current workloads. ITIL advocates can promise cake tomorrow as much as they want, but adoption rarely goes anywhere until the ITIL processes themselves get automated.

As per another sub-thread, many vendors promise this, but only a few deliver. Hint: the vendors who can do it are the ones who aren't scared to show you the tech, either in your own labs or at a real live customer. If they do golf outings or dinners as well, as a way of greasing the skids, that's fine, but IME (and I work for a vendor) the projects where one vendor is selected on a handshake basis are ripe for takeover by another vendor, using a more professional approach, within twelve to eighteen months. On the other hand, if the sales rep does their homework and gets proper support for a technical evaluation, the project is usually a success (barring SNAFUs) and the customer becomes an advocate for buying more $PRODUCT within his or her company.

Re:Just how much documentation can you read?

[Hognoxious]

The problem with the whole idea of "if we only had enough documentation and change control" is that it becomes a non-trivial event to actually read through the documentation.

Been there. The previous guy had left documentation for everything he'd done - thousands of pages - with no structure to it at all. The answer was usually in there, but to actually find it, you had to already know it.

For me, quality beats quantity. However quantity frequently looks more impressive.

Re:Just how much documentation can you read?

[hsthompson69]

My wet dream on this has always been building some sort of internal google for our company. I figure if we can get people to write *anything* down at all, no matter what form, be it a tweet, a blog post, a word doc, or some nasty power point, and if google actually indexed those thousands of pages, maybe, just maybe, we super-geek-techs could use our hax0rz skills at googling when we're doing production support.

Of course, I'm sure that the implementation of a good search engine across every possible document store our company uses is a seriously non-trivial task, but it just might help with the lack of structure problem.

When it comes to documentation, I think a basic rule of thumb is to document what people want, not what you think they need. Writing documentation as a single person effort is usually an exercise in misplaced imagination, but when you have someone *ask* you a specific question, and you answer it, you've actually built "quality" into the documentation creation process. All too often someone asks a dev for documentation, but they can't provide a single real example of a specific question they'd like the documentation to answer. "What is the physical architecture" is a useless question. "Give me a list of all the servers being used by App A, their IP addresses, netmasks and gateways" is a useful question.

The real PITA is when someone tries to come up with a template (because they assume that every project must have the same bits and pieces), and then everyone spends their time trying to explain why section 4.8.1 on mainframe batch programs isn't applicable to PHP apps.

Re:Just how much documentation can you read?

[Scroatzilla]

Regardless of whether people are *actually* interchangeable or not, turnover is a fact of life in the real world. Some forethought regarding how to capture and maintain documentation is crucial, along with the understanding that knowledge is a by-product of everything that you do.

As such, following a knowledge management procedure should be part of everyone's job description. If organized and maintained properly, a knowledge base should present many ways of drilling down into the information, so that not knowing which key word to search for isn't the end of the world.

Why is it that we can find little obscure facts on the Internet so easily, yet can't find significant facts within a finite data set within our respective companies? It seems that people accept the meme that there's no such thing as a good knowledge management system and, therefore, it's not worth spending any time (money) on implementing one. So, people stockpile information in their little wikis, which die because people eventually get sick of maintaining them.

Re:Just how much documentation can you read?

[EvilBudMan]

Did you say artistic or autistic? Autism is more like it.

Re:I was torn between modding this up and commenti

[by (1706743)]

We (Apple)...

...

...has become a diploma mill for Flash game programmers; sadly, I would not hire recent graduates from there...

Sounds about right to me!

Odds of +1 funny over flamebait/troll/offtopic: slim to none. I just hope your 2548 dies before you can mod me down!

Re:Take responsibility and stop the magical thinki

[AK Marc]

The problem, though, is not so much with the products that IT "has to deal with" as with the fact that IT departments either actively choose the penny-wise-but-pount-foolish course of action of applying band-aids rather than dealing with problems properly in the first place, or because -- when the decision is not theirs -- they simply fail to properly advise the units that are making decisions of the cost and consequence of such a short-sighted approach.

I've found the problem to almost always be the last thing listed. It's the contractor syndrome. "If you give me $1,000,000 now, I'll save your $500,000 a year for the rest of the time you'd have used that." Well, they think you are lying. They think that you wouldn't actually save the $500,000 a year, but would take the $1,000,000 this year and add it to your budget as a permanent line item, costing them $1,500,000 a year, rather than saving $500,000.

You can blame the IT director/manager/CIO/whatever for not being convincing enough, but there seems to be a pattern where people bid low then have massive overruns where the highest bidder would have been cheaper. As such, the people the IT person is talking to are often so jaded they don't trust anyone with price estimates.

When IT units don't take responsibility for assuring the quality of the IT infrastructure, surprisingly enough, the IT infrastructure, over time, becomes an unstable house of cards, with the IT unit pointing fingers everywhere else.

And when the IT units have the responsibility, but not the authority to fix things, what then? Most all places tie the hands of IT then complain when the solution isn't perfect.

When the corpus is more bandage than not...

[Anonymous Coward]

You have a mummy. And they have amazing powers. This is also why all software hates Abbot & Costello.

Scrap the old model.

[w0mprat]

Design your user interface as a standards complient web interface. Why is there any need to code a native client to a system at all? A web interface can easily and rapidly be tweaked as browsers evolve, tweaking or rewriting binary applications is much more involved.

house of bandaids n/t

[Anonymous Coward]

n/t

ignoring the SDLC

[WiPEOUT]

While the technology environment can degrade over time due to poor processes, just as important is getting it right in the first place. In my experience, the IT team often lacks any architectural capability when the department is set up, typically because it's "too small" or their needs are "simple". As a result, nobody with the necessary qualifications/experience/mindset really thinks about the business' strategic requirements, they just go for "best practice". Unsurprisingly, "best practice" by itself simply means whatever fad the technology industry is pushing at the time.

Many of the posters above are right in saying that simply throwing it all out isn't the solution, however, throwing away all the assumptions, determining the strategic requirements and then reconciling those with the existing landscape is the way to do this. Sometimes that then results in things being replaced, but sometimes it does not.

Every other business unit does (or should be doing) this during the corporate strategic planning cycle, why does IT not? Sadly, the reason is normally the organisation only really has one "primary" business unit and it excludes IT from the strategic planning altogether. It's IT's job to convince the decision makers that they ought to be partners in strategy, but IT's normal lack of rigour in processes and particularly in financial management that prevents them from having the ammunition to fight for a place at the table.

At the time it was good

[AHuxley]

Seems to be the syndrome. Everybody used Intel, MS and other devices to save cash and swap out over the next few years.
Why buy a huge expensive box that would rot in place when you could buy a cheap unit and swap it out later.
Nobody understood really where tech was going.
They went cheap so when they had better cash flow and market direction they could slot the future in at a fair price.
The good news is someone made the right calls or got consultants who did. They might have got big boxes and it scales or a smaller setup up that was designed right and it can scale too.
If your mission critical the feds or states will bail you out, for the rest, learn, share, name brands that worked or failed and move on.

Microsoft doesn't start over

[drumcat]

For what it's worth, Microsoft outsourced their IT to India. You're asking a question of fountains, and your bosses see extraneous plumbing...

A contrarian opinion

[Anonymous Coward]

This reads like those "articles on investing" written by mutual fund companies. After all, isn't the publisher in the business of selling software and services? If your stuff works, it will continue to work for the same requirements if you don't mess with it. Sometimes there's more risk in starting over: "Hey (insert-born-in-1990's-name here) let's rewrite the airliner's flight control software as a facebook app, it's out-of-date."

A Guy Named DepEnds is not part of the solution

[Anonymous Coward]

Where does one start?

1. Management is old. They grew up with horses, not computers.The times were booming, they were arrogant, and never really learnt what computers were. These old fucks think saying "I don't know" brings about immediate death and dishonor by Klingons. Never back down. Never let them see you sweat. Never say I'm sorry. WEAKNESS!! Bluff and Bluster and Bullshit. Testosterone, Balls not Brains. Well, it has caught up to them. And their portfolios are down 50%

2. Computers should have changed every profession. But the established professions won, and IT pros were never given the status of doctors and lawyers. IT are glorified houseboys that fix the gizmos.

3. Management cannot tell a good programmer from a bad programmer from a superstar.

4. We needed Leonardo Da Vinci, or Newton to be the super programmer, the man who mastered many fields to head up the premier vanguard computer company, to blaze new trails, to go where no one has gone before, to rewrite the book. Instead we got the man child Bill Gates (spraying the ocean into the clouds), The Control/Acid Freak Jobs, and the 'I am CEO, bitch' Zuckerberg. Not good role models.

5. Sales, of course, saw the stupidity of upper management, and has sold them a never ending supply of silver bullets. Go Sales! Good Work. At least part of the world still works.

6. Here we sit, a bunch of old obsolete programmers, generations of wasted Cobol, Fortran, Visual Basic programmers, GENERATIONS wasted so that BIGCOMPUTER could sell another round of snake oil languages. No upgrades, no transitions. Willful, purposeful obsolescence of GENERATIONS of the best talent computing will ever seen, purposely obsoleting the founders.

7. Here we sit, Obama pumping out new regs and laws, that need to be complied with, 1099 form for every purchase over $600. LOL!

8. Here we sit, with piles of obsolete XP gear, OFFICES FULL of obsolete computers so that BIGCOMPUTER can sell another round of upgrades.

9. Here we sit, the WITH NO CAPITAL to upgrade with, and no where to borrow it from.

10. Here we sit, no fresh round of cheaper and cheaper and cheaper programmers anywhere on the planet

11. Here we sit, with the economy in the crapper, the very currency we get paid with threatening to disappear faster than privacy on Facebook

12. Management wants disposable, interchangable cogs. Management HATES craftsmen. How long have we been called RESOURCES?

13. The disposable people religion and outsourcing has all organizations gutted of institutional knowledge and continuity.

14. The emerging world and upstarts like google are inside the OODA loop of the establishment. Fear. There is nothing deadly than an old man, losing his grip.

15. No succession planning. No new blood. No grooming. The oldest of the boomers and the silent generations still run things. Their cold dead hands will need to be broken from the wheel. Where is Bill Gates stable of new hot shot new superstars? No where. Because these old fucks can not and will not share the stage.

16. Corruption. The last 15 years have been built on corruption, gambling and Ponzi schemes. If you are under 30, all you have every known is the bubble world. Why produce value, when you can leverage and gamble? Rape and Pillage?

17. IP laws. Lack of R and D. Nobody wants to spend money on innovation. And who let the lawyers in? You know, lawyers. The guys that still keep ALL THEIR FILES ON PAPER? Which profession speaks gobbley gook and hides behind confusing langauge again?

18. Time has sped up. Shit is changing outside the walled garden and the barbarians don't care about your delusions or your status quo.

19. External Threats. The developing nations buy more oil and cars than the G17 countries. Yep.

20. Documentation. There is none. And the primaries and principals and the founders are gone.

What are we going to get? Some dufus with a cool handle, a diamond encrusted ipod and blog with groovy graphics, born with a silver s

Try maintaining code from the 1960s.

[apdts]

The summary thinks code for PIIs is old. HA! My employer is still running programs where the base was written in the late 1960s. IBM mainframe assembler code. It has had dozens of developers with differing styles (and skill levels) working on it over the decades. It's still being patched when there's no way around it, but it's no surprise that result of all those years is unmaintainable. IS management is afraid of replacing it due to how fragile it is and they think it works although I find issues with the results it generates on a fairly regular basis. How much sense does that make?

Re:I was torn between modding this up and commenti

[k8to]

Interesting. My career goal is to do quality work maintaining and improving existing product lines. It's hard to find these jobs!

Re:I was torn between modding this up and commenti

[zaffir]

It seems to me like the people that are interested in the new sexy, are not going to have the experience to work on something like the mainline kernel anyways. Need someone to work on the Safari app on the iPhone? I imagine a new CS grad that's slightly above average could fit the bill. But hardcore kernel work? It's not like you're going to take on all willing applicants for something like that.

To me, the barrier to entry seems much higher for at least your kernel example. Is that the case?

Re:I was torn between modding this up and commenti

[lonecrow]

I am shocked! Shocked I tell you. Apple applicants are only attracted to shiny new things? And all along I thought it was just the customers.

Re:I was torn between modding this up and commenti

[gillbates]

Everyone wants to work on the latest and greatest stuff, no one wants to maintain or even release patches.

I don't really know how to address this, except by the people who think they are going to be the next great video game designer remaining unemployed.

Here's how you address it: you hire one of those 9 out of 10 CS graduates who "Just got in it for the money". Had you offices in the Midwest, you'd have no problem finding programmers whose only ambition is to crunch out brain-dead code until they can move into management. Trust me, I work with these people and they're even worse than the primadonnas interested only in the "cool" things. Naturally, not everyone can be the next game programmer, or work on cool things, but you probably don't want to hire those whose only ambition is to do the grunt work.

Typically, the primadonna has to have his ego coaxed into doing the grunt work. But you can usually count on him to do it fast, and not to make a total mess of things. Granted, some people have a higher estimation of their abilities than their peers. But at least someone passionate about coding can be inspired to improve their code; they'll actually accept coding standards once reasonably explained. But here's a short list of problems with the typical "career type":

1. Because they don't have the intelligence or the initiative to do things right, they'll happily plod along, even when the given design can't possibly work, or can't be delivered on time. And when it does fail, rather than trying to understand *why* it failed, or *what* they could do differently next time, they blame their coworkers/subordinates, etc.
2. They are more sensitive to the political implications than the technical ramifications of their decisions. Consequently, they'll often run with an inefficient, or sometimes even incorrect design so as to placate their superiors. And once again, the blame always lies with *someone else*.
3. And speaking of blame, they'll frequently blame others when things go wrong, and even sometimes when they don't. There are *certain people* at the office around whom I can't have a technical discussion with coworkers because they understand neither about what we are talking, nor that such conversations are a normal part of the job. I've actually been reprimanded for discussing architectural decisions, because "we've already decided on the architecture..." Which is great, but the fact that you've decided doesn't help me understand it better. Supposedly, we're all mind readers here, and no discussion is necessary.
4. The career types usually promise unrealistic deadlines, and write horribly unmaintainable code. After all, writing code is just a stepping stone into management, and maintaining that code will soon become *someone else's* problem, not theirs...
5. And perhaps the worst part is that they have a corrosive effect on teamwork and morale. With a politician in the office, *no one* wants to do the grunt work out of fear that it will adversely affect their career.

It's easier to convince a rock-star programmer that documentation is necessary than it is to convince the career-track political programmer that a race condition is a problem, that architecture matters, that maintainability and scalability are important. Just the other day, I had a department manager question the value of writing reusable code - in fact, he was so hostile as to suggest that it wasn't worth our time to make code reusable... (And not only that, but reported to my boss that my suggestion otherwise was "distracting to what we're trying to accomplish here"...)

I know the starry-eyed programmers can be a handful at times, but those indifferent to technical issues will lay a minefield in your company. Suddenly, years after they've moved on, you'll find your new hires telling you the projects they built aren't worth salvaging, that you'll have to start over, etc... I've seen these types move into management and turn an otherwise fun profession into a death march. You don't want the stupid, or the political, types of people writing code. They'll set your company up for failure every time.

Re:I was torn between modding this up and commenti

[Eli Gottlieb]

Funny thing, I'm graduating next year and *like* doing kernel work... My actual track at school is Programming Languages & Compilers, but I've known how to do low-level stuff since high school. Would you happen to have a contract email address you can send me?

Comment removed

[account_deleted]

Comment removed based on user account deletion

Comment removed

[account_deleted]

Comment removed based on user account deletion

Comment removed

[account_deleted]

Comment removed based on user account deletion

Obsolesence

[Anonymous Coward]

There is non-trivial value in IT systems and supporting hardware and frameworks that can be expected to continue to operate for many many years perhaps as much as several decades without significant change.

This concept is contrary to the views of many "modern" software and hardware manufacturers who have a vested interest in continuing maintenance and support and therefore have no reason to invest heavily in testing as whatever bugs their customers report will be fixed next week by the next compile'n ship patch.

By letting the inmates run the asylum -- failing to insist on systems designed from the core to be maintainable, expandable and operate in a sane and correct manner you as an IT manager are only encouraging crappy products and outcomes from your vendors.

Call me a heretic but I don't see software technology improving much over the past two decades. There are a lot more choices and more opportunties to aggregate the works of others to your advantage saving you time and money but very little has changed in terms of underlying core design principals and fundemental understanding of the nature of the space. There are no meaningful design automation systems for general purpose applications.

Re:Obsolesence

[drsmithy]

Call me a heretic but I don't see software technology improving much over the past two decades. There are a lot more choices and more opportunties to aggregate the works of others to your advantage saving you time and money but very little has changed in terms of underlying core design principals and fundemental understanding of the nature of the space.

This is like arguing vehicles haven't improved in a century because the "underlying core design principals and fundemental understanding of the nature of the space" haven't either. It's dumb.

Re:I was torn between modding this up and commenti

[Animats]

Mainline kernel people are difficult to hire...

Well, what do you expect? Apple's kernel is a warmed over Next kernel, which is a warmed over Mach kernel, which is a warmed over BSD kernel. There's probably legacy code in there from the 1980s.

Re:I was torn between modding this up and commenti

[Anonymous Coward]

You need to find locate people from other vendors that work on the parts of their UNIX operating system that you want resources in. I do support for commands and libraries (libc and most non-networking commands) identifying problems to line of source.

Your problem is that most people who are happy doing that sort of thing usually don't go out actively searching for jobs since they're happy doing it where they currently work.

Re:Kludges are short-time fixes and long-time prob

[Corbets]

The IT department didn't pick the app, and isn't allowed to not support it. They can't switch to another app (usually apps like this have little or no competition, and they're probably locked-in anyway).

Which is, in and of itself, not a problem. IT should not be selecting the software used, because IT doesn't understand what the business needs. IT should give feedback on the technical details in a business-understandable fashion (i.e., the risk of data loss with this product is high, because the producer has a history of not releasing updates when problems are discovered), but it's the business' decision.

However, a strong procurement department also needs to come into play here. When software is purchased, a contract should be drawn up with the producer, safeguarding the rights of the purchasing organization and - if necessary - providing legal venues to enforce them. In this way, you can minimize the impact of hurry-to-release applications by forcing the producer to fix them when you discover problems.

Unfortunately, that's a much easier task for multinationals than it is for SMEs.

Re:It's good that windows 64 kills the old 16 bit

[mlts]

With virtualization, the 16 bit applications may end up remaining with us indefinitely. I have a relative who is still using an insanely backlevel version of WordPerfect, and most likely when she gets a new computer, I will end up putting that version of WordPerfect in an XP Compatibility Mode, or a VMWare Workstation Unity session for her, so even though the program still can only understand 8.3 filenames, it still will interoperate with a 64 bit OS.

Another business I have consulted for has an old program that they used to use that would dial their bank via a modem, get changes, then put them in some sort of spreadsheet. This is now still living in a Windows 3.1 virtual machine so they have access to the old records at anytime.

Obsolescent by design

[SplashMyBandit]

It is interesting that many decision makers choose solutions that become obsolescent by design. By this, I mean that some products are deliberately deprecated and made incompatible with later versions by their developer in order to force the purchaser is made to re-buy what is essentially the same functionality over and over again. This behavior makes less sense when there are often usable alternatives that don't suffer the same vendor deprecation. This applies to software/development stacks, operating systems, document formats etc.

And this is why we need job mobility in IT ...

[golodh]

The problem with IT is its binary nature: it either works (as in does what it needs to do) or it doesn't. There is nothing that to squeak, groan, give off heat, smoke or wear out (except perhaps its personnel). That definitely makes maintenance, and especially designs to minimize or avoid maintenance a low-priority item.

We're all familiar with a particular philosophy of IT management: "Just fix it!". We do, after perhaps noting that this may cause difficulties later on. The answer we get depends on the amount of politeness and insistence we exercise in delivering that message. To cut a long story short: if we're very polite we'll get a pat on the head and a managerial comment along the lines of: "We'll deal with it later on then."; if we're neutral we'll get a comment like: "Just do it, Ok?"; if we're less than polite we get fired (probably after we apply the fix).

In the mean time our manager (no slouch himself) has figured out the optimum time he wants to remain in his particular job. That typically means: he want to get out before the effects of decisions *he* made will become apparent. He isn't worried about correcting poor judgments from his predecessors (as long as it can be done without a fuss: those predecessors just might have moved higher up the ladder and might make things difficult for *him*), the only thing he's worried about is his own record. And that will be closed the instant he leaves.

The typical mistake made by IT personnel is to stay too long in the same job. Don't make the mistake of believing this helps your manager. Sure, it's often a god-sent for a manager to have someone who actually knows what's going on. The downside however is that it plays hob with his possibility to request more staff to do the work and a larger budget for expensive boondoggles because there's always Joe Smith who knows the system so well he can fix it. It makes him look good for keeping things under control, but it restricts his scope for self-aggrandizement. It's also de-motivating for IT personnel to have to clean up the mess you were forced to create (under the firm and slightly condescending guidance of a new manager who has deal with the problems his predecessors caused) all the while being prevented from spending 50% more time and effort "doing it right" than "just fixing it". It's also damaging to your career prospects: you're supposed to *know* the systems you built (even if you never were allotted the time to document them, let alone to update that documentation), and because you do (and because those systems are of course critical) there is no incentive for promoting you or allowing you to move on to a different area.

In buoyant economic times the answer is simple: switch employer every 3-5 years or so. You will face new challenges (read clean up the mess left by others and their managers), but at least you're cleaning up a mess you didn't make (you get to see fresh goofiness and you can learn new things from that), and you can't be held responsible for the way it is. Take a queue from your manager and try to stick to about the same time in your job as he did (plus maybe 1-2 years but not so long as to have to fix the next big crash). In the current economic climate that strategy is of course in jeopardy. Nothing for it then: simply install as much Open Source software as will escape the eye of your manager (because it will usually work better, it's often better documented, it offers more scope to fix things, you can take that particular skill with you, and it's usually a lot more fun).

Barrier to entry

[tlambert]

Need someone to work on the Safari app on the iPhone? I imagine a new CS grad that's slightly above average could fit the bill.

Not really. Safari on the iPhone is memory constrained, and can't tolerate memory leaks or other issues; there's very little headroom there. It's also really performance sensitive, since the code efficiency is directly proportional to battery life. If you are working on something that's baseline software on the iPhone, it has tighter requirements than third party applications.

That said, there are a number of universities that we've loaned employees as faculty to teach iPhone/iPad development; those people might be more likely to get a job working on something like that, but it's generally not an entry level position (all of the position postings, with the exception of the Pittsburgh office, that I've seen require 2+ years of experience, but admittedly, I didn't look at all 368 of the open req's for the iPhone).

There's a kernel position that doesn't require experience, if you have a masters: position #4636787.

We also have internships in various areas, which would give you a foot in the door: position #4727175 (covers a wide range of internships, including iPhone and other CoreOS positions).

-- Terry

Re:Take responsibility and stop the magical thinki

[Anonymous Coward]

My boss recently told me to "stop playing with that Linux thing and do some real work" - oblivious to the fact that it's been running on our production servers for more than five years.... So what I told my boss -- get this --- "tell this to Google too" -- didn't still get it!
   

Contact info

[tlambert]

I'm easily findable, but I'd suggest doing an internship this year. We generally make offers to interns who impress us. NB: Apple internships are paid, and there's a living stipend on top of that.

http://www.apple.com/jobs/us/students.html

(This also shows the currently open new grad jobs).

-- Terry

Re:Contact info

[Eli Gottlieb]

Either this year or last year I actually submitted a resume and application for an Apple internship... never heard back. I'd still put one in, except that I already have work for the summer: an REU and a Summer of Code project.

Missing the truth as usual

[Whuffo]

Those big shops that get talked about here are not what you describe them as. I spent seven years at one of the biggest and - well, I wish I had those years back. The first thing you need to keep in mind is that the budget doesn't include new code or new hardware and actually it's going to require a headcount reduction and the rest of you guys can just pick up the slack.

Maybe you'll get to spend some time fixing things - but you're more likely to be sucked into some deathmarch project where you don't have enough time to complete it much less debug or document it. Those that come after point at these things and call them sloppy work and say how it could be better if we'd just documented what we'd done. Those fools will have their eyes opened very soon.

Let me explain some basic facts to you: IT is a cost center; it adds nothing to the bottom line. When management is looking to cut costs, it's always among the first targets. And if you do the job well and things work reliably - they won't have any reason to believe you're necessary. First rule of working in large IT shops: keep your resume updated. After enough lay-offs, most of the remaining staff will quit. Then the company coasts along for a while until things start falling apart then hires a bunch more IT people at a higher rate to make things work again. And the cycle repeats, over and over.

Those who blame management for their woes are unaware of how much BS their manager puts up with and how much he tries to keep the executives from tearing the department apart. And after so many years I can only come up with one solution for the whole mess: make IT a chargeable service and charge the various departments / desks for the IT services they request. Try suggesting this at your company and see what they say - they'll say that nobody would request IT services under these conditions. OK, so now we know what the rest of the company feels that IT is worth. I'm sure glad to be far away from that dysfunctional clusterfuk.

Re:I was torn between modding this up and commenti

[Helen O'Boyle]

To me, kernel and other generally-invisible platform internals *are* the sexy parts, because they require serious geek skill, and often a combination of both software AND hardware know-how to code around hardware bugs, meet perf targets, etc. If these parts don't work, that Flash game is going to have a hell of a time impressing anyone.

You're comparison is invalid...

[brufleth]

A house of cards has to be carefully thought out and assembled. Your comparison is invalid.

Never Cry Wolf

[nicholdraper]

Please, have you seen the Walt Disney movie "Never Cry Wolf?" If your IT integration skills lead to a mountain of we'll say junk for the sake of politeness, then maybe its your own skills. I worked at a place where a group of programmers constantly complained that they weren't given the resources to do good work. I was working on firmware and their group was responsible for the Windows software part of the project. Finally our boss came to me and asked me to take over their project. So they were let go and I took over their responsibilities. Good grief, they were coding in C++ and I had given them some simple C code to integrate. I had expected them to take my small c file and create an object out of it. Instead they had cut and pasted the guts of my small code into methods in C++ objects that had grown to thousands of lines. It took me a couple of years, but I managed to refactor all their code and port the system to Java and I added automated testing. I was able to improve the reliability from about a six percent failure rate to less than a one percent failure rate. (Our customers told us that our competitors products ran at about an eight percent failure rate.) If you find yourself with kludge after kludge and go and see the movie "Never Cry Wolf" and then spend less money on Moose Head beer and more on software training.

Re:I was torn between modding this up and commenti

[TheRaven64]

There is - some of it's very crufty. Some of the newer parts are quite well designed though. The scheduler is quite well designed. The newer VM stuff has a nice design but is in a serious need of tuning. Some of the concurrency stuff needs a complete rewrite - acquiring synchronisation primitives is painful, the new 'amazingly fast' locking that they use for GCD is marginally better than a FreeBSD mutex, and between one and three orders of magnitude (depending on load) faster than a Darwin mutex. Part of this is a userspace problem (not optimising for the uncontended case, which is the most common in good code), but a lot of it comes from the route down through the myriad kernel layers when sleeping a thread.

It is a viscious cycle

[jav1231]

We literally bought a piece of hardware that by the time it left the shelf, the project implemented, and everything was brought up online we had a brand new piece of hardware that the vendor had EOL'd. It's not just a software problem. As long as hardware changes are ever accelerating, the tendency to "make software fit" is going to be a problem. Everyone wants to make "new" instead of "work."

More Like A Mummy

[b4upoo]

When it is all bandages and no meat left inside - the mummy is like over corrected software.

Re:I was torn between modding this up and commenti

[ckaminski]

Dear Apple Insider:

I am a slogger, I am a fixer of bugs, an exterminator of dastardly demons. I don't have the attention span for a 3 month feature development project, or new shiny sexy. I don't care. But I'll spend 6 months tracking down an esoteric timing bug in a application, and slaughter it.

My 5 year plan is, and always has been, to be a fixer of problems, not a creator of new ones, and I have a 15 year track record of proving it.

Re:I was torn between modding this up and commenti

[Animats]

Some of the concurrency stuff needs a complete rewrite - acquiring synchronization primitives is painful, the new 'amazingly fast' locking that they use for GCD is marginally better than a FreeBSD mutex, and between one and three orders of magnitude (depending on load) faster than a Darwin mutex. Part of this is a userspace problem (not optimising for the uncontended case, which is the most common in good code), but a lot of it comes from the route down through the myriad kernel layers when sleeping a thread.

That problem in Mach is part of what gave microkernels a bad name. QNX, which is a real microkernel (about 65K of code) does thread dispatching, locking, and message passing very fast, in constant time, and without long interrupt lockouts. Those are the functions which must go fast in a microkernel, because they're used so much. In QNX, locking a mutex in the uncontested case is about three instructions in-line, with no system call. Those three functions are most of what the QNX kernel really does. In Mach, they were an afterthought, written on top of BSD.

This really belongs in the "when is it time to rewrite" thread.

Re:I was torn between modding this up and commenti

[Hognoxious]

My 5 year plan is, and always has been, to be a fixer of problems, not a creator of new ones, and I have a 15 year track record of proving it.

[PHB] Do all your projects overrun by 300%? [/PHB]

The internet is useful because google made it so.

[hsthompson69]

Why is it that we can find little obscure facts on the Internet so easily, yet can't find significant facts within a finite data set within our respective companies

I think I answered this in another comment, but the short answer is, we need google inside the corporate firewall. If you have a viable search engine, then it doesn't matter how knowledge is captured and maintained -> lord knows google hasn't forced anyone to adopt certain documentation best practices, they just slurp everything in and spit out relevant results. Having a good search engine can help mitigate the whole "knowledge base du jour" syndrome (although possibly replace it with a "search engine du jour" syndrome).

Anyway, I do understand that an internal search engine is not a trivial task, but I'd rather let each project team have their own little wikis, as long as I can go to one page, put in a search term, and get results from all of them.

Re:Take responsibility and stop the magical thinki

[sjames]

There's a lot to that. No matter how many rules and procedures are in place, it can't replace a bunch of people doing "the right thing" at any given point. If managers could REALLY write procedures with enough detail to get the job done, they would be programmers and if people could actually follow procedures to the degree necessary (and not resign out of unbearable tedium) they would be computers. If you want to see the truth of that, give a manager a book on C and tell them to write a CRM system. You'll notice how the "code" they write will have too many built-in assumptions to even be pseudocode. It certainly won't compile and run.

Try to break things down too much into a bureaucracy and the people can no longer know what "the right thing" might be and even if they can see far enough to get an idea they won't have enough autonomy to actually implement "the right thing".

Re:Kludges are short-time fixes and long-time prob

[EvilBudMan]

Oh, I'm getting it Microsoft apps (sarcasm)....hahaha

Re:I was torn between modding this up and commenti

[EvilBudMan]

Designing a simple game for a cell phone that might sell for a couple bucks being a winner may also be 1000 to 1 but at least it doesn't take an army to do.

Re:Kludges are short-time fixes and long-time prob

[turbidostato]

"Which is, in and of itself, not a problem. IT should not be selecting the software used, because IT doesn't understand what the business needs."

Neither should the bussiness units because bussiness units don't understand the grand IT scheme where everything should fit.

But that's no news... OK, I'll rewrite: that *shouldn't* be news since that's "team playing 101".

"Unfortunately, that's a much easier task for multinationals than it is for SMEs."

Not because SMEs 'per se' since on small companies is *way* easier for everybody to know everybody and for informal comunication channels to stablish worthy thourough information nets. The problem is that on SMEs the "cult to the personality" and the "you won't tell me how to drive my bussiness" is even stronger than in big corps.

Re:Take responsibility and stop the magical thinki

[EvilBudMan]

When all else fails use a hammer.

Re:Take responsibility and stop the magical thinki

[EvilBudMan]

--And when the IT units have the responsibility, but not the authority to fix things, what then? Most all places tie the hands of IT then complain when the solution isn't perfect.--

Apparently, this is the norm. Get ready to be self sufficient because this will not last long.

This is a feature of being a cost

[Colin Smith]

To be blunt, most IT departments act like cost centers and don't provide any strategic value.

IT is treated as a cost in the accounts, they are cost centers. The drive is always to reduce costs, so IT services are centralised, standardised, outsourced and the interfaces between IT professionals and the business, reduced to the "help desk". There is simply no way that the IT dept can effectively know what "value" is with respect to the business units.

It's just the way accounting practice is set up. Accounts are grouped and centralised to make costs visible => IT becomes an industry standard cliche. It might make a difference if the IT funding was centrally calculated, then simply added to the business unit budgets in an appropriate proportion to do with as they pleased.
 

You're looking for the Google Search Appliance

[kcbnac]

http://www.google.com/enterprise/search/gsa.html

Or the 'mini' (Up to 300,000 documents)
http://www.google.com/enterprise/search/mini.html

Here's the relevant wiki page:
http://en.wikipedia.org/wiki/Google_Search_Appliance

Google Search Appliance

[kcbnac]

I just linked another person to this in this thread:

http://it.slashdot.org/comments.pl?sid=1663346&cid=32362936