Slashdot Mirror
The Cult of DevOps
packetrat writes "I was at OmniTI's Surge conference today, which turned out to be, among other things, a meeting of the cult of DevOps. Ars Technica covered the keynote and some of the presentations, but some of the best stuff is in the comments. Google CIO Ben Fried told the tale of a really poorly engineered trading application at Morgan Stanley that he was associated with, and how the way IT was structured there contributed to that engineering and to its spectacular failure, costing the bank untold millions in stock trade processing fees from its institutional customers. He said what he learned from cleaning up the mess has informed how Google runs its IT operations, and a culture that promotes generalist skills. A lot of how he describes Google's approach sounds like the DevOps kool-aid a lot of the other speakers were serving, but it also sounds like common sense — are most IT organizations really that poorly run that developers are totally unaware their software is sending messages that are generating network storms, or network engineers are clueless enough about QoS to route leased lines into their data center through their public-facing Internet?"
Was that a rhetorical question or do you really have to ask?
I'm not sure I can take anybody who calls an attempt to make IT and Development more aware of each other a cult, seriously.
The traditional way of doing things didn't work for 30 years. Why is it that when people are trying to make (and apparently making) a difference to how companies work, they're regularly denigrated by a large subset of the very people whose working lives they're trying to improve.
Haters gonna hate, I guess.
are most IT organizations really that poorly run that developers are totally unaware their software is sending messages that are generating network storms
How is it the IT organizations fault the developers are unaware of the effect of their software? Is it the IT departments responsibility to debug software?
Having to work for a living is the root of all evil.
A while ago I thought most OSS application and framework projects including such Gnome and KDE are in trouble, due to the large use of the fumble-around development approach. Also known as the first code then think approach. All the great model-based, model-driven and agile development methods seam to be far away from the way many OSS projects are developed.
However, lately I came out of my ivory tower and stood eye in eye with experts from the industry who largely believe that they are professionals and really do great stuff. They also use the fumbling approach. The main difference is the call it agile. Even though it is far away from such an approach. I always thought that one of the problems of our new economy company back in the 2000 originated from being too deep in code and too light on design, planning and documentation. But it looks like, that tinkering is a more widespread way of software development. So I guess that leaves me with bad management (which was not my responsibility).
Most of these tinkering approaches originate in the absence of developer discipline and the "Add this quick"-management method. but I am telling nothing new. We all know for decades now what is wrong in software development. A lot of people wrote books on patterns (design and otherwise), but in the end if no one follows these patterns the problems remain.
The "Add this quick"-management originates at large by a misconception of IT and its importance for businesses and other organizations.
The answer to that question is "yes". In my experience what is it uncommon is to find a truly competent IT organization. And normaly the biggest it is the worse the problems are. But, what could you expect in an age where your IT group is composed with the cheapest guys you could hire with no experienced (and more costly) ones?. Or what about the clueless IT managers with little or no experience on IT?. They can't plan against what they don't know. Yeah, as a colleague told me not long ago, IT life is nowadays like war: long periods of boredom punctuated by moments of sheer terror.
From what I've seen (I work as sysadmin), that most of the problems are caused by the rotation of people, companies are not willing to give value to experience, and always want to make a new bet, switching contractors, giving emplooyees management positions and letting total noobs do all the jobs that actually matter (but is not visible?).
As soon as someone has the right skills and internal knowledge, he deserves a better wage, but only a tiny fraction actually make it there, it has been posted here, rotation is the only way to improve salary, and people are willing to pay twice as much for contractors than for employees
I'm positive, don't belive me look at my karma
I've been around enough to know that this 'new' 'DevOps' philosophy is just the way it always has been done at many successful companies making extensive use of technology.
I have come to associate the phrase 'enterprise IT' with those who *don't* work that way, and make their lives needlessly complicated. Of course, every last party to the mess will generally recognize it and know what could hypothetically be done about it, but only bitch in private about it and rarely ever push for meaningful change. The reason is simple, so long as it is a complicated mess, it requires a great deal of human care and feeding, meaning job security. Management can't force things to change without huge risk as everyone has sufficiently entrenched themselves.
XML is like violence. If it doesn't solve the problem, use more.
That last was a rhetorical question, right? You could have stopped at "Are most IT organizations really that poorly run". Yes. Upwards of 95% of CIOs are completely unqualified to hold their position. They are grossly overpaid talking heads with NO real understanding of the technical underpinnings.
Seriously. The single largest problem that most IT organizations have is that the people running them are completely unqualified to execute the duties required. It is not that IT workers are overpaid. Good IT workers are worth their pay several times over. The problem is that far too many are overpaid for their actual skill and knowledge level.
I like to compare current IT workers to the medical personnel of the 1700s. Many of them had some extremely rudimentary knowledge of how a body worked, but much of what many of them believed was rubbish and they had no idea about the depths of their ignorance. They have good intentions, but they are going to milk you for every penny they can and many honestly believe they deserve the pay.
Finally, those at the "top" of the food chain, highest in management, are also most likely to be the most unqualified. They rose to their position through personal connections and snake-oil patter.
Until IT becomes a disciplined hard science with the same kind of internal reviews and requirements as the medical field, you are going to have an awful lot of hedge-witches, snake-oil salesmen, and field medics running around causing more harm than good.
Yeah, that's exactly what we need. More rules, more books describing how to do *doing* something. Meta-meta-meta-everything...
And more companies that take a methodology which has quite sensible premises and transform it to a paper-pushing-based freak-child.
...developers are totally unaware...
You're supposed to sniff the kool-aid, not drink it FYI.
The real issue is the word developer. Nowadays that can mean anyone.
We need special notations like developer+1 (extra dice) or developer^5 (power of five developers)... also because the limited ANSI character set must be conquered in the face of mathematical oppression. Yeha!
These problems are, by far, much worse when the software is developed via outsourcing, and the worst when this outsourcing is done offshore.
There's absolutely no incentive for a programmer in Bangalore or Pune or Mumbai to give a damn about software he wrote that's running in NYC or London, operated by people at some other company. There's no incentive for him to even bother to develop the software correctly. He won't have to deal with the system when it fails. He won't have to deal with the angry administrators or users. He won't have to deal with the costs the faulty software will result in.
In fact, it's in this offshore programmer's best interest to not even learn how to develop software properly in the first place. After all, that'd take some small degree of effort.
At least in-house American or British developers have to deal with the people affected by their mistakes. They have to see their co-workers suffer. They have to hear the upper management complain. It hurts their pride and sense of professionalism when such mistakes are made. These factors combine to form great incentive to learn how to do the job properly, and then to actually go ahead and do it properly. While American- or European-developed software isn't perfect, anyone who has worked with it will know just how much better it always is than the crap developed in third-world nations.
Whenever you see someone dismissing what you think is a genuine concern, it's not because they don't get it. It's because they don't care. Everyone has their own priorities. Dev's priority is functionality. Security is just a necessary burden. But no one prioritizes burdens. Managing priorities is a management job. So if you see a system in which important priorities are not given weight, that's just poor management. Management's job is identifying priorities and then creating an incentive structure (I don't mean salary, I mean day-to-day incentives) which emphasizes the important and de-emphasizing the unimportant. Btw, that doesn't mean that management gets to ask everyone "what are your priorities?" and pretend they have done their part. Asking that question is tantamount to shifting the burden of identifying the priorities to those who don't have that responsibility.
Any guest worker system is indistinguishable from indentured servitude.
This cancerous attitude happens in any project, across any set of roles, once things get big enough. If the organizational culture is rigidly defined roles and CYA, this crap is inevitable.
There's no silver bullet for this obviously, this is a pure management problem.
I am little concerned about Fried and his conclusions. He ran all these specialized silos without an Architect(s)?? Of course if was cluster fuck, that is the dumbest thing I ever heard of. And yet this guy kept his job at the Bank and then got a job at Google.
Here is how you are suppose to do things, ass hat. Each silo has a Architect/Manager who oversees the generalist that run the day to day ops in that silo. These silos would be Voice, Network, AppDev, Server/Storage, whatever works for you. Implement PMO and Change Control. Then Once a day/week all the Architects get together to decide any changes to anything. All items are discussed and any Architect is allowed to ask questions and raise concerns. Think knights of the round table.
Professionally, & for "enterprise-class" business development solutions done in Client-Server designs (or more complex ones using Terminal Server/Citrix, or going "cross-platform" to larger systems (think "Big Iron" midrange/mainframe class stuff thru middlewares)) & the "general overall breakdown" (hopefully not missing too many parts, I had to think about it) of who does what & when:
BUSINESS/SYSTEM ANALYST RESPONSIBILITIES: Know the projects' data & processing requirements for input and what needs to be done for outputs, Where the pertinent data resides (if it needs conversions or batching, or if it can be done via pure SQL straight out of a database etc.) AND, who needs to get to it, & when (sometimes for security, this is necessary too). Use cases come into play here to go to mgt. after speaking to end users &/or dept. heads also (use cases usually go to mgt.).
DBA RESPONSIBILITIES: Make sure the data is accessible (or filtered off in a "view" instead, only showing the pertinent material but not all (possibly confidential info., etc.)) & setup stored procedures IF the coders are not allowed to, many are though (better than SQLExec calls directly from code both in terms of security AND network traffic). Sometimes, this means creating views, reindexing (create &/or drop) indexes, tables, & a lot more... This is one of the crucial FOUNDATION parts really (because if the overall schema of it is bad, it can NEVER be a good efficient system imo).
SOFTWARE ENGINEER/ARCHITECT/LEAD DEVELOPER RESPONSIBILITIES: In combination with the DBA + System/Business Analysts inputs? This is the guy that makes the overall design of the package "go" - They're the guys that make "the foundations" work & create the overall design architecture in data AND CODE for it to work (coding goes along with the ride here too). Use cases come into play here also as an "aid to design". Delegation of workload & responsibilities to subordinates & schedules for deadline come into play for they too here.
PROGRAMMER/ANALYST RESPONSIBILITIES: Write the code that the System Analyst, Software Engineer, + DBA outline to you as to process & data requirements, that makes the "show run" from the front end (generally) so users can utilize it for business planning &/or decision making.
PROJECT LEADS &/or MANAGEMENT RESPONSIBILITIES: Make sure all goes above as planned, on schedule, & not over-budget (and to "run interference" for dev. teams/DBA's, with the end users/customers). They also make decisions based on use cases & other constraints, & sign off on the whole show to go into motion (or, to "kill it" too).
---
Then, you have a "bug testing/killing" cycle, via a Q/A dept. (IF it exists, many times it does not) & regression testing...
OR
You let a select group of primary "end users" beat up on it for a week or two in a testbed environs, offering suggestions for changes they need (be they useability issues, or actual problems in data results (the most important part is the latter)).
FINALLY - THIS IS WHERE "IT" COMES IN:
They have to do the hardware & networking requirements, initial deployment (once they're informed on what rights etc. users need (if need be, making new usergroups for easier mgt.)) &, that the data is safe/secured (& backed up), AND, where analysis of network segment packet traffic can be an "issue" (which is part of HOW/WHY stored procedures help a LOT vs. "packet storms") because here, they can determine if a network segment(s) needs to be "busted up" more into more manageable & monitorable parts (especially in the case of network congestion), OR if query designs need improvements (vs. packet storms as the article calls them) - this last part alone can STOP having to use QoS as a "bandaid on a bulletwound" in fact, many times... they also do the maintenance later for users too.
* So - Once a larger program "passes muster" for those things
Trust me: all IT shops are mismanaged and all for the purpose of making a buck or appeasing some fat fuck and his or her wallet / purse... The IT market in America is a complete mess--much like everything else at the moment--but the good thing is that it really is a sleeping giant. The capabilities in this country has staggering amounts of potential, and as soon as it makes its way through the constant barrage of red tape and time-wasting policies, it will be unstoppable... Unfortunately for right now at least, it's going through that phase of maturity where the Baby Boomers--those that know nothing about computers--are holding all the pivotal positions responsible for the very mismanagement of things. Once they die-off, people of more modernized ilks will take over and re-factor processes to leverage this dormant cache I'm eluding to... That's when things will change. Mark my word...
They see things as a triangle: developers, Q&A and IT, and in the middle DevOps.
That's not at all how it works from my experience.
While developers and Q&A are very close, developers are not particularly close to IT at all, while Q&A is.
Testing is what requires large infrastructure set-ups. Development doesn't really: the Q&A guy directly gives you access to a working configuration that demonstrates the problem. As a software developer I never have to interact with IT.
... are most IT organizations really that poorly run[?]
Well, yes. This is news to you?
The biggest problem I saw at my workplace, was lack of vision from the direction. They don't have clear goals set so they don't have a strategy to get there and they don't have tactics to wins the battles over the different road-block. They wait until they are hold against the wall to make a random decision, they don't listen to the feedback from the talented professional they hired, they kill home-grown project that are on budget and on time only to buy some monstrosity from commercial "partners" and they allocate more resource to failed project of little strategic importance...
I worked hard to change the mentality of my devs, and I succeeded. I single-handedly brought our dev team from outdated and badly followed 1990 practices to 2010 practices that are followed automatically as almost the enforcement is automated, we have gain a lot's of velocity but we are now idle as they cannot plan as fast as we implement, since they as I said they lack vision.
Devs need to drink more of the DevQA cool-aid too.
In an attempt to answer the barely articulated question buried amidst the run-on editorializing, I will answer "yes". Many otherwise extremely talented developers carry little understanding of how their code performs once exposed to chaotic and nondeterministic production environments. Likewise many ops engineers hold little appreciation for the motivations of and the pressures on those developers.
Much of this isolation stems from culture, but at least as much is from experience. Being paged at 3am because an entire pool of servers simultaneously ran out of file handles or because some unknown engineer failed to properly sanitize their input tends to temper ones soul. Having your deployments constantly stalled because one failed to meet some ill-defined barely document seemingly nebulous operational requirement too tends to galvanize ones opinion about "those people". Both teams scream, "they just don't get it," I lot. And they are both right.
The result if these tentions are unaddressed is clear. Many live system outages and stymied projects.
Devops, cult or not, is a conscious effort to bridge these frequently opposing forces inside technical organizations. The teetering platform of fast, cheap, and good need not be out of balance. Rather it is the express job of the devops engineer, by having equal interests both in execution and in sustainability, to create a world where deployment happens freely and fast, where quality is not sacrificed, and where we are not mindlessly throwing money at the inefficiencies created by conflict.
The kool-aid is actually a pretty tasty beverage. You might take a sip.
A gig I had at one of the Big 5 Banks, devs had no access to production as part of the security policy. In at least one case the result was a SysAdmin's (whom had previously worked for Morgan Stanely) solution which allowed for unlogged redirection of pages (also a violation of the security policy).
That same gig one of their "outstanding successes" was .Net app (not in source control) that needed to be rebooted every 45 minutes to keep from crashing. It was quietly rewritten and retired within 6 months of launch.
The wiki page on "devops" sums up part of the idea to involve "deep cross-departmental integration"
While there is a lot of agile practises around Google, including "stand-ups", iterations, scrum, planning poker, and whatever, there is hardly any "cross-departmental integration". Communication is as poor as at any other huge behemoth. Just like we've read about multiple internal teams fighting each other at Microsoft and Nokia, so do many teams at Google create similar products, both externally and internally.
In fact, there are multiple projects and initiatives to cut down on duplicated projects. Think about that one for a second.
the software industry in silicon valley by and large does not have a role 'architect of product'. we have architects of technology. the product comes together as various teams: tech pubs, qa, tech support, professional services, it, ops, manufacturing, marketing -- all contribute their efforts to take the technology delivered by the typical development team and transform it into a product. DevOps is one attempt at mitigating or filling this gap to varying degrees depending on the folks involved. Now what is interesting is, "where does DevOps" report into? CIO? is so, where does Development report into? and now we are back to ownership...
Wish I had mod points handy.
The separate department structure for computer systems encourages everyone to take the approach of 'Not My Problem' even when they could fix it. From a broader perspective, the barriers among Dev/QA/Ops are artificial: it is all computer people doing computer-ish stuff. External constraints are vastly simpler than the construction analogy -- no opcode shipments stuck in Customs, no delays from rain or freezing temperatures. If the computer industry can not overcome these lesser barriers there is little hope of handling bigger barriers such as Marketing|Development.
And they really need to be overcome. Industry has spent a truly astonishing amount of money on 'computer systems' with frequently negative results. Recall the number of anecdotal reports of large computer system efforts that are abandoned after YEARS of work for many millions of dollars. Although the computer system company has a 'loss' of anticipated future revenue from the project, the host company loses real money and irreplaceable time. Further, they are often left with a confused and discouraged work force that will affect them for years to come.
Bent, folded, spindled, and mutilated.
Historically a sysadmin has been able to and does write, fix software, and administering large scale systems has always meant templating and then deploying from updated templates... which means having a build system which can build to a template usually automatically (it's boring) which apparently is now called continuous integration. You also need an infrastructure designed to easily deploy changes. VM or physical is almost totally irrelevant, VMs make life a little easier.
Lets see... Infrastructures.org for example was definitely there mid nineties (and still is, cool.). That's almost a whole generation ago, how old are you?
I have no idea what makes people think this devops stuff is new. Is it just the fact that there are now thousands of newbies trying to make names rediscovering good practices? Is it just that we now have more large scale systems, or is it that we now tend to have highly distributed vs centralised systems?
And it's not development. It's engineering. The difference is maths.
Go on, get off my lawn!
Deleted
Yeah, I'm a longtime sysadmin, and yeah, I'm against this whole DevOps thing. Why? Because over the years I've learned that while developers can write some awesome code, they're generally shit when it comes to managing production systems. They leave development tools scattered about the system. They install Linux web servers with X11 enabled. They solve problems by writing code first, rather than analysing the problem properly. Don't get me started about the large Drupal installation I just overhauled, where the DevOps team went and coded the crap out of everything, rather than using the tools that Drupal and the Drupal devs provide.
In short: No DevOps are allowed anywhere near any of my systems.
I think it's more of a case of developers having to now do a lot of sys admin work, not wanting to do it, not having the expertise to do it, but trying to make out that they can 'develop' things away and it will all be easy. Small companies have this problem, and small IT companies like to believe they can just hire in sys admins and farm crap out to places like Heroku and Engine Yard.
In my last job I pointed out to our developer when the development work dried up, as it does, that our clients paid for their applications to be kept running and live and it was the sys admin they were paying for. He was most upset.
I'm guessing in the 90s, it was pretty rare to actually use those practices. A new shop would have one server called "server". Then it would get a few, named after the Seven Dwarfs. Eventually, you would get to the point of needing to get a new sysadmin who could handle large-scale deployments (or have the old sysadmin change the way they worked), but that would be around the post-IPO stage (if ever), not the "three guys, one of whom is figuring out how to script EC2" stage.
By scripts.
Add to that that devOps is much, much harder than system engineering. A DevOps engineer needs to know enough about multiple programming languages *and* algorithmics *and* data and telecom infrastructure to tweak the programs that are really controlling the network. There aren't many people like this at the moment (so if you want to move as a programmer to a $200k/year job, understanding the packet datapath and it's controlling plane is where it's at, provided you're good at optimizing algorithms. The first company to make a real control plane that merges control of the application inside the host and control of the network above the host, controllable by an api is going to make a bundle).
Most sysadmins and netadmins these days know a bit of scripting, and have lots of knowledge over specific details of the OS/switches/routers/firewalls, like the access subsystems or networks and firewalls. But almost none of them could tell you exactly what happens, in code, when ssh refuses or accepts a connection, and what access that implies for all parties involved. I've yet to meet the first sysadmin that can tell me how to optimize a non-trivial problem. An example devOps issue that is requested is to increase that access (e.g. we control the firewall, and we would like it to log ssh connections. Can you tweak the ssh daemon with as little interference on the client side as possible. And then use that to log all access to production systems, in other words : execute a man-in-the-middle attack on the firewall to get full accountability of the production network).
But this is child's play compared to what the devOps guys really want. They want an API they can use for problems like "we have a network with nodes N, links L(n1, n2, capacity, cost function), peering links P, and data transmission requests from the applications R(app1, app2, dataSize, profit, constraints). Configure the applications and network so that cost is minimized, profit is maximized and all specified constraints are followed (e.g. "maximum latency for video : 2s, max latency for voip 50msec, ...). Oh and do tell us which improvements to the network would bring the most benefit in terms of $ per increase in maximum bandwidth transmitted."
If you have a company whose profit is mostly determined by the capacity of it's network, any advantage you have in this is going to pay out hugely. For a netflix, amazon, bing, microsoft's online office and other apps, google, yahoo, any online advertising company, any online voip company, ... having a functioning algorithm like this means a vast advantage over competitors.
The big issue is that it's much easier for a capable programmer to become a devOps engineer than it is for operational people to become devOps engineers. Not that we have any kind of surplus of capable programmers at all, but I doubt that even if they have no choice many sysadmins will become good programmers.
So the issue is, this is about automating network/system operations the way ford automated car production : if you work in operations, one of 3 things is going to happen :
1) you move to development/engineering, and spend most of your time writing and debugging programs
2) you become a robot, a call center employee, the interface between a computer system and external parties, in other words, your job is reduced to "you want fries with that ?" level
3) you're fired (which it will be for a lot of people)
For the vast majority of people in operations it will mean a gradual move to option 2, and after a few years option 3 if you didn't manage to get recruited for engineering ... Of course this does mean that if you do succeed it will mean an increase in salary, potentially a big one. But most people won't succeed (which is of course the reason for the salary increase in the first place).
And that is of course the big issue. The IT workforce is going to drop massively if these guys succeed, and consist entirely of call-center employee
I do realize some businesses have a completely seperate IT department that is responsible basically for hardware and only hardware, and some other group that develops/sets up the software required to run the business. The developers then are expected and end up treating the actual servers, the storage, the network, as "black boxes" that the stuff just runs on. This has several immediate problems:
1) If the software has peculiar timing requirements, or gets put under heavy enough load, this whole "black box" thing falls apart, as either CPU cycles, perhaps RAM, storage speed or network speed becomes inadequate for the job. These may not fail gracefully, and diagnosis can become difficult when the develoeprs and IT are so isolated from each other.
2) Various other fails, I've heard of a few examples (on The Register) where some place loved virtualization, they'd implement a primary VM and a failover, but nobody bothered to realize they were BOTH running on the same physical machine! Or, something makes "backups" but they are actually just sitting on the same storage array.
Honeslty, if some project is going to be large enough to need at least one dedicated machine, then those who develop it should have carte blanche over what kind of hardware it is running on. If they wish to call this DevOps and pretend it's a new thing, then so be it. I can say, when I worked at a university surplus department, we kept university IT far away from the systems we ran our internal use software on.
Moo00o0oo0o000o
It can be summed up like that: If the problem will arise in an area he isn't responsible for, and if implementing it that way regardless is cheaper, faster or less problematic for his department, he will implement it that way.
Running the traffic through the internet instead of the LAN? Why not? It's not going to be the development's problem (since that's operation's problem), and it's faster to do it that way, so it's done that way. A user interface that makes users beg to break the programmer's fingers and make him a consultant? Not the programmer's problem, it's the user's problem, and if he can implement it faster with a shabby interface, it will be done that way.
The problem isn't that the people wouldn't know that these problems will surface. Often they know that quite well. The problem is that this is not a criterion, while the time they spend on solving the problem is. Hence, whatever lets him get it faster off his table will be the way it is done.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
...and I'll even try to remember to check back in case people respond.
Assumption: the trading application being critiqued is the same one that was there when I was an IT consultant at Morgan Stanley. I left in August 2000.
I know the application well. It was developed by a department headed up by Vinny (whose name is withheld because...I'm senile and don't remember it). I worked in the department that wrote the messaging infrastructure that was used by every application on the sell side of the firm.
If the application is the same one then the mere fact that it was still in use when Mr. Fried left is a testament to the application's effectiveness. Would they do it better now if they were writing it from scratch? Of course. But Morgan Stanley has 3,000-4,000 IT staff in its ranks so they could easily do so if the application were as bad as he says. And the messaging infrastructure...well, I have no love for the original authors (no hard feelings Steve and Arthur...I'm lying of course), but that subsystem was extremely robust, predated TIBCO and Talarian, and provided more functionality that those two products until after they were on the market for several years.
cults are stupid.
if you want to name yourself something stupid, i don't care. Stupid people do stupid shit because they are stupid.
Be seeing you...
I don't want to be an asshole.
But if you don't adapt, you might loose your job. Especially in IT.
Is that something people are surprised about ?
New things are always on the horizon
Be less of an asshole instantly- learn to spell "lose".
Lots of people are going to lose their jobs, because only the better ones will be retained. Managing systems manually is a skill that will go the way of car building skills. That's a lot of people.
And IT is just about the only sector that is supposed to be adding jobs to the economy. It will not go down smoothly once this starts happening.
Sorry, English is not my first language
New things are always on the horizon
It could also be that we end up being a lot more effective. That is after all the whole point of IT. I've never made much sence to me why people did the 'manual labor' in IT.
But if I look at education in my home country: the Netherlands in Europe, I can tell you there aren't even enough people studing programming to fulfill the available positions (well, I don't know the real numbers, but that is what they keep telling us in the local IT-press and looking at the people that get hired I'm not all that surprised).
So maybe there is room for people to be retrained.
I don't know, we'll see.
New things are always on the horizon