I'll second the parent's story with my own experience at one of/.'s favourite multi-national computer companies.
It used to be (pre-2001) that IT support was handled by our own local people. You'd walk over to their office, tell them the problem and they'd fix it. Then they decided to centralize this. So our local IT office only did UNIX stuff. All the Windows and networking stuff was sent to a central office, still in the US, run by reasonably competent people (2002-2004). It was a little slower, they weren't down the hall, but they still (eventually) got stuff fixed. Now, enter the third round of cost-cutting: they moved all networking support to Brazil (I work in the US), and all Windows support to Bangalore, India. None of this would be bad (relative to remote US people) if they actually hired competent people there. Apparently though. in order to maximize their "savings" they hired the very cheapest "IT" people in Brazil and India, in other words, people who have very little training and know nearly nothing.
Bottom line: it took me 2 MONTHS! to get a bad ethernet port fixed, despite daily phone calls and frantic appeals to management (response: "we all need to go through the right channels for support"). The amount of my time lost during those 2 months was probably 2/3, working from home was the only thing that really worked. My net cost to the company is $100/hour. This "cost saving" philosophy is short-sighted and idiotic beyond belief. Recall this is at the largest IT/hardware/software company in the world.
So my point is simply that hiring technically incompetent people to do "IT support", even when paying for lost time at a fairly high rate, is absolutely not limited to Best Buy. It is the result of extremely bad and short-sighted management decisions, made by managers who themselves are (technically) clue-disabled, and only understand "the bottom line" and powerpoint presentations.
Feel free to share this with anyone. I got many of these nuggets from other people anyway, so any claim of originality on my part is immediately suspect.
One other tiny "motherhood and apple pie" good practice I thought of is that some (most?) compilers have flags you can turn on to generate various runtime checks (eg: http://www.nersc.gov/vendor_docs/ibm/vac/compiler/ ref/ruoptchk.htm ). They usually aren't all that useful, but on the other hand they are "free", they can be used at least during nightly regression-testing without any extra effort.
Documentation can certainly help... a lot, both in the form of formal documentation and perhaps even more importantly through extensive in-line code comments. Documentation should describe the features, functional pieces, how they interact, perhaps what some of the important data sturctures are, what each function does, what the return codes are, etc...
A rather useful approach is to require of the authors specially-formatted in-line comments for every function or feature implementation, *at the time/place the relevant code is written*. These formatted comments can then be (frequently) auto-extracted by a script to generate the formal docs, while still being in the code too. One can generate user-docs and programmer-docs seperately via this method.
Somewhat along the same lines of author responsibility, write an automatic regression-test system, tied into your code control system (eg: SVN, CVS), with a requirement that for each new feature that is written, the authors write a regression test and supply "golden data" for that test. If it fails the nightly regression test, have the script find who committed code since it last passed, and make them fix it or even roll back their changes. Require that everyone run some amount of regression tests before committing code.
Other things can also help to reduce the complexity: minimizing interaction between functional code chunks and having simple function interfaces. Functions should be small, but not too small (thus spreading the work over too many fcts and layers of fcts). Funtions should have well-defined, minimalistic behaviour. Avoid fancy syntax or language constructs as much as makes sense; everything else being equal: simple==good, sophisticated==bad. These minimize side-effects and speed-up isolating the inevitable bugs.
There are also other good practices such as always returning useful return codes and *checking them*, liberally using assertions or exceptions, and coding "paranoid" (assume variables can have bad values and check for them). Code so you catch bugs/errors early and often. (eg: Initialize pointers to NULL: seg-faults are a beautiful thing compared to erroneous results reported after-the-fact by a customer, which affected his design, which went to manufacturing and then failed QA.)
Descriptive variable and function names are very useful.
Duplication of code (eg: cut-and-pasting code, maybe making a small change) is almost always bad and should be well-justified if ever done.
Then there are things like static and dynamic code-checkers (lint, flexelint and valgrind, purify) to catch bugs and memory issues. They can be integrated with your regression testing system.
The above fall under the broad umbrella of "coding for debugablility". That is, assume that most of the time spent on this code (over the next 20 years for example) will be spent figuring out bugs or "undocumented features", and try to make that process as easy as possible.
All of the above take a fair amount of time, not that they shouldn't be done, but in a resource-constrained system, which most people work in, they often fall by the wayside. (Try telling your manager "I spent a month writing better comments in the code", or "this took an extra week because we had to clean up the code", never has worked so well for me.)
The core problem is that after all of this goodness is done, in significant programs you still have a lot of complextity, due to the complexity of the underlying problem and the many parts which justifiably (and combinatorially) interact with and depend on each other. Think of something like the simulation of a large complex physical system or a modern OS kernel, even a modular and well-designed one. You still have many, many arbitrary decisions about what each piece is, how everything fits together, how things affect each other, about the methods/algorithms you choose, their weaknesses and strengths, about mathematical interactions between the different parts. Problems and thus programs can pract
In addition to some of the suggestions made so far I would add a good automatic regression-test system which runs every night, and reports problems (build failures or result diffs). I've made mine so they "find the guilty" (whoever committed code since the last good regresstion test).
I recently put together a list of Fundamental Coding Truths after musing about this topic and why it was so hard to plan software development:
1. Software is not at its core a collection of a few clever algorithms.
Rather, it is primarily (in the ways that matter) a huge collection of arbitrary
choices and random implementation details.
The algorithms that consititute the mathematical/logical basis of a piece of software
are an important, but very small (eg: 1%) and relatively very simple part
of the overall code.
2. Code complexity is pretty much exclusively determined by the (combinatorial)
number of interactions between pieces. Each interaction requires at least one decision
and usually many more.
3. Because of #1 and #2, deep, intimate familiarity with the code (this vast
collection of implementation details) is only ever fully knowable to the original
author(s) who made these uncountably many, mostly arbitrary decisions.
(Familiarity by secondary authors/maintainers comes primarily from
re-writting sections of code.)
4. Because of #3, programmers are not interchangeable. The efficiency with
which a person can navigate the code, implement or even imagine changes
is almost entirely determined by how familiar they are with these many, many small
details. The ratio of efficiencies between a primary author and another
equally talented coder is very large (eg: 100). Because of this, the original
authors of a section of code are usually the only ones who are ever able to efficiently
modify or restructure it. This becomes rapidly more true as the the size and
complexity of the code increases.
5. Because of the complexity of code (the number of interactions and interdependencies),
debugging and maintenance constitutes the vast majority (eg: 99.9%) of the work
required by a piece of software over its life.
6. Because of the complexity of code (number of interactions between components), it is
very hard, if not impossible, to predict with any accuracy what will be involved in implementing
a given change. Even for original authors, unintented side effects are almost inevitable, and
the primary determinant of the length and difficulty of a task lies in finding and rectifying
unintented consequences or unforseen interactions. Because of this, the uncertainty in the time
it will take to execute a change is very large.
(eg: 10x range in 95% confidence limit of time estimate, say 1 day-2weeks).
7. Because of the complexity of code, bugs are an inevitable byproduct of writing code. It is hard
to predict how long it will take to find and repair bugs as that depends on how many side effects are
involved, which is not known until the repair is done and "fully tested". The only way to avoid bugs
completely is to not write code. There are things that can minimize bugs or speed up finding/fixing
them, but they will always exist.
In June 2005 of this year, the median sale price of pre-existing homes was:
739K$ in Westchester County, NY
419K$ in Putnam County, NY
356K$ in Dutchess County, NY
500K$ seemed like a good average.
(Source: http://www.nysar.com/pdfs/monthmedian.pdf)
I'm a libertatian at heart and I really don't mind as long as these companies are at least not given tax breaks to offshore work. Companies should be allowed to do stupid things, and I should be allowed not to buy from them. As long as the gov't isn't using my money to encourage corporate stupidity.
Personally I think this outsourcing thing is a bit of a corporate fad which will, if not disappear at least become more rational as companies realize that it isn't as *universally* a good idea as it might seem at first blush.
The true costs of outsourcing are often higher than the financial savings, which in turn are often less than they seem at first.
My brother-in-law is responsible for helping set up IT centers for Bank of America in India and now China, and he emphatically agrees. The communication issues are huge, and largely because of that, you often don't get the kind/quality of of product out that you do with domestic IT groups. It is *not* because the foreign groups are less technically competent, educated, smart or industrious, but the intangible aspects of getting a team to be effectively productive make it a lot less effective than it seems on paper.
Also, since saving money is a prime motivator for going overseeas, they often try to save money over there by getting the cheapest people (least trained/educated) available abroad. It's not like they go and recruit the top 10% of IIT's graduating class.
I've always thought that locating suport-desk or other 'low-level' jobs made a lot more sense for a variety of reasons: same language/easier communication, values/social norms, time-zones, transportation.
On a related issue, I've always wondered why large companies didn't locate programing work centers near large universities in cheaper cities.
I graduated about 10 years ago from one such large midwestern university and discovered that "all" (most) the good jobs were in the Bay Area or the Boston-NY-DC megalopolis. I can program from the dark side of the moon, and even now most of the people I work with are remote. Yet I have to work in the NY burbs where anything with 4 walls and a roof costs 500K USD. I would much have prefered to live in some mid-sized midwestern city (with good amenties, schools, university), near my family, etc... They could have paid me 1/3 less and I would still come out ahead financially. It costs about 2 times more to live here than in my university town, which means I have to earn 3 times more (gross) to reach parity. My company even has a branch sales office there, just none of the programming/engineering jobs. Those midwestern universites crank out a LOT of grads many of which would like to stay in the region.
... is why this kind of thing didn't happen sooner and doesn't happen more often. Ok, those of us south of the [USA/Canada] border are stuck with Bush till 2008 and given the political/financial strength of the right in the US, it is likely that the trend of passing laws restricting the rights of the individual and favouring those of large corporations will continue. Fine. So WE are screwed for a while. Why does the rest of the world seem to feel this overwhelming urge to go along with this BS? If United States wishes to act like a pariah state, great, treat it like one. I like the idea of Canada closing the border. Just because we elected a bone-head doesn't mean the whole world has to put up with the stupid laws he supports. (yes, and Clinton passed some bad ones too!). What if the rest of the world started rejecting visa applications from the US? What if they started giving the US bad "ratings" for our labour protection laws?
I am an american but grew up in Europe. The educational system I was in is (or was at least) very specialized.
Despite a number of flaws, it worked very well, because the people who wanted to study hard were tracked into challenging, competitive/selective, and demanding curicula. We had mostly technical classes (lots of math, physics, chemistry, engineering), but also some foreign languages, history, geography. litterature.
The most important aspect of my section was that everyone took it vey seriously (students, professors), and while we had fun too, we worked really hard. I guess the main point is that because it was competitive/selective, we *wanted* to be there, those that didn't... left. Having gone on to get a PhD in Physics and an MS in CS (uiuc.edu), I can still honestly say that I worked harder in HS than at any other time in my life... and loved it. We viewed ourselves as professionals and for the most part enjoyed the experience (math/science really can be lots of fun!).
This is probably not for everyone. Some people, for a variety of reasons, are uninterested (or unable) to pursue that kind of education (by which I mean serious, intense and academic, not only scientific), which is fine. Those people should be provided useful ways to pursue some other kind of education, be it a "less intense/focused" track or a "tradesman" track. We need poets and airplane mechanics too. Those are no less valid choices or careers.
My best friend was one such person who, while not in the least bit dumb, was not interested in lots of math/physics. He chose to go into a metallurgy track, and came out at 18 being a highly-skilled metallurgist/machinist. His was hardly a wasted HS experience, quite the opposite. He knew things about metal and could do things with it that were quite amazing. He knew substantially more about metals than I did as a "science/engineering" student.
If I recall correctly, in both the US and there, about 50% of students actually graduate with an "academic" degree. In the US though, that degree represents a lot less actual usable knowledge, and the other 50% are simply left to twist in the wind. Think about the cost of discarding (or under-utilizing) the potential of half of your population.
So I guess my point is that tracking is good, if done right: 1) Give some general-ed classes to everyone, but allow them to pursue their interests. 2) Make it as demanding/competitive as possible, 3) Don't toss out people who don't have an interest in purely academic pursuits, or can't make it there, they deserve a good education also.
Slashdot Mirror
I'll second the parent's story with my own experience at one of /.'s favourite multi-national computer companies.
It used to be (pre-2001) that IT support was handled by our own local people. You'd walk over to their office, tell them the problem and they'd fix it. Then they decided to centralize this. So our local IT office only did UNIX stuff. All the Windows and networking stuff was sent to a central office, still in the US, run by reasonably competent people (2002-2004). It was a little slower, they weren't down the hall, but they still (eventually) got stuff fixed. Now, enter the third round of cost-cutting: they moved all networking support to Brazil (I work in the US), and all Windows support to Bangalore, India. None of this would be bad (relative to remote US people) if they actually hired competent people there. Apparently though. in order to maximize their "savings" they hired the very cheapest "IT" people in Brazil and India, in other words, people who have very little training and know nearly nothing.
Bottom line: it took me 2 MONTHS! to get a bad ethernet port fixed, despite daily phone calls and frantic appeals to management (response: "we all need to go through the right channels for support"). The amount of my time lost during those 2 months was probably 2/3, working from home was the only thing that really worked. My net cost to the company is $100/hour. This "cost saving" philosophy is short-sighted and idiotic beyond belief. Recall this is at the largest IT/hardware/software company in the world.
So my point is simply that hiring technically incompetent people to do "IT support", even when paying for lost time at a fairly high rate, is absolutely not limited to Best Buy. It is the result of extremely bad and short-sighted management decisions, made by managers who themselves are (technically) clue-disabled, and only understand "the bottom line" and powerpoint presentations.
God help us all.
One other tiny "motherhood and apple pie" good practice I thought of is that some (most?) compilers have flags you can turn on to generate various runtime checks (eg: http://www.nersc.gov/vendor_docs/ibm/vac/compiler/ ref/ruoptchk.htm ). They usually aren't all that useful, but on the other hand they are "free", they can be used at least during nightly regression-testing without any extra effort.
Documentation can certainly help... a lot, both in the form of formal documentation and perhaps even more importantly through extensive in-line code comments. Documentation should describe the features, functional pieces, how they interact, perhaps what some of the important data sturctures are, what each function does, what the return codes are, etc...
A rather useful approach is to require of the authors specially-formatted in-line comments for every function or feature implementation, *at the time/place the relevant code is written*. These formatted comments can then be (frequently) auto-extracted by a script to generate the formal docs, while still being in the code too. One can generate user-docs and programmer-docs seperately via this method.
Somewhat along the same lines of author responsibility, write an automatic regression-test system, tied into your code control system (eg: SVN, CVS), with a requirement that for each new feature that is written, the authors write a regression test and supply "golden data" for that test. If it fails the nightly regression test, have the script find who committed code since it last passed, and make them fix it or even roll back their changes. Require that everyone run some amount of regression tests before committing code.
Other things can also help to reduce the complexity: minimizing interaction between functional code chunks and having simple function interfaces. Functions should be small, but not too small (thus spreading the work over too many fcts and layers of fcts). Funtions should have well-defined, minimalistic behaviour. Avoid fancy syntax or language constructs as much as makes sense; everything else being equal: simple==good, sophisticated==bad. These minimize side-effects and speed-up isolating the inevitable bugs.
There are also other good practices such as always returning useful return codes and *checking them*, liberally using assertions or exceptions, and coding "paranoid" (assume variables can have bad values and check for them). Code so you catch bugs/errors early and often. (eg: Initialize pointers to NULL: seg-faults are a beautiful thing compared to erroneous results reported after-the-fact by a customer, which affected his design, which went to manufacturing and then failed QA.)
Descriptive variable and function names are very useful.
Duplication of code (eg: cut-and-pasting code, maybe making a small change) is almost always bad and should be well-justified if ever done.
Then there are things like static and dynamic code-checkers (lint, flexelint and valgrind, purify) to catch bugs and memory issues. They can be integrated with your regression testing system.
The above fall under the broad umbrella of "coding for debugablility". That is, assume that most of the time spent on this code (over the next 20 years for example) will be spent figuring out bugs or "undocumented features", and try to make that process as easy as possible.
All of the above take a fair amount of time, not that they shouldn't be done, but in a resource-constrained system, which most people work in, they often fall by the wayside. (Try telling your manager "I spent a month writing better comments in the code", or "this took an extra week because we had to clean up the code", never has worked so well for me.)
The core problem is that after all of this goodness is done, in significant programs you still have a lot of complextity, due to the complexity of the underlying problem and the many parts which justifiably (and combinatorially) interact with and depend on each other. Think of something like the simulation of a large complex physical system or a modern OS kernel, even a modular and well-designed one. You still have many, many arbitrary decisions about what each piece is, how everything fits together, how things affect each other, about the methods/algorithms you choose, their weaknesses and strengths, about mathematical interactions between the different parts. Problems and thus programs can pract
In addition to some of the suggestions made so far I would add a good automatic regression-test system which runs every night, and reports problems (build failures or result diffs). I've made mine so they "find the guilty" (whoever committed code since the last good regresstion test).
I recently put together a list of Fundamental Coding Truths after musing about this topic and why it was so hard to plan software development:
1. Software is not at its core a collection of a few clever algorithms.
Rather, it is primarily (in the ways that matter) a huge collection of arbitrary
choices and random implementation details.
The algorithms that consititute the mathematical/logical basis of a piece of software
are an important, but very small (eg: 1%) and relatively very simple part
of the overall code.
2. Code complexity is pretty much exclusively determined by the (combinatorial)
number of interactions between pieces. Each interaction requires at least one decision
and usually many more.
3. Because of #1 and #2, deep, intimate familiarity with the code (this vast
collection of implementation details) is only ever fully knowable to the original
author(s) who made these uncountably many, mostly arbitrary decisions.
(Familiarity by secondary authors/maintainers comes primarily from
re-writting sections of code.)
4. Because of #3, programmers are not interchangeable. The efficiency with
which a person can navigate the code, implement or even imagine changes
is almost entirely determined by how familiar they are with these many, many small
details. The ratio of efficiencies between a primary author and another
equally talented coder is very large (eg: 100). Because of this, the original
authors of a section of code are usually the only ones who are ever able to efficiently
modify or restructure it. This becomes rapidly more true as the the size and
complexity of the code increases.
5. Because of the complexity of code (the number of interactions and interdependencies),
debugging and maintenance constitutes the vast majority (eg: 99.9%) of the work
required by a piece of software over its life.
6. Because of the complexity of code (number of interactions between components), it is
very hard, if not impossible, to predict with any accuracy what will be involved in implementing
a given change. Even for original authors, unintented side effects are almost inevitable, and
the primary determinant of the length and difficulty of a task lies in finding and rectifying
unintented consequences or unforseen interactions. Because of this, the uncertainty in the time
it will take to execute a change is very large.
(eg: 10x range in 95% confidence limit of time estimate, say 1 day-2weeks).
7. Because of the complexity of code, bugs are an inevitable byproduct of writing code. It is hard
to predict how long it will take to find and repair bugs as that depends on how many side effects are
involved, which is not known until the repair is done and "fully tested". The only way to avoid bugs
completely is to not write code. There are things that can minimize bugs or speed up finding/fixing
them, but they will always exist.
In June 2005 of this year, the median sale price of pre-existing homes was: 739K$ in Westchester County, NY 419K$ in Putnam County, NY 356K$ in Dutchess County, NY 500K$ seemed like a good average. (Source: http://www.nysar.com/pdfs/monthmedian.pdf)
I'm a libertatian at heart and I really don't mind as long as these companies are at least not given tax breaks to offshore work. Companies should be allowed to do stupid things, and I should be allowed not to buy from them. As long as the gov't isn't using my money to encourage corporate stupidity.
Personally I think this outsourcing thing is a bit of a corporate fad which will, if not disappear at least become more rational as companies realize that it isn't as *universally* a good idea as it might seem at first blush.
The true costs of outsourcing are often higher than the financial savings, which in turn are often less than they seem at first.
My brother-in-law is responsible for helping set up IT centers for Bank of America in India and now China, and he emphatically agrees. The communication issues are huge, and largely because of that, you often don't get the kind/quality of of product out that you do with domestic IT groups. It is *not* because the foreign groups are less technically competent, educated, smart or industrious, but the intangible aspects of getting a team to be effectively productive make it a lot less effective than it seems on paper.
Also, since saving money is a prime motivator for going overseeas, they often try to save money over there by getting the cheapest people (least trained/educated) available abroad. It's not like they go and recruit the top 10% of IIT's graduating class.
I've always thought that locating suport-desk or other 'low-level' jobs made a lot more sense for a variety of reasons: same language/easier communication, values/social norms, time-zones, transportation.
On a related issue, I've always wondered why large companies didn't locate programing work centers near large universities in cheaper cities.
I graduated about 10 years ago from one such large midwestern university and discovered that "all" (most) the good jobs were in the Bay Area or the Boston-NY-DC megalopolis. I can program from the dark side of the moon, and even now most of the people I work with are remote. Yet I have to work in the NY burbs where anything with 4 walls and a roof costs 500K USD. I would much have prefered to live in some mid-sized midwestern city (with good amenties, schools, university), near my family, etc... They could have paid me 1/3 less and I would still come out ahead financially. It costs about 2 times more to live here than in my university town, which means I have to earn 3 times more (gross) to reach parity. My company even has a branch sales office there, just none of the programming/engineering jobs. Those midwestern universites crank out a LOT of grads many of which would like to stay in the region.
... is why this kind of thing didn't happen sooner and doesn't happen more often. Ok, those of us south of the [USA/Canada] border are stuck with Bush till 2008 and given the political/financial strength of the right in the US, it is likely that the trend of passing laws restricting the rights of the individual and favouring those of large corporations will continue. Fine. So WE are screwed for a while. Why does the rest of the world seem to feel this overwhelming urge to go along with this BS? If United States wishes to act like a pariah state, great, treat it like one. I like the idea of Canada closing the border. Just because we elected a bone-head doesn't mean the whole world has to put up with the stupid laws he supports. (yes, and Clinton passed some bad ones too!). What if the rest of the world started rejecting visa applications from the US? What if they started giving the US bad "ratings" for our labour protection laws?
I am an american but grew up in Europe. The educational system I was in is (or was at least) very specialized.
Despite a number of flaws, it worked very well, because the people who wanted to study hard were tracked into challenging, competitive/selective, and demanding curicula. We had mostly technical classes (lots of math, physics, chemistry, engineering), but also some foreign languages, history, geography. litterature.
The most important aspect of my section was that everyone took it vey seriously (students, professors), and while we had fun too, we worked really hard. I guess the main point is that because it was competitive/selective, we *wanted* to be there, those that didn't... left. Having gone on to get a PhD in Physics and an MS in CS (uiuc.edu), I can still honestly say that I worked harder in HS than at any other time in my life... and loved it. We viewed ourselves as professionals and for the most part enjoyed the experience (math/science really can be lots of fun!).
This is probably not for everyone. Some people, for a variety of reasons, are uninterested (or unable) to pursue that kind of education (by which I mean serious, intense and academic, not only scientific), which is fine. Those people should be provided useful ways to pursue some other kind of education, be it a "less intense/focused" track or a "tradesman" track. We need poets and airplane mechanics too. Those are no less valid choices or careers.
My best friend was one such person who, while not in the least bit dumb, was not interested in lots of math/physics. He chose to go into a metallurgy track, and came out at 18 being a highly-skilled metallurgist/machinist. His was hardly a wasted HS experience, quite the opposite. He knew things about metal and could do things with it that were quite amazing. He knew substantially more about metals than I did as a "science/engineering" student.
If I recall correctly, in both the US and there, about 50% of students actually graduate with an "academic" degree. In the US though, that degree represents a lot less actual usable knowledge, and the other 50% are simply left to twist in the wind. Think about the cost of discarding (or under-utilizing) the potential of half of your population.
So I guess my point is that tracking is good, if done right:
1) Give some general-ed classes to everyone, but allow them to pursue their interests.
2) Make it as demanding/competitive as possible,
3) Don't toss out people who don't have an interest in purely academic pursuits, or can't make it there, they deserve a good education also.