Slashdot Mirror
App Developers Spend Too Much Time Debugging Errors in Production Systems (betanews.com)
According to a new study, 43 percent of app developers spend between 10 and 25 percent of their time debugging application errors discovered in production. BetaNews adds: The survey carried out by ClusterHQ found that a quarter of respondents report encountering bugs discovered in production one or more times per week. Respondents were also asked to identify the most common causes of bugs. These were, inability to fully recreate production environments in testing (33 percent), interdependence on external systems that makes integration testing difficult (27 percent) and testing against unrealistic data before moving into production (26 percent). When asked to identify the environment in which bugs are most costly to fix, 62 percent selected production as the most expensive stage of app development to fix errors, followed by development (18 percent), staging (seven percent), QA (seven percent) and testing (six percent).
This reads like an ad for software I sell.
Pick any enterprise CI/CD/Automated test vendor. Good job marketing.
I am curious what "too much time" is (hint it's different depending on what your software DOES). Please validate that statement. We make the same statement, but we're selling you software.
I found that app developers are spending too much time filling out surveys rather than doing quality design work that results in less bugs, or addressing the bugs proper.
Hey, here's an idea: Build programs from the start that don't have bugs? Slow down your production process so you have time to catch them? Do without the fancies so there are less vulnerability points? Or just don't spend time fixing bugs! Make the person buy the next version of the software!
43 percent of app developers spend between 10 and 25 percent of their time debugging application errors discovered in production
That seems like an odd metric, but it doesn't surprise me. Production support has always been expensive. Especially if you can't create a full production-like environment with real world data and stupid users to test with.
This is due to finance cheaping out and not allowing the purchase of an exact "test" system to work on. Also, the rush to production is often more important than checking to be sure it all works.
That said, its all a risk/reward thing. Maybe its often better to screw up production here and there than to spend tons of money and time on testing. It all depends if you're building software for a web site or a Mars mission. What is the impact of a failure, and is it recoverable?
Ninjas don't carry tic tacs
The in-house ERP system I work on has a great test environment and a huge suite of unit tests, and the corporate environment is pretty well defined, so few (if any) show-stopper bugs ever make it to production and those that do I can reproduce relatively easily.
On the other hand, I also spend time programming machine automation and the idea of having a completely separate independent machine to test your changes on is impractical. Machines that cost hundreds of thousands of dollars don't have test systems - you just have to be careful.
Somewhere in the middle are programs that have to operate on a wide variety of systems (Android phones or desktop PCs). You can employ lots of test procedures but it's doubtful you'll be able to test every different production environment prior to release. Not even Microsoft can do it 100%. That's just the nature of the game, and you do the best you can. It's the cost of having general purpose computers.
Yeah, there's always been bugs in production code and it wastes time+money. No, problems like 100% complete test data coverage haven't been solved yet.
It's been that way since the beginning of computing/engineering. And it won't ever be solved. Maybe by an utter stroke of genius but certainly not by some new hipster solution chasing unicorn money.
How is "management telling people to put it into production as soon as the basic functionality works" not one of the common causes of bugs? At almost every job I've worked at, QA and Engineering would say "We need this much time to test and fix bugs before launch", and management would say "Too bad! Sales already told someone we're launching tomorrow, so we're going live with whatever we have then!"
It isn't the lack of a good test environment, or good test data, it's being told by management that you aren't going to have any time to test...
Apppy app developers don't use LUDDITE debugging software.
Crapps!
Even if you can reproduce all of the hardware exactly, you are never going to get the same kinds of results that putting software in the hands of real users will get you.
I'd say far more important than exact hardware duplication of a production environment, would be ease of replication of real production data into QA servers for replication of issues.That includes at any time in QA being able to use the account of any real production user as if you were them... The ability to do that easily saves SO MUCH TIME. Yes it opens up some privacy implications but as we can plainly see few people really care about privacy, or at least value it nearly as much as solid software.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
I spent some time working QA on a carrier-level system that was being developed for what was at the time Cingular. The biggest problem is that the investors that propped-up the company wanted it to ship as absolutely as soon as possible, so the company could go from a money-sink to a money-producer for them. Our investor was some heir to a fortune that was made in chemicals back in the day, he didn't really know anything about the technology of telco-grade communications systems. He was ill-qualified to even know if his money was going to producing something functional.
The idea (basically take-in anything, process it for meaning, and then turn around and convert and resend or else store and notify) was a good one and at the time there wasn't really anything else on the market doing this at carrier-grade. The problem was, while the central core of the product was reasonably well written, so many input/output daemons and filters were just garbage. The rush to get the product making money meant it shipped well before it was ready, and in the end it became the only sale that the company had.
A couple of years later the whole product/project could've been had for something like $200,000. They'd sold the only production system for more like $1,000,000.
Do not look into laser with remaining eye.
I often find it's due to changing requirements from management, or to this odd idea that they have that they can't possibly tell us the entire process that we're working on, so we either end up not understanding the end goals (because of lack of knowledge), or have to refactor things in order to implement the steps they only told about after we finished the first part of the project.
Mr. Hu is not a ninja.
Apps are very appy when you have to app. Testing is an important part of apping.
I think also that it is understood that bugs and computer errors have grown to be the norm and are accepted as just part of having a computer. Why fix it if we won't lose any business over it? We can push the "blame" for the bug on the user's "toxic" or "invalid" system or any number of reasons, hell, even blame it on the latest Windows update, making it Microsoft's "fault", because that update wasn't in our test environment.
What is the impact of a failure (bug in release)? Nothing.
investor of one project don't want to invest now time and resources in producing unit and functional tests. if project has good test suite, debugging in production should be a really marginal.
I did a six-month contract as an software tester internship after college, where I came across a crash bug on the test server that I could reproduced 100% of the time. My supervisor could not reproduced the bug, and approved the patch for production server. The production server crashed immediately from the patch. Engineers determined that a major code rewrite was required to fix the underlying problem. The production server was offline for three days and cost the company $250K in lost revenue. My contract wasn't renewed, one-third of the division got laid off after I left, and further budget cuts doomed the project. As for my supervisor, he got promoted into management.
Historically, software spend most of its time being maintained. And developers spend more time with old stuff than creating shiny new stuff. Whether finding problems with something you've inherited or something you're trying to use, we spend our time dealing with other peoples efforts (good or bad) ...
We release alpha software and our customers are debuggered!
When asked to identify the environment in which bugs
are most costly to fix, 62 percent selected production as the most expensive
stage of app development to fix errors, followed by development (18 percent),
staging (seven percent), QA (seven percent) and testing (six percent).
If that sentence means what I think it means, the old 100 times cost transitions between the software process stages is debunked. A flaw found in staging and QA still travels through the development and testing in traditional models. Do modern tools and automation really skew the estimates this much?
They must be using Agile methodology!
Their apps run like shit and they keep bolting on more features but refuse to fix the shit. We'll see how that pans out for them.
I just created new software at ErrorUnit.com that exactly tackles this issue. It creates C# unit tests (MSTest or NUnit) Mocking all the data accessed so far via EF, the method parameters, and all the class properties/fields (even private). Or it can create a unit test for where you are paused in the Visual Studio debugger.
Lifes a game play to win!
80/20 rule. Get the 80% coverage cases done and when the "edge" cases pop up, deal with them as they go. I used to spend a huge amount of time coding defensively (lots of checking for nulls, lots of brainstorming "what could go wrong") and my bosses said I was taking too long. Apparently, debugging looks like you're working and is money well spent when trying to proactively solve issues is fappery. As long as the checks cash...
Yeeeeeup, exactly this. Goddamn do I ever wish I had the resources available to me to actually do my job properly. Company wont provide resources for unit testing the hundreds of variables for our data entry forms that all inter-relate to one-another. Think of it as a massive fucking configuration matrix that shits all over itself. I've proposed for years entirely replacing said system with something extremely simple, but am always shot down. And since we don't have the resources to properly unit test the system in place now, instead there is an audit log that simply tracks our staff's actions with the system, and when an issue arises, it usually takes only a few minutes to debug and push a fix live.
Ask the Scrap-hiaparelli Mars mission how they tested their system..
That's not the most prevalent issue. The main issue is the malpractice of Agile methodologies. What happens when you jam a 2 week task into a 1 week time box? Corners get cut in the code, the unit tests, QA test plans and technical debt accrues creating unpredictable results when someone changes brittle code in the future. Most companies are not interested investing in REAL environments and continuous delivery pipelines with:
All of this takes a lot of effort and you don't get it for free running around like a chicken with your head cut-off. Ignore it and you reap what you sow especially in larger scale software efforts.
We'll make great pets
Modern appy app apps don't have LUDDITE errors, they only have apps that app other apps! Only LUDDITES have to debug LUDDITE errors in LUDDITE software!
Apps!
The headline should read:
Project managers don't invest enough time and resources in testing and validation before going to production.
You think developers make these kinds of shit decisions???
Tell us something we don't know...
I wrote a awesome testing program that resolves the problem of differences between test and production but I can't get it to run in a production environment.
She was like chocolate when she drank... semi-sweet at first and then increasingly bitter.
It should also be noted that your Angry Birds game crashing isn't the same as, say, Schiaparelli.
The cesspool just got a check and balance.
It's the standard triangle. You can cut from one at the increased detriment of the others. As long as the others are finite resources you always have to cut somewhere. The problem so many developers can't understand is that the 'where' is a business problem, not a theoretical engineering issue.
If it's more important to remain under budget, or be first to market, yeah, quality might suffer big time, and it's easy to ignore the academic's concept of a perfect engineering development lifecycle with a full QA and test system that, by it's very nature, must be more expensive than the actual production system itself (because it's the production system PLUS extra bits for testing).
Companies learn to handle this fairly well - or they go out of business. They gauge the severity and frequency of errors their users are willing to tolerate to keep them around the top of the maximized profit curve.
This means that as much as you want to refactor all that crap code, it just doesn't pay out to the company. ... and so on.
It means that while you'd love a perfect QA test environment, 3 VM's the lead dev set up on one of spare dev systems is going to have to be good enough because the money for the hardware isn't there.
You'd love to make a fully functional semi-autonomous system to manage common issues, instead of making devs work a rotating on-call shift but it's not financially worth it.
What so many of us don't understand is that our job as software devs or other technical engineers is NOT to make a high-functioning, beautifully coded, well maintained product. Our job, the reason we were hired, is to build revenue for the company. Anything else is just a byproduct of work towards that goal. If you can make more money with an app that crashes every hour than you would from spending 3 months testing it, then that's what you do.
Not that it doesn't irritate me too, every time one of my products is pushed out the door without a proper shakedown, but you gotta face reality.
The prime directive of anyone associated with building software for end users must be to create bug free, secure systems that are effortless for people to use.
This needs to flow throughout an organization - whether you are the architect, designer, marketer, developer, tester, accountant, whatever. Everyone must be on the same page when it comes to this goal. Everyone needs to really understand what that entails in practice.
I've been both on the building, and receiving end of things when this goes wrong - and it goes wrong too often than is necessary, primarily because an organization does not have that unifying goal. From a user's perspective it sucks because you end up with a confusing mish-mash of tools with no unifying concept behind the interfaces, and which fail to integrate data effectively to avoid redundancy. 'Painful' is a good adjective that describes using such systems. From the developer's point of view you end up unable to do your best work. Finance or management doesn't provide the right resources, time, or unifying definitions for the solutions in the company's stable - everything seems to be a one-off that you end up throwing over the wall until the next project comes along. Responsibility and ownership is minimal at best - leading to long nights debugging production code, and too many times devolves into finger pointing and recriminations.
Given the current state of affairs I think it is time for people to find new concepts of how software and systems development really should work for all of us.
One thing that occurs to me is we should stop rewarding companies / projects (in the case of open source) for producing poor quality systems and software. If you want to build crufty systems for yourself, that's one thing. Don't foist that off on the public. A way to make it easy for end users to identify such systems could be a certification mechanism - an independent body that could look at various criteria to rate software and systems on an scale (e.g. unrated, low quality, medium quality, high quality, etc). The criteria used could be things that matter - such as bug history, security bug history, availability of code for independent review, complexity vs. simplicity of code reviewed, ease of use, ease of integration with other systems and data, etc.
Similarly, I think development tools, and organizations and companies that develop tools and systems should similarly be rated to allow potential consumers and users of their work to make more informed decisions.
Lodragan Draoidh
The more you explain it, the more I don't understand it. - Mark Twain
It is a price you pay for time to market approach. Unfortunately most of the companies do not go back and re-pay debt by fixing quality of the software. They prefer to move to the next project. Of course you can show around your success with a project under the budget. Someone else will pay for your Technical Debt. Good luck.
Halfway through the summary I could already tell that this survey was a piece of poorly concealed advertising. I clicked on the link to confirm my intuition, and guess what? ClusterHQ is releasing two new products to take care of JUST THAT PROBLEM. After all, it is well known that throwing money at a problem, imagined or not, is an effective remedy to all woes.
I wonder why I even bother reading this site anymore.
Jamming 2 weeks of work into 1 week is going to result in cut corners no matter what methodology you're using (or even what line of business you're in, for that matter.)
If you switch to a methodology where you're estimating in 6 month blocks and you're off by 100% like that, you're now 6 months off schedule instead of one week off -- that's even worse!
Not to say agile isn't misimplemented regularly, but if you're not schedules are off by that much of a margin, you need to start by looking at how you're generating time estimates before you bother changing your entire methodology.
That's what happens when your entire development pipeline aims to put a prototype into production.
Also, "Just validate user input on the front end, it'll be fine once it hits the server" is a recipe for disaster.
If app developers put fewer bugs into the software in the first place, they wouldn't have to deal with them in production.
Most of the bugs I've seen are due to things like craptastic design (if "design" is even the right word for software cobbled together from bits that almost but not quite do what the user needs), sloppy code practice, inadequate testing ("hey, it started up, must be good!") and ignoring all those errors and warnings written to the logs as long as the software struggles along doing something. Oh yeah, and crappy (non-existant) fault tolerance ("what do you mean, the app may not be able to access the network resource it needs to? that's not my fault!").
Yeah, once in a while a legit bug creeps in (obscure race condition, corner use case), but that's fairly infrequent compared to plain sloppy design/coding/testing.
(Currently in production support, worked in development for 20+ years.)
That's because only LUDDITE developers test things. APPS APPS AGILE APGILE KAKPILE!
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
It's a well known exponential curve, that apparently needs to be re-learned for every "new technology".
A bug found during requirements analysis has a cost to fix of 1
A bug found during high level design has a cost to fix of 10
A bug found during detailed design has a cost to fix of 100
A bug found during coding/unit test has a cost to fix of 1000
A bug found during system test has a cost to fix of 10000
A bug found in production has a cost to fix of 100000
General Relativity: Space-time tells matter where to go; Matter tells space-time what shape to be.
Fewer bugs.
--
Signed,
BA in Eng Lit, MBA.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
We get hung up on developer costs but never on rework and fix costs. There is constant pressure to deliver untested features to make sales but never much accounting for customers who will walk at the first opportunity or sales which get cancelled due to bugs.
And it has never changed. Watefall, 6 signma, kanban, agile, rapid proto=typing, devops etc. has not made a difference. I have seen no improvement at all over close to 30 years. And people wonder why I drink.
putting the 'B' in LGBTQ+
I have never seen a methodology survive its first contact with sales.
putting the 'B' in LGBTQ+
Respondents were also asked to identify the most common causes of bugs
Surely the cause of bugs is programmers getting it wrong (or, if you want to go to a higher level, errors in the design or specification). All the cited reasons don't cause bugs, they merely prevent their detection.
As for the most environment where bugs are most costly to fix, I would suggest that would be once they reach the consumer and can only be fixed by a product recall? Although once they reach orbit, that can be a pretty expensive place to apply a fix, too.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
It all depends if you're building software for a web site or a Mars mission. What is the impact of a failure, and is it recoverable?
For the Mars mission:
a) about 186mph
b) no
http://www.space.com/34472-exo...
If you post it, they will read.
In all my years developing apps, I only had one live bug and it was basically due to uploading the wrong version to production. Some of my apps are over 50k lines of code! Yet, I can't find anyone hiring a software engineer. Its rough to know your stuff, and hr to not be able to tell you know how to code properly.
God spoke to me
Only if the bugs are reasonably countable, which in the case of some poorly-written software they are not. :)
This is why data modelling languages like UML or mathematically complete languages like Haskell were created. How many development studios use such tools?
There have been some language improvements like classes and polymorphic functions but mostly, studios want the power to slap code and data together in a limitless morass. Add to that, scope creep as the system tries to provide a uniform interface and big-data synergy, which is increased as everyone climbs onto your project (and budget). Then you've got a lack of in-house change management, halving the QA testing period, stupid users and even incorrect system specifications defined at the very start of the project, to ensure the new, improved system is flawed.
Yeesh. Read Knuth and investigate some Haskell, TLA, Coq, and even F#.
The courts spend too time time debugging errors in the legislature.
Why is it that since lately people who obviously have no clue about agil methods are bashing them on /.?
Actually every point you bring up are corner stones of all agile methods I'm aware about.
Cost free eBook I read (by iBook/Kobo/Amazon/ObookO/Gutenberg etc.): "The Green Odyssey" by Philip Jose Farmer.
In the old days, if you found a bad bug in production code it often meant you had to stop the assembly line of shrink-wrapped boxes filled with floppy disks or CDs and pull back everything that was unsold in the channel. If your channel was big enough, it could cost $ millions. Today, you just patch your download sites and have the running software on your customer's machine automatically download and update it sometime after hours (or even during work hours). Companies are much less hesitant to ship buggy software when they know it can be fixed later at nominal costs.
Where I used to work - big telco software firm whose software generates 80% of the phone bills in the US we had a simple solution to the problem of testing to scale.
We had two identical setups one for production and one for staging. After UAT was almost over we would deploy to staging and then continue UAT on the staging with real world data till the day of cutover (Use Oracle Active-Passive to keep both in sync for the production data while not copying over UAT data to prod)
On cutover day we would change the network switch to now point to the new setup and run scripts to delete the dat created by UAT.
The nice part was now the Prod setup (a bank of 8 servers with 4 quad core CPUS each) was now our backup machine. We would switch it to passive and continue to keep it in sync with prod for at least 7 days. If something horrible went wrong with the new setup. Changing back to the earlier prod machine was a network switch flip. The scripts were a little more difficult this time over especially if the software bug had messed up the data but it was still easy.
Once a production was stable the old prod was now used as staging for the next prod.
What this meant is we did UAT on machines with identical config as the prod machines . It solved a lot of issues and since we also used the machines as the prod backup machine during cutover the cost was taken from the operations budget and not the testing budget.
Our System test and UAT environments were almost as good but not as good as prod and most testing and UAT was done there but the last batch of UAT on the big iron gave good confidence and made cutover day a lot less stressfull than it used to be.
**Life is too short to be serious**
... or at least the nice things we buy turn to crap shortly after. There are way too many developers who don't understand what the hell they're doing, and they're layering their crap over many other layers of crap. This is development by random walk.
I'm a QA Engineer for a living and I can say, from experience, that most of the bugs that make it into production are due to tight deadlines or because not enough discovery was done before the app started it's development phase.
I've seen major defects get placed on the back burner in order to make a deadline, roll out into production, only to get an influx of customer complaints for that exact defect.
I've also seen major design problems get discovered halfway into the testing process and be ignored because trying to fix that would cost more money in the long run.
Time is everything when it comes down to the tech industry. People want faster, better, more content, NOW NOW NOW and because of it, apps receive less attention than they deserve.
Only 25%. You'd be lucky if you have a QA env. In a small shop here is how it goes. Idea, code, does it compile? yes = production! Then the bugs present themselves in all the splendor and you repeat the process. Move fast and break things.
I wouldn't be so skeptical. Things like this happen (happened to me, too). However, the "interns saving the world" cry wolves far too often, while lacking the confidence or knowledge to break the news straight to the right people (the devs in GP's case). And as one gets older, it's easier to remember the times when you were right :)
WYSIWIG, but what you see might not be what you need
That's a huge reading assignment list; can you just say what you mean?
Slow down your production process so you have time to catch them?
That causes end users to choose a competitor's software with tolerable defects over your unfinished vaporware.
Do without the fancies so there are less vulnerability points?
That causes end users who rely on "the fancies" to choose a competitor's software that offers "the fancies".
I do an ops/engineering role...and the amount of time we have to get dev involved is staggering.
1) QA has 30 envrionments but NONE match production
2) QA has sanitized data
3) QA doesn't really understand the work flow
4) Dev has access to QA to 'help out'
5) dev doesn't create ANY log files
Only 2/5 of those problems I blame on development....but if they at least wrote logs it would be easier to diagnose issues.
About 2-3 weeks a go a program that ran for about a year broke. No logs existed to see what the problem was...as development was looking at firewalls, networking, database errors I joked "why aren't you looking that this broke on the first day of the year with 2 digits for month AND 2 digits for day" Turns out that was the problem, it forced a 15 digit field to 16 digits.
Yes. As I understand I4ko's post, the feature is "required by regulation". Without it, there is the big fat goose egg of 0 revenue.
1. The best test environment is production.
2. Code yesterday, deliver today, think tomorrow.
3. Beware the above code. I have only proved it correct, not tested it (after Knuth)
And then, there's always the wisdom found in The Tao of Programming, which was banned at my last job.
Lastly, we should all be happy for crappy software, as it keeps many of us employed. We should also appreciate the fact that users and management are never content with stable, useable, useful software, as it does not usually incorporate the "latest and greatest" technology. Stability and utility are apparently not desirable qualities of software.
Don't bother debugging production bugs.
Programmer's dream.
This is lazy thinking, but unfortunately common in large organizations.
If you have, particularly, a large production system, it is expensive to have a copy of production. The money could be used more effectively than to tape over poor processes. If you're doing this, you may as well call the "test" system "prototype" and just get on with fix-on-failing.
If you need an exact copy of the data (which has regulatory and compliance issues in and of itself) for developers to iterate over issue by issue, it is a great indication that your software is badly designed and not layered in any meaningful way.
If you haven't bothered creating non production test suites that exercise the system correctly, now is a good time to start.
App developer here.
Something is missing here; namely we spend more time debugging issues found in production, because they get reported. Almost every app nowadays has a crash logger that reports all crashes. Libraries like Twitter's Crashlytics are awesome like that. You get all crashes reported to you, including a ring buffer of the last 100 log messages. It's really, really awesome and I've solved problems in production that wouldn't ever be found normally.
8 of 13 people found this answer helpful. Did you?
They've been doing it for years, I find it fascinating how easy it is to rebuff most of the claims. But, I think it shows the industry is just really poor at executing it and end up with Fragile instead.
Change is certain; progress is not obligatory.
It's just impossible to test everything in your test enviroment. There is NO test suite that will allow you to test everything with a 100% gaurantee it will be bugfree in production. Yes, maybe in a simple very limited app on a very limited OS it will be possible. But with the very extensive configurations (settings/drivers/etc) with modern OSses which run on a gazillion different configured hardware, it's just impossible.
Ofcourse the marketing people of those test suites (and a lot of developers who swear by test driven development (which ofcourse is a good practise, that aside)) will make you believe it is possible to create bugfree software... But then reality strikes..........
Meanwhile, here in the valley, the latest trend in testing is not testing. Eliminate your QA department today!
I blame Agile and the mockery that management made out of it. We are agile now means that we can put stuff on the sprint backlog at will and declare it done after three weeks. Agile also makes documentation a bad thing, so we stop writing stuff down, including requirements. We are agile now, we can change our minds at will and everything that is inconvenient (such as fixing bugs found by QA before release) is postponed, reason: we hit it early next iteration. Agile enables those who do not want to commit to a design approach causing requirements (if there are any) to be vague at best. How is QA supposed to determine if devs build what fulfills the customer needs? Especially when management rather spends money on more devs and more new toys and more JS frameworks instead of equipping QA properly and giving QA and support veto rights to block a release? Agile is the death to quality and I am surprised that devs spend that little time on bug fixing in production. Seems as that users put up with quite a bit of crap before they complain. The mobile apps have groomed them well...or did you ever come across a mobile app that actually works reliably well?
N/T
See subject: Thanks for letting me KNOW you're an advertiser w/ your useless "media studies" BA Hognoxious https://slashdot.org/comments....
* :)
(Explains it all as to WHY YOU TROLL ME (& get your ass kicked too, lol https://ask.slashdot.org/comme... )
Yes... you're a fucking ADVERTISER w/ that "MEDIA STUDIES" moron LIBERAL ARTS DEGREE (& used to annoying folks, like your STUPID "app, Apps, APPS" bullshit you were caught in also here now parent to this very post https://developers.slashdot.or... ...)
Let's see - iirc, that'd make YOU about the, oh... 4th one I've busted in advertisers trolling me here on /., proving you're stupid in that I've caught you as I have others here doing what YOU do (trolling me).
APK
P.S.=> Grow up you fucking loser, grow up & get a REAL JOB, wageslave (instead of being a pain in the ass annoyance to everyone here)... apk
Every software developer knows this. Unfortunately, software shops are organized to work this way. They reward the developers to develop and the testers to test.
I used to work at a place where the test manager would preach how important it was to identify bugs early, quoting statistics about the costs increasing to fix bug at later phases. She was implying that the developers should solve this problem on their own of course. When asked to provide anything resembling comprehensive test data up front or reproducible workflows later, she went into full cognitive dissonance mode and insisted the effort wasn't worth it.
None of which was cheap, and none of which didn't require support from the very top.
It's very impressive that the business did all that. They are, at least according to many people's experience, the exception, and not the rule.
If someone is "fragile" it is hardly a problem of the "method" used ... such teams will be producing bad software ... or good software in a badly way produced, regardless of method.
Cost free eBook I read (by iBook/Kobo/Amazon/ObookO/Gutenberg etc.): "The Green Odyssey" by Philip Jose Farmer.
I have never seen a methodology survive its first contact with sales.
That's because sales always asks for flying, sparkly unicorns the defecate gold bricks without considering whether it's actually a reasonable expectation to have. Oh and they promised it to a client so if you could make that happen so they could get their commission and gain favor with the CEO, that would be great. k thx bye
We'll make great pets
But, I think it shows the industry is just really poor at executing it and end up with Fragile instead.
Oh that would be a step up in the environment I'm in. It's all cargo cults here. You have it really good if you get at least Fragile. That means someone is actually trying to do it but failing at execution.
We'll make great pets
If you switch to a methodology where you're estimating in 6 month blocks and you're off by 100% like that, you're now 6 months off schedule instead of one week off -- that's even worse!
You apparently have never worked at a place like this: https://www.youtube.com/watch?...
Trust me that happening once every 6 months is waaaay better than every 2 weeks.
We'll make great pets
"If Engineers built buildings the way Programmers write programs, the first woodpecker that came along would destroy civilization."
If you program it properly in the beginning, with the error checks in place, then it takes -less- time, not more. 8-)
App Developers Spend Too Much Time Debugging Errors in Production Systems
One thing I don't see anyone talking about here is the need to reduce the amount of change (SLOCs, FPs, whatever), done between QA cycles. Any serious system is going to have some sort of QA cycles before becoming available to the consumer.
Take SLOCs for instance. Say, from last release to production to the first QA cycle, you have 1000 SLOCs (or FPs) of change. And experience tells you that, in your organization, it takes 2 weeks of QA to test that much code change. And your first QA cycle is 2 weeks.
So, you get feedback from the QA cycle. You know, bugs and shit. So you and your team go back and fix the errors. Now onto QA cycle 2 (2 weeks.) But wait, from the QA release you submitted to QA the first time and the new one you are about to test, you have 1500 SLOCs of change. But your 2nd QA cycle is still only two weeks instead of three.
See the problem? Logically, one should extend the amount of QA on the 2nd cycle to cover the potential bugs in the new, larger delta (or negotiate *not* to fix all bugs and just ship as-is, patch later.) But in not doing so, the new release now has a higher probability of carrying undetected/serious bugs that could not be caught for lack of QA time.
It goes without saying that TDD/BDD won't help. There is a limit to how much you can automate testing of integration logic, usability, performance, etc.
This pretty much falls into the category of changes that are 1) done at the last minute, and that 2) aren't tested enough.
THAT IS THE KIND OF SHIT THAT MAKES UP WAKE UP AT 2AM TO TROUBLESHOOT A CUSTOMER ISSUE.
A similar scenario is when you deliver N features. They all go to QA which test the features according to plan. You get QA feedback and fix some shit. But in addition, now you decided (or someone forces you) to add a new feature. The previous features go through one QA cycle, and they and their fixes go again through a second QA cycle. The new feature however, it only goes through one QA cycle (the 2nd one).
Assuming the 2nd QA cycle is the last one (because you are like "shit we have a fucking deadline and the cost of development is $1K a day if we missed it"), you will get some feedback, but any bugs will go unfixed, or you have to stop the train to have another QA cycle (thus missing the deadline and raising the cost of development.)
So most likely, it will go as-is, with untested/unknown bugs due to the nice new feature stupidly added at the 11th hour.
In short, don't add unnecessary shit at the last minute. Keep your development efforts to debug shit you get from QA.
And no, continuous delivery does not necessarily absolves you or helps you to get away with it. It only helps you to push fixes and new releases faster (and if you are doing shit correctly, it will be more about features than bug fixes.) If you do not throttle non-critical changes as you get closer to shipping, your quality will suffer no matter what.
We do that with our database, exporting the entire thing nightly and importing into a new version. We did that at first just to verify the export was good (I've been burned too many times by bad Oracle exports). But we pointed our test setup to that database instead.
It was really nice because we had yesterday's data to test software on and the users could test data changes themselves without having to do it on their production system.
And the few times Test was down, we knew the export was bad (altho our master IT dept didn't believe us ntil they tried importing it themselves).
I can think of real-world examples where this sort of thing happens:
Video Game industry - Sure some older games had re-releases that fixed some issues, and some games were crazy buggy (youtube Sonic 3 glitches). However, the games typically "just worked" in the 80's and 90's. Compare that to today with multi-GB day-one patches that *should* have been part of the gold disk... had sales/marketing/management not put an improbably deadline on development.
OS Development - See all the zero-day bugs in Windows, OSX, Linux, and mobile OS's. There's a reason we have things like BlackHat, Pwn2Own and such.
IDE and plugin development - Visual Studio is a bloated mess. Eclipse is a resource hog that sometimes refuses to save files, other times refuses to open them or even gain mouse focus. Code::Blocks has it's own wierdness to try to use.
Macromedia/Adobe Flash - Need I really say more? This crap is broken by design.
Really, I think this sort of thing is in all software development. Proper practices are the exception, not the rule.