Slashdot Mirror
Performance Bugs, 'the Dark Matter of Programming Bugs', Are Out There Lurking and Unseen (forwardscattering.org)
Several Slashdot readers have shared an article by programmer Nicholas Chapman, who talks about a class of bugs that he calls "performance bugs". From the article: A performance bug is when the code computes the correct result, but runs slower than it should due to a programming mistake. The nefarious thing about performance bugs is that the user may never know they are there -- the program appears to work correctly, carrying out the correct operations, showing the right thing on the screen or printing the right text. It just does it a bit more slowly than it should have. It takes an experienced programmer, with a reasonably accurate mental model of the problem and the correct solution, to know how fast the operation should have been performed, and hence if the program is running slower than it should be. I started documenting a few of the performance bugs I came across a few months ago, for example (on some platforms) the insert method of std::map is roughly 7 times slower than it should be, std::map::count() is about twice as slow as it should be, std::map::find() is 15% slower than it should be, aligned malloc is a lot slower than it should be in VS2015.
It's stupid to call them "the dark matter of programming bugs". We were just accustomed to this being the way Microsoft did things, not a bug, a feature.
That stems from Microsoft, originally writing for IBM, being paid per thousand lines of code. As such it made sense that software was not written efficiently because the programmer was not rewarded for efficiency, it merely had to fit within available memory. Unfortunately it seems that this practice has not stopped given the sheer size of Microsoft operating systems relative to the amount of end-user functionality that has been added since the days of say, Windows 3.1.
Do not look into laser with remaining eye.
>> that the user may never know they are there
They will if they try to run a lot of them on a machine with finite resources, like a phone. Or it's a process that's iterated frequently, like a "big data" operation. But if the end user STILL doesn't notice it...then it's hard to call it a bug.
On the other hand, the performance/just-get-er-done trade-off is well known to programmers of all stripes. (At least I hope it is - are people really finding new value in the article?) There's the quick and dirty way (e.g., a script), and then there's the "I can at least debug it" way (e.g., a program developed in an IDE), and then there's the optimized way, where you're actually seeing if key sections of code (again, especially the iterated loops), are going as fast as possible. Generally your time/cost goes up as your optimization increases, which becomes part of the overall business decision: should I invest for maximum speed, maximum functionality, maximum quality, etc.
I came here to say this, mostly.
I *know* that there are plenty of places in our software that I could spend an hour or two, and rewrite an algorithm to run in 1/5th the time. And I don't care at all, because the cost is too low to measure, and usually, performance bottlenecks are elsewhere.
Who really cares if I can get a loop to run in 800ns instead of 1500ns, when the real bottleneck is a complex SQL query 11 lines up that joins 11 tables together and takes 3 full seconds to run?
I have no problem with your religion until you decide it's reason to deprive others of the truth.
Yes. This is about opportunities for optimization, not bugs. All experienced developers know that premature optimization is evil. You write code to be easy to understand and maintainable first, then do a performance analysis and decide what is and is not "good enough". If you can't show that there is a true issue then you better be able to change it in a way that doesn't sacrifice readability, maintainability, reusability, etc. or you are a hack.
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
Maybe in your world, but when weighted down with sloggy operating systems and minimal memory (typical of many Windows 10 installations TODAY), code can get pretty slow.
For a very long time now, there have been libs that add breakpoints to examine how long processes are taking, think: debug mode, that can pinpoint problem areas pretty easily. Not enough coders use them.
It gets worse when a user has 94 Chrome tabs open, something in Office, and an AV app running.... all on a laptop whose processor speed is measured in furlongs per fortnight.
Yeah, SOME computers are way faster, and some have been habitually overloaded with things outside of a coder's control yet their app still must perform within a "reasonable" amount of time. Blah.
---- Teach Peace. It's Cheaper Than War.
It is a losing battle to try and solve performance in the programmer space. The Compiler does a much better job of optimization due to a multitude of compiler trics including both Static and dynamic analysis, cache analysis and so on. The programmer trying to write the most efficient code should rather spend his/her time trying to use out of the box algos as far as possible as the compiler knows how to fine tune those. next they should run a profiling tool like jprofiler and see where the job is actually spending its time rather than trying to say this is probably the heaviest part of the program. With multiple cores and multiple instruction pipelines and optimizing compilers the bottleneck is oftentimes not where we would think it to be. Once we find the bottleneck using a profiling tool than we can optimize it. In most cases 2% of the code is causing 98% of the bottleneck so its a much better use of programmer time (which is of course more expensive than computer time in most cases) to work backwards.
1 Write your code so that its correct irrespective of efficiency,
2 profile and then
3 fix the bottlenecks
rather than trying to find the most efficient algorithms before you write your code.
**Life is too short to be serious**
Programmers love to use the cop-out
"Premature Optimization is the root of evil"
dogma which is complete bullshit. It tells me your mindset is:
Except later never comes. /Oblg. Murphy's Computer Law:
* There is never time to do it right, but there is always time to do it over.
As Fred Brooks said in Mythical Man-Month.
Which can be translated into the modern vernacular as:
* Show me your code and I'll wonder what your data structures are,
* Show me your data and I'll already know what your code is
There are 2 solutions to this problem of crappy library code.
1. You are benchmarking your code, ALONG THE WAY, right?
Most projects "tack-on" optimization when the project is almost completed. This is completely BACKWARDS. How do you know which functions are the performance hogs when you have thousands to inspect?
It is FAR simpler to be constantly monitoring performance from day one. Every time new functionality is added, you measure. "Oh look, our startup time went from 5 second to 50 seconds -- what the hell was just added?"
NOT: "Oh, we're about to ship in a month, and our startup time is 50 seconds. Where do we even begin in tracking down thousands of calls and data structures?"
I come from a real-time graphics background -- aka games. Every new project our skeleton code runs at 120 frames per second. Then as you slowly add functionality you can tell _instantly_ when the framerate is going down. Oh look, Bob's latest commit is having some negative performance side effects. Let's make sure that code is well designed, and clean BEFORE it becomes a problem down the road and everyone forgets about it.
2. You have a _baseline_ to compare against? Let's pretend you come up with a hashing algorithm, and you want to know how fast it is. The *proper* way is to
* First benchmark how fast you can slurp data from a disk, say 10 GB of data. You will never be FASTER then this! 100% IO bound, 0% CPU bound.
* Then, add a single-threaded benchmark where you just sum bytes.
* Maybe, you add a multi-threaded version
* Then you measure _your_ spiffy new function.
Library Vendors, such as Dinkumware who provide the CRTL (C-Run Time Library), _should_ be catching these shitty performance bugs, but sadly they don't. The only solution is to be proactive.
The zeroth rule in programming is:
* Don't Assume, Profile!
Which is analogous to what carpenters say:
* Measure Twice, Cut Once.
But almost no one wants to MAKE the time to do it right the first time. You can either pay now, or pay later. Fix the potential problems NOW before they become HUGE problems later.
And we end up in situations like this story.
Same here
The users are getting a correct result. Good.
The developers moved on to something else that's also important. Good.
The machine is doing 15% more work than strictly necessary... Is it slowing down the users? No. Are we getting hammered by the electricity bill? No. Is the machine getting tired? No. So what exactly is the problem?
Like the real Donald (Knuth) said: "premature optimization is the root of all evil (or at least most of it) in programming".
Being an old fart, in my day, I remember the worst performance problems were caused by programmers with their own badly written library of functions and objects that they included everywhere, most of those were from their very first weeks of being a programmer and they sucked badly.
Congratulations - you've invented the "message queue".
In 1973 or 1974 I was the systems programming manager at the National Academy of Sciences after our IBM mainframe had been upgraded to the first version of the OS supporting virtual storage. And many programs, mostly Fortran programs, were running much slower than they used to. The problem was two-dimensional arrays and how they were initialized and searched. If you're looping through the wrong subscript in a big array, you cause a page fault every time you increment it. Flash storage makes that a much smaller problem than it is with disk drives, but I'm sure most programmers today have no idea that this problem exists or how to handle it.
It would take as long as it is cheaper to run the inefficient query than recoding it.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Videogame programmers care *very much* about all these sorts of performance issues. Not coincidentally, many videogame programmers use custom containers, and nearly ALL of them use custom allocators for exactly this reason.
That being said, not everyone programs real-time pseudo-simulations like we do. But you should very much care about ensuring the most basic building blocks of code everyone uses are highly optimized at the very least. The more often code is called, the more attention should be paid to ensuring its as optimal as it should be.
I'd be curious to hear MS's response to these issues. It could be that there were some deliberate reasons for the choices made, or possibly some unintended consequences to the solutions offered, but it's hard to say without a fairly deep knowledge of the STL internals MS uses (based on Dinkumware, I think?)
Irony: Agile development has too much intertia to be abandoned now.
So it's highly dependent on what you think could happen, and what satisfies your sense of sufficiency. Of course, your time needs to be weighed against the cost of failure from the bug manifesting itself. Despite all that's been invested in development methodology, this is remains a black art.
Computers are today faster than they need to be for nearly all applications the average office runs into.
For nearly everything except rebooting. I still don't understand the problem. It's better than it was, but we're still back in the old days of waiting for the TV to warm up.
“He’s not deformed, he’s just drunk!”
Yes, ... umm... no.
Not anymore. Yes, it still matters with high end, AAA titles. For everyone else, there's Unity. And Unreal Engine. And ... something I forgot now. Game programming used to be one of the few areas where you really needed top programmers that can come up with creative ways to cut a few extra cycles. Just think of the infamous 1/sqrt(x) hack.
I wouldn't expect many who currently claim to be game programmers to understand it. Let alone come up with something close to it.
Even in games, efficient code isn't the be-all, end-all anymore. Being able to "cheat" with the graphics and making low poly shit look great with creative texturing is where the money is today, it seems.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
Real programmers use only the 1 and 0 keys
Keys? Real programmers use jumper wires directly on the memory bus pins of the CPU.
No. Butterflies.
You can optimize for dev time or CPU time. Which is cheaper?
The inexcusable is optimizing for neither. e.g. server side javascript.
John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
[...] looks a little more complicated [...]
A different programmer looks at it a year later, determines that it looks too complicated, and refactors the code to be more simpler "in a far more elegant, better scaling and maintainable solution."
Sayeth the noob who didn't think about how long testing the change would take...
Agreed that replacing tested/working code with new "more efficient" code does incur a re-validation cost.
On the other hand, that's also an argument for writing the more-efficient implementation the first time, rather than waiting until some later release. Since you know it's all going to have to go through the testing cycle at least once, why waste your QA group's time testing slow/throwaway code, when you could have them spend that time testing the code you actually want your program to contain? (Assuming all other things are equal, which they often aren't, of course)
The shortest distance from A to B is a straight line.
I don't care if it's 90,000 hectares. That lake was not my doing.
Fundamental library code is either as fast as possible, or useless. You know know who or how the library code will be used, so you have to assume plentiful use cases where every instruction matters. The std::map code is particularly bad (even in CLANG) .
When you're delivering the end product, sure, don't optimize until proven necessary. That's a different world than library code. Not every thing is your thing, surprising as that may be.
Socialism: a lie told by totalitarians and believed by fools.
This is why I do all my coding in PHP and JavaScript. Performance concerns are jettisoned before the coding even starts.
For an explanation: http://thedailywtf.com/article...
Isn't this why we have profilers like Valgrind to identify slow functions?
Anons need not reply. Questions end with a question mark.
Klingon coding...
Copy con: program.exe
Then enter structure, opcodes and data with Alt-keypad.
Comments...where would those go?
John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
You really must grow out of help desk and learn some useful stuff...imbecile fucker. And stop pretending you understand something about computers or even know how to read English errors. You are a fucking waste of space on this planet.
That's kind of harsh to say to an AC. :P
Its bad assumptions. Like cache-misalignment or TLB thrashing. The same code will run full-speed on one system but 10x slower on another with slightly different cache characteristics. Performance might even change with an upgrade to the OS or even just a system library that re-arranges memory layout in a trivial way.
That's why I'd like to see a nondeterministic compiler some day. Take dozens of good-ish but alternative decisions differently every time. I once did some very rudimentary checks with a Scheme compiler that already generated very good code but there were some weird things going on with inlining. Tiny changes in inlining thresholds were changing the performance deterministically with a factor of several. I suspected the way the code interacted with the CPU was the culprit. But since all CPUs are different (and new ones come almost every year), hardcoding the assumptions into the compiler seemed hopeless. It ultimately started looking like a solution space search problem.
Ezekiel 23:20
Although, a "scripting" language without a serious performance bug may actually process a larger data set faster than a program written even in a system language like C, if said latter program contained a serious performance bug. But not making obvious mistakes is hardly "optimization". At the risk of making a little bit too contrived example, the decision not to use a bubble sort is not something I'd call optimization. It's just common sense.
Ezekiel 23:20
It's not the use of programming languages, it's use of completely inappropriate, daily-WTF-worthy, algorithms and mechanisms. We do development that involves interfacing with various libraries and components provided by third parties, and holy fsck, it's a wonder some of this stuff works at all, let alone inefficiently. You look at some of the duct-taped-together crap that we have to deal with and in some cases literally can't see why what they're doing works, it's pure coincidence that the way they're using the APIs happen to produce an actual result rather than an error code or a hang.
Who really cares if I can get a loop to run in 800ns instead of 1500ns, when the real bottleneck is a complex SQL query 11 lines up that joins 11 tables together and takes 3 full seconds to run?
To misquote a not-quite-famous Congress critter: A nanosecond here, a nanosecond there, eventually we are talking about real time (minutes).
You do not exist in a vacuum. You think your application is the only one but I have hundreds of "applications", written by people just like you, who thought that the real bottleneck is elsewhere so why worry about this particular bottleneck?
This attitude is why I can notice the delay in a particular window opening up or why I see "hiccups" in the smoothness of my screensaver.
I am sure that numerous studies were performed to say that the user will never notice and that these "minor" delays do not affect anything... but these studies are wrong. I notice.
(Granted, you seem to be addressing a server application where the application truly is the only application that matters, but my point about unnecessary wasting of CPU cycles still stands. Should you go to the extremes that ancient assembly language hackers used to go through to make something not possible into possible? No. Each programmer should be trying to be as efficient with CPU time as possible.)
"Someone needs to talk to the tree of liberty about its ghoulish drinking problem." by ohnocitizen