If you don't know how your library deals with thread safety, then you either need a new library or you need to read the docs.
The single most important rule of threading is to have very well defined data interfaces, and never violate them. That's it. Claiming that threading is "toxic waste" is, I'm sorry, simply wrong.
Everything is moving toward more threading, not less- we need programmers who can begin thinking in that mode rather than those who just write it off. Threading doesn't *have* to be complicated unless it's in a design that's screwed up from the start and there's insufficient data abstraction and isolation. Once there is, if your design as simple and consistent synchronization requirements, following them should be trivial. If it's not, you've screwed up in the design phase.
The fact that you *can* screw up threaded code is no reason not to use it. You can screw up a lot of things in code that won't be caught in testing or at compile time, but we do them because they give benefit or because they're necessary. Threading's the same way.
It's really not that hard if you've got a decent design. Believe me, I've done a lot of it. Is it trivial? No. But any programmer worth his salt at this point should know how to do it.
The "threaded programming is hard!" whine is what's got us into the situation where there's a disconnect between the hardware we have available and the design of the software.
It really isn't that hard. If you have your data reasonably managed and your system reasonably abstracted, then determining the portions that can safely run on other threads (or safely with some synchronization) should be easy. If it's not, your design is borked to begin with and should be rethought.
Debugging threads really isn't that bad, either, tbh. I've spent more time tracking down obscure memory issues in c/c++ code than I have tracking down threading issues in any language.
This is very true, and I think this is part of the reason that many current applications have such poor performance- they get to the end stage and realize that they can't fix the problems without more resources than they have left, so they just throw up their hands and say "screw it".
I don't entirely agree with you here. A lot of current applications *do* suffer from CPU-induced latency after user interactions, and the problem is simple: they don't differentiate between the things that must get done before control is returned to the user, and the things that need to happen in response to the action but can be allowed to happen whenever resources are free. Even when the problem is resource-access latency, multithreading can be a win because that latency no longer contributes to the latency that the user perceives if it happens on a background thread.
Something as simple as tossing function calls off on a background thread to deal with some of these tasks would do a great deal to improve latency from the user's perspective, and is really quite trivial to implement. Most programmers don't do it, though. Part of that is that in most situations there aren't ready-made solutions- you can't just say "run this function call on a background thread", you've got to go through the pthread creation process, etc. (Apple's Cocoa framework is actually an exception to this with it's NSOperation).
The situation is analogous to that of an interrupt task: Do absolutely as little as possible before returning; everything else should happen on some other thread.
I agree with you regarding optimization, but it's been my experience that many applications *can* benefit from these sorts of simple multithreading techniques- the programmers just don't do them, either from lack of ability or lack of resources.
This is true to a point. The problem is that, in modern applications, when the user *does* do something there's generally a whole cascade of computation that happens in response- and the biggest concern for most applications is that app appear to have short latency. That is, all of that computation happens as quickly as possible so the app can go back to waiting for user input.
There's a lot of gain that can be gotten by threading user input responses in many scenarios. Who cares if the user often waits 5 minutes before input? When he *does* do something, he wants it done *immediately*. The fact that it's a tiny percentage by wall time doesn't change the fact that responsiveness here is a massive percentage of user perception.
The first link from Apple Insider is definitely a bit on the biased side. The second link, though, is to MarketWatch, and is a little better on the fact/rant ratio.
Uh? You can run linux on a mac just fine. Not the iphone, true, but the others, yes.
Not to mention that Darwin, the unholy marriage of mach and BSD that mac os x is built on, *is* open source. It's the graphics layer on top that's not, and that's not built on anything open source anyway.
Apple actually releases quite a bit of open source code. You need to get a better handle on the actual facts.
I'll agree wholeheartedly with that. In the meantime, though, people shouldn't ignore it simply because it has some flaws- time has shown that the advantages of POSIX far outweigh any such existing flaws.
What does "language" have to do with it? Software authors have to pay attention to many standards beyond that of the language they're working in. If they're working with XML, they should pay attention to the XML standards. If they're interacting with an operating system, they need to pay attention to the standards that the OS is written to- in this case POSIX.
The application authors don't need to know *how* the operating system implements the standard- they just need to know what they can and can't expect from those standardize functions.
The fact that ext3 was broken in this regard is a performance bug in ext3.
Also, it is very much necessary on ext3. The commit delay was short on ext3 (5 seconds), but a *lot* can happen in 5 seconds, and a lot of data can still be lost in that time period.
There's exactly one thing to take away from this. If you want data to be guaranteed to be on disk, call fsync(). You'll note that Linus doesn't at all disagree with this- what he's basically saying is that "ext3 implements fsync() in a braindead way that makes in painful for the whole system if people do the right thing."
That's not an API issue. That's an ext3 issue, and then it's an issue with application developers coding with the expectation of ext3.
Every scenario where data integrity is of importance to the user.
Note that this is not every scenario. Many, many programs use temporary files for storage, and in those cases, the corruption of those files during a crash is of no consequence. I've written and used applications like this in the past.
At the same time, I've written and used programs for which I could not countenance data corruption. And, for at least the programs I choose to use, those programs take the necessary measures to prevent it.
Consider an application like PostgreSQL. It doesn't much care what it's running on- because it uses the API appropriately, it can get verifiably correct behavior on *any* operating system (barring some hardware that actively lies- no app can be expected to be correct in the face of things that lie to it).
I *am* a developer, including of software that requires data guarantees. I have no sympathy for application developers who don't take such considerations and then complain when some other software doesn't compensate for their own bugs.
Apparently, you don't know how to *deal* with real life.
POSIX *does* tell you what happens if your OS crashes. It says "as an application developer, you cannot rely on things in this instance." It also provides mechanisms for successfully dealing with this scenario.
As for fsync() being a performance issue, you can't have your cake and edit it too. If you don't want to pay a performance penalty, you can lose data. Ext4 simply only imparts that penalty to those applications that say they need it, and thereby gives a performance boost to others who are, due to their code, effectively saying "I don't particularly care about this data" - or more specifically, "I can accept a loss risk with this data."
Normal applications have a reasonable expectation that the OS doesn't crash, yes. And usually it doesn't. Out of all the installs out there... how often is this happening? Not very. They've made a performance-reliability tradeoff, and as with any risk... sometimes it's the bad outcome that occurs. If they don't want that to happen, they need to take steps to reduce that risk- and the correct way to do that has always been available in the API.
As for forgetting POSIX... it's the basis of all unix cross-platform code. It's what allows code to run on linux, BSD, Solaris, MacOS X, embedded platforms, etc, without (mostly) caring which one they're on. It's *highly* relevant to the real world because it's the API that most programs not written for windows are written to. Pull up a man page for system calls and you'll see the POSIX standard referenced- that's where they all came from.
Saying "Forget POSIX. It's irrelevant in the real world." is like people saying a few years ago "Forget CSS standards. It's irrelevant in the real world." And you know what? That's the attitude that's dying out in the web as everything moves toward standards compliance. So it is in this case with the filesystem.
You're right. The correct thing to do is to *always* call fsync() when you need a data guarantee, *regardless* of which FS you're on. The fact that not doing it in the past hasn't caused problems isn't the problem- those calls are the correct way of handling things.
I disagree. "Writing software properly" apparently means taking on a huge burden for simple operations.
No. Writing software properly means calling fsync() if you need a data guarantee.
Pretty sure no one in their right mind would call using fsync() barriers a "huge burden". There are an enormous number of programs out there that do this correctly already.
And then there are some that don't. Those have problems. They're bugs. They need to be fixed. Fixing bugs is not a "huge burden", it's a necessary task.
POSIX is like a contract. KDE is breaking the contract and then whining about it to ext4- which isn't breaking the contract. Just as in a court, KDE here doesn't have much of a leg to stand on.
The only calls that say data is written are the fsync() family (or files opened with O_SYNC.
Those calls do not lie.
Unfortunately, application developers are assuming that other calls say something they do not, in fact, say. This is where the problem comes in. close() does not guarantee that data has been written yet. fsync();close(); does.
This is the attitude that has the web stuck with IE.
There's a standard out there called POSIX. It's just like an HTML or CSS standard. If everyone pays attention to it, everything works better. If you fail to pay attention to it for your bit (writing files or writing web pages), it's not *my* fault if my conforming implementation (implementing the writing or the rendering) doesn't magically fix your bugs.
If what you say is true there would be no need for the fsync() function (and related ones).
Read the standards if you want. The filesystem is only bugged if it loses recent data under conditions where the application has asked it to guarantee that the data is safe. If the app hasn't asked for any such guarantee by calling fsync() or the like, the filesystem is free to do as it likes.
... is the one feature I never see under OpenOffice release notes: Improved performance.
I keep trying OpenOffice, under multiple OSes... and I keep removing it in frustration. Eye candy? That's the last thing we need when the program is already so very painful.
The previous measurement had 6 significant digits of precision. They just happened to be inaccurate.
Note that the new estimate seems to have *less* precision (assuming that only the first 4 digits are significant), but is claimed, at least, to have more accuracy.
Just for an example of poor quality (though opposite to your comments about convolution), I was once handed a paper (I don't recall the journal or author, unfortunately, as this was several years ago) on fault detection in distributed networks.
The entire point of the paper was "If you send a request and you don't get a response back for a while, something probably went wrong." I read over it a couple times, hoping I was missing something that actually had substance to it. No luck.
That said, I don't think the academic community is entirely full of crap, or just riding on coattails. I do think a lot of that goes on, though, and I think it really pollutes the overall signal/noise ratio in the related journals- and from a distance, it does tend to just blur together into "crap".
Take a look at the latest issue of Computer- there's an interesting (if technically fairly light) look at performance ratios of various multicore designs- the present "cpu surrounded by lots of simple processing cores design, as implemented by a CPU + GPU combination, turns out to have the best performance/space and performance/watt ratios.
Sure, there are some situations where you can do massive parallelization but each individual thread needs the flexibility of a full CPU, but there are at least as many, if not more, situations where most or all of those threads just need simple serial execution over the data. The current model does wonderfully for that.
I say this, btw, as a programmer who does a great deal of data crunching. I'm very much looking forward to OpenCL's widespread availability, especially under linux- I've got a system in place now that could greatly benefit from it.
That's actually an interesting way of looking at it; I don't think I'd ever thought of refactoring as algebra on programs, but you're right, there are a lot of similarities- making structural changes that do not change the result/meaning.
That said, we're getting a little far afield of what the original point was- namely, that algebra is not a requirement for programming; while refactoring is often very helpful, I'm not sure I'd call it a core programming requirement or a necessary beginning programming skill. Far be it from me to argue that algebra and refactoring are not *useful*, but I think that's a bit of a different discussion.
Slashdot Mirror
Yes, we do.
Then again, this is slashdot. Since when have most commenters *ever* RTFA?
If you don't know how your library deals with thread safety, then you either need a new library or you need to read the docs.
The single most important rule of threading is to have very well defined data interfaces, and never violate them. That's it. Claiming that threading is "toxic waste" is, I'm sorry, simply wrong.
Everything is moving toward more threading, not less- we need programmers who can begin thinking in that mode rather than those who just write it off. Threading doesn't *have* to be complicated unless it's in a design that's screwed up from the start and there's insufficient data abstraction and isolation. Once there is, if your design as simple and consistent synchronization requirements, following them should be trivial. If it's not, you've screwed up in the design phase.
The fact that you *can* screw up threaded code is no reason not to use it. You can screw up a lot of things in code that won't be caught in testing or at compile time, but we do them because they give benefit or because they're necessary. Threading's the same way.
It's really not that hard if you've got a decent design. Believe me, I've done a lot of it. Is it trivial? No. But any programmer worth his salt at this point should know how to do it.
The "threaded programming is hard!" whine is what's got us into the situation where there's a disconnect between the hardware we have available and the design of the software.
It really isn't that hard. If you have your data reasonably managed and your system reasonably abstracted, then determining the portions that can safely run on other threads (or safely with some synchronization) should be easy. If it's not, your design is borked to begin with and should be rethought.
Debugging threads really isn't that bad, either, tbh. I've spent more time tracking down obscure memory issues in c/c++ code than I have tracking down threading issues in any language.
This is very true, and I think this is part of the reason that many current applications have such poor performance- they get to the end stage and realize that they can't fix the problems without more resources than they have left, so they just throw up their hands and say "screw it".
I don't entirely agree with you here. A lot of current applications *do* suffer from CPU-induced latency after user interactions, and the problem is simple: they don't differentiate between the things that must get done before control is returned to the user, and the things that need to happen in response to the action but can be allowed to happen whenever resources are free. Even when the problem is resource-access latency, multithreading can be a win because that latency no longer contributes to the latency that the user perceives if it happens on a background thread.
Something as simple as tossing function calls off on a background thread to deal with some of these tasks would do a great deal to improve latency from the user's perspective, and is really quite trivial to implement. Most programmers don't do it, though. Part of that is that in most situations there aren't ready-made solutions- you can't just say "run this function call on a background thread", you've got to go through the pthread creation process, etc. (Apple's Cocoa framework is actually an exception to this with it's NSOperation).
The situation is analogous to that of an interrupt task: Do absolutely as little as possible before returning; everything else should happen on some other thread.
I agree with you regarding optimization, but it's been my experience that many applications *can* benefit from these sorts of simple multithreading techniques- the programmers just don't do them, either from lack of ability or lack of resources.
This is true to a point. The problem is that, in modern applications, when the user *does* do something there's generally a whole cascade of computation that happens in response- and the biggest concern for most applications is that app appear to have short latency. That is, all of that computation happens as quickly as possible so the app can go back to waiting for user input.
There's a lot of gain that can be gotten by threading user input responses in many scenarios. Who cares if the user often waits 5 minutes before input? When he *does* do something, he wants it done *immediately*. The fact that it's a tiny percentage by wall time doesn't change the fact that responsiveness here is a massive percentage of user perception.
The first link from Apple Insider is definitely a bit on the biased side. The second link, though, is to MarketWatch, and is a little better on the fact/rant ratio.
Uh? You can run linux on a mac just fine. Not the iphone, true, but the others, yes.
Not to mention that Darwin, the unholy marriage of mach and BSD that mac os x is built on, *is* open source. It's the graphics layer on top that's not, and that's not built on anything open source anyway.
Apple actually releases quite a bit of open source code. You need to get a better handle on the actual facts.
I'll agree wholeheartedly with that. In the meantime, though, people shouldn't ignore it simply because it has some flaws- time has shown that the advantages of POSIX far outweigh any such existing flaws.
What does "language" have to do with it? Software authors have to pay attention to many standards beyond that of the language they're working in. If they're working with XML, they should pay attention to the XML standards. If they're interacting with an operating system, they need to pay attention to the standards that the OS is written to- in this case POSIX.
The application authors don't need to know *how* the operating system implements the standard- they just need to know what they can and can't expect from those standardize functions.
This really is an ext3 issue, not "fsync()".
The above says it all.
The fact that ext3 was broken in this regard is a performance bug in ext3.
Also, it is very much necessary on ext3. The commit delay was short on ext3 (5 seconds), but a *lot* can happen in 5 seconds, and a lot of data can still be lost in that time period.
There's exactly one thing to take away from this. If you want data to be guaranteed to be on disk, call fsync(). You'll note that Linus doesn't at all disagree with this- what he's basically saying is that "ext3 implements fsync() in a braindead way that makes in painful for the whole system if people do the right thing."
That's not an API issue. That's an ext3 issue, and then it's an issue with application developers coding with the expectation of ext3.
Every scenario where data integrity is of importance to the user.
Note that this is not every scenario. Many, many programs use temporary files for storage, and in those cases, the corruption of those files during a crash is of no consequence. I've written and used applications like this in the past.
At the same time, I've written and used programs for which I could not countenance data corruption. And, for at least the programs I choose to use, those programs take the necessary measures to prevent it.
Consider an application like PostgreSQL. It doesn't much care what it's running on- because it uses the API appropriately, it can get verifiably correct behavior on *any* operating system (barring some hardware that actively lies- no app can be expected to be correct in the face of things that lie to it).
I *am* a developer, including of software that requires data guarantees. I have no sympathy for application developers who don't take such considerations and then complain when some other software doesn't compensate for their own bugs.
Apparently, you don't know how to *deal* with real life.
POSIX *does* tell you what happens if your OS crashes. It says "as an application developer, you cannot rely on things in this instance." It also provides mechanisms for successfully dealing with this scenario.
As for fsync() being a performance issue, you can't have your cake and edit it too. If you don't want to pay a performance penalty, you can lose data. Ext4 simply only imparts that penalty to those applications that say they need it, and thereby gives a performance boost to others who are, due to their code, effectively saying "I don't particularly care about this data" - or more specifically, "I can accept a loss risk with this data."
Normal applications have a reasonable expectation that the OS doesn't crash, yes. And usually it doesn't. Out of all the installs out there... how often is this happening? Not very. They've made a performance-reliability tradeoff, and as with any risk... sometimes it's the bad outcome that occurs. If they don't want that to happen, they need to take steps to reduce that risk- and the correct way to do that has always been available in the API.
As for forgetting POSIX... it's the basis of all unix cross-platform code. It's what allows code to run on linux, BSD, Solaris, MacOS X, embedded platforms, etc, without (mostly) caring which one they're on. It's *highly* relevant to the real world because it's the API that most programs not written for windows are written to. Pull up a man page for system calls and you'll see the POSIX standard referenced- that's where they all came from.
Saying "Forget POSIX. It's irrelevant in the real world." is like people saying a few years ago "Forget CSS standards. It's irrelevant in the real world." And you know what? That's the attitude that's dying out in the web as everything moves toward standards compliance. So it is in this case with the filesystem.
You're right. The correct thing to do is to *always* call fsync() when you need a data guarantee, *regardless* of which FS you're on. The fact that not doing it in the past hasn't caused problems isn't the problem- those calls are the correct way of handling things.
I disagree. "Writing software properly" apparently means taking on a huge burden for simple operations.
No. Writing software properly means calling fsync() if you need a data guarantee.
Pretty sure no one in their right mind would call using fsync() barriers a "huge burden". There are an enormous number of programs out there that do this correctly already.
And then there are some that don't. Those have problems. They're bugs. They need to be fixed. Fixing bugs is not a "huge burden", it's a necessary task.
Nothing- except that it's not in the spec.
POSIX is like a contract. KDE is breaking the contract and then whining about it to ext4- which isn't breaking the contract. Just as in a court, KDE here doesn't have much of a leg to stand on.
They are.
The only calls that say data is written are the fsync() family (or files opened with O_SYNC.
Those calls do not lie.
Unfortunately, application developers are assuming that other calls say something they do not, in fact, say. This is where the problem comes in. close() does not guarantee that data has been written yet. fsync();close(); does.
This is the attitude that has the web stuck with IE.
There's a standard out there called POSIX. It's just like an HTML or CSS standard. If everyone pays attention to it, everything works better. If you fail to pay attention to it for your bit (writing files or writing web pages), it's not *my* fault if my conforming implementation (implementing the writing or the rendering) doesn't magically fix your bugs.
No. It's not.
If what you say is true there would be no need for the fsync() function (and related ones).
Read the standards if you want. The filesystem is only bugged if it loses recent data under conditions where the application has asked it to guarantee that the data is safe. If the app hasn't asked for any such guarantee by calling fsync() or the like, the filesystem is free to do as it likes.
... is the one feature I never see under OpenOffice release notes: Improved performance.
I keep trying OpenOffice, under multiple OSes... and I keep removing it in frustration. Eye candy? That's the last thing we need when the program is already so very painful.
Precision != Accuracy.
The previous measurement had 6 significant digits of precision.
They just happened to be inaccurate.
Note that the new estimate seems to have *less* precision (assuming that only the first 4 digits are significant), but is claimed, at least, to have more accuracy.
Um, we're talking about Microsoft here. It might be an *expensive* hotel, but I'm not sure I'd call it a *nice* hotel...
Just for an example of poor quality (though opposite to your comments about convolution), I was once handed a paper (I don't recall the journal or author, unfortunately, as this was several years ago) on fault detection in distributed networks.
The entire point of the paper was "If you send a request and you don't get a response back for a while, something probably went wrong." I read over it a couple times, hoping I was missing something that actually had substance to it. No luck.
That said, I don't think the academic community is entirely full of crap, or just riding on coattails. I do think a lot of that goes on, though, and I think it really pollutes the overall signal/noise ratio in the related journals- and from a distance, it does tend to just blur together into "crap".
Take a look at the latest issue of Computer- there's an interesting (if technically fairly light) look at performance ratios of various multicore designs- the present "cpu surrounded by lots of simple processing cores design, as implemented by a CPU + GPU combination, turns out to have the best performance/space and performance/watt ratios.
Sure, there are some situations where you can do massive parallelization but each individual thread needs the flexibility of a full CPU, but there are at least as many, if not more, situations where most or all of those threads just need simple serial execution over the data. The current model does wonderfully for that.
I say this, btw, as a programmer who does a great deal of data crunching. I'm very much looking forward to OpenCL's widespread availability, especially under linux- I've got a system in place now that could greatly benefit from it.
That's actually an interesting way of looking at it; I don't think I'd ever thought of refactoring as algebra on programs, but you're right, there are a lot of similarities- making structural changes that do not change the result/meaning.
That said, we're getting a little far afield of what the original point was- namely, that algebra is not a requirement for programming; while refactoring is often very helpful, I'm not sure I'd call it a core programming requirement or a necessary beginning programming skill. Far be it from me to argue that algebra and refactoring are not *useful*, but I think that's a bit of a different discussion.