Slashdot Mirror
Fooled by Randomness
The main topic of the book is the fact that all humans are simply terrible at judging probabilities. Taleb is a securities trader, so a lot of the book revolves around financial probabilities, but his argument is mainstream and requires absolutely no knowledge of the markets. The book details examples of people wildly misjudging risks and probabilities in many contexts. Often that misestimation is understandable in advance of certain events, but harder to excuse after they've occurred; Taleb hits pretty hard on what he calls "data snoopers," his term for people who back-fit theories to existing data.
One of the most notorious examples is the Bible code (which has been thoroughly debunked), but Taleb argues that analysts who spend their time trying to find patterns in historical market data are no different: if you try long enough and hard enough, you will unavoidably find apparent regularities, which can be extremely compelling when seen in isolation. In context, though, they dissolve into nothing but meaningless statistical anomalies. Taleb rightfully compares these searches for meaning to the famous monkeys and typewriters parable.
Taleb's best example of poor probability intuition is probably the infamous survivor bias, which is our tendency to be disproportionately impressed by success. We almost always ignore the fact that, for one success story, there are many failures. But we seldom hear about the failures (just like we never hear about the many theories that didn't fit the data). So it's all a game of numbers: out of 10,000 traders, a few are statistically bound to be successful, even if they are nothing more than lucky idiots. The fact that they succeeded does not mean anything. It doesn't mean that they are bad traders, but it doesn't mean that they are good traders either, because on average somebody had to succeed.
One of Taleb's hot buttons is that people tend to be too confident in what they know. He argues convincingly that we should take everything, including science, with a grain of salt. Writing about Karl Popper, he points out that there are only two kinds of scientific theories: those that are demonstrably false, and those that are not yet demonstrably false. An irksome (but sadly true) observation, yet most people behave as if what they know is eternal truth. One could of course argue that Popper's observation is but another kind of truth, but I tend to put a lot more trust in people who question what they know than in people who don't.
Another of Taleb's peeves is the human tendency to over-attribute every random event (the old post hoc, ergo propter hoc). For instance, a commentator saying that "the Dow went down ten points today on concerns about Iraq" is talking nonsense: there is no way anyone can tie such a small market move to any particular reason. I found this specific point (which in retrospect is blindingly obvious) especially enlightening, as I am embarrassed to admit that, until now, I just accepted those market comments at face value.
Taleb also has some fun at the expense of economists and analysts, especially those whose predictions turned out wrong, but who claim that the theories were in fact right, and that the facts simply weren't supposed to be that way. This is what he calls denial of history, and is common among investors and gamblers (the two being of course close cousins).
The style of the book is informal and funny, and often meandering. We hop from one topic to the next, which occasionally may detract from the book's continuity, but overall the author's points come through loud and clear. Ironically for a man who advocates self-doubt, Taleb is starkly self-confident, though not in an irritating way.
Taleb is an intriguing, multi-cultural, iconoclastic character that has been around Wall Street for a while, and now runs his own small firm. Malcolm Gladwell (author of The Tipping Point, an absolute must-read for anyone who owns a brain) has written an excellent article that shows how Taleb's reasoning runs counter to just about every bit of conventional Wall Street wisdom. If you're interested in the markets, especially derivatives, and how Taleb trounces most of Wall Street's voodoo doctors, this moderately technical interview from 1996 is worth reading too.
Overall, a warmly recommended book.
You can purchase Fooled by Randomness from bn.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
Anyone else think of the Slashdot lameness filter when reading this?
I work at the crossroads between finance and technology so I eagerly read this book. I agree that many, many people in finance are particularly susceptible to making statistical errors. It is amazing how many people are heavily rewarded for dumb luck or what amounts to survivorship bias. There are some smart people in this industry, but there are more people who have had a lucky run.
What ruined the book for me was Nassir's incessant and shameless self-promotion. He shows the same hubris that he finds reprehensible in others and he seems to be trying to settle a few old scores by maligning people.
But the most ridiculous part of the book is when he starts drawing links between finance and neuroscience and trying to explain traders behavior based on our very preliminary understanding of brain physiology. Utter crap.
The basic message is sound: Wall Street is full of people getting paid millions who haven't the slightest understanding of math, science or technology. But that message is clear in about 10 pages and the rest is self-serving, self-promoting tripe.
Anyone who has read and understood a half-decent statistics book is already aware of selection bias. The old "post hoc ergo propter hoc" fallacy is something that everyone with a brain should know about, but unfortunately few do.
So, in short, if you're ignorant, this book is for you. If you have a decent education, you can skip it.
I am a conservative. I don't think that *everyone* is where they are because they deserve it; just that there is a definate relation between action and reaction. So not everyone is in the position they are in because it's what they deserve, but they're not all in it because of bad luck either. I'm a firm believer that overall, hard work will improve your situation.
I think the truth is more complex than your cardboard liberals or conservatives would have us believe.
:) )
The truth is that while some people may become successful or suffer terribly largely through luck, there are situations where you can drastically improve or destroy your life through your own choices.
You can't blame everything on circumstance, but you can't take credit for every good thing that happens to you, either.
That's why I can't stand cardboard conservatives or cardboard liberals. The world is so much more complicated than either of them give it credit for.
(Yeah, I know, off-topic
Not representing or approved by my company or anybody else.
Um, yes, that was a super reductionist post. Obviously, there are shades of grey, and even the most hardcore C or L doesn't look at things in a 100% absolute way that I've suggested.
They are just generalizations, and it's too bad that I'm gunna get modded down to hell for it.
I probably should have left off that last line. It's still a valid point, for the most part.
"Old man yells at systemd"
...there are only two kinds of scientific theories: those that are demonstrably false, and those that are not yet demonstrably false.
;)
Well, yeah - that's why they are theories and not laws. The same sort of argument is made by fundamentalists against the "Theory" of Evolution. It can't be proven. It's only a theory (so let's go back to something infinitely less provable/disprovable such as the divine). Theories are meant to be correct in so much as there is overwhelming evidence toward their ideas.
The book appears to make some good points, however. (Just from the synopsis provided.) The infinitude of random factors that may cause market fluctuations makes prediction completely (probably?) intractable. In CS lingo, it would appear to be NP-Complete.
Conservatives tend to think you are where you are because you deserved it. Dumb and lazy people are poor, smart and hard working people are rich. Dumb people get hit by trains, smart people comment on how dumb they are. Your situation is a result of your disposition.
Liberals go the other way. Luck, circumstance, and opportunity play huge roles in where you are. Poor people are poor because of luck and circumstance. People get hit by trains because they might have just plain been unlucky. Your situation is a result of your environment, including dumb luck.
Personally, this is the single biggest reason I can't stand conservatives. It bothers me to no end how capable they are of assuming that anybody in a bad situation is there because they deserve it.
This is why both hard line views annoy me. I imagine the truth is right smack in the middle of these two outlooks. It is just as much folly to assume everything is based on luck, and that hard work, skill, intelligence, whatever, have nothing to do with it.
Finkployd
Thats called the Just World mentality. Its not a Just World .. unfair things happen through dumb luck.
I've found this is the single biggest commenality in partiasian schools of thought:
Conservatives tend to think you are where you are because you deserved it. Dumb and lazy people are poor, smart and hard working people are rich. Dumb people get hit by trains, smart people comment on how dumb they are. Your situation is a result of your disposition.
As a conservative, I'll take the bait. I think that "deserve" is too complex an issue to really know (abused as a child, ends up as a criminal, etc.). It is a core conservative belief, however, that people's fates are in their own hands. I certainly won't sign on to that smart and hard working people automatically get rich, but I've had pretty extensive personal experience with why people end up poor:
#1 reason: being part of a couple that gets pregnant outside of marriage. If you're the woman, you now have the financial burden of a child without the full support of a father. Additionally, you don't enjoy the economies of scale enjoyed when two or more income earners live together. If you're the father, the latter point still applies, and you have child support to pay.
#2 Reason: not completing high school. In our society, for better or worse, this is a signal to all employers that you haven't got your shit together. There are, however, many cases where high school dropout succeed as entrepreneurs.
#3 Reason: Drug / alcohol abuse. No explanation necessary, presumably.
Here's the defining characterstic of a conservative: the belief that if someone works hard and consistently, they can improve their condition. Not everyone who believes this is a conservative, but if you don't believe it, you can't call yourself a conservative. To address your definition of a conservative, I would say that conservatives believe that in almost every circumstance, people are poor as a consequence of poor choices they made.
So, I'm one of the kind of people you can't stand. I suspect that you don't know me very well. I believe that power over one's circumstances comes from persistent personal action and not from authority. Do people end up with bad luck out of no action of their own? Of course, everyday. But is that the defining circumstance of human existence, or is persistence, faith and audacity?
If your bitterest enemies are people who hack the heads off civilians, then I would say you're doing something right.
So hard work does nothing, eh? It's all luck and circumstance. Pfff! Every time I've stopped worrking my situation got worse, and every time I started working again, it got better. What luck I must have had lately, since I've been doing better the whole 8 years I've been working where I am. I've even been lucky enough to have people pay me when I do side work for them on weekends!
Are you on crack ?
This is my favorite quote above:
People get hit by trains because they might have just plain been unlucky
Provably false. People get hit by trains because they're on the railroad tracks when the train goes by. This has nothing to do with luck, or circumstance. It has to do with bad decision making. Nobody's car ever "stalls out" on the tracks like in the movies. People try to "beat the train"... and fail.
If a job's not worth doing, it's not worth doing right.
So all people who work three jobs and live in cardboard boxes should just work harder?
Lars T.
To the guy who modded me down from perfect to terrible Karma - Apple haters still suck
The solution is to PICK something important that may not be meaningless and random. After all, the world just might not be meaningless and random. This is kind of like Pascal's wager. If you go around believing everything is meaningeless and random and act accordingly like it is, than you gain nothing. You'd sit home and drink beers and watch Jerry Springer all day because theoretically, you aren't any worse or better off than Mother Theresa, Bill Gates, or any other "successful" person. If, however, you pick a goal or ideal to work toward and give it meaning, your life does have meaning and value. At the very least, if others around you deem your efforts worthy, you will gain the love and affection of others (which MOST people are biologically wired to need and crave). Therefore, it is possible to gain something (the pleasure of being valued) out of the void of randomness by giving your life some purpose, even if that purpose is ultimately meaningless in the big scheme of things.
So keep playing the game of life. You've got nothing to lose and only stand to gain.
I read it on Slashdot!
<a href="http://www.joblessjimmy.com">Work is dumb and so is Jobless Jimmy.</a>
I'm just as suspicious of claims that it's all luck as I am that it's all skill.
Miko O'Sullivan
Yes yes they should. With a couple of qualifers.
8 years ago I was barely making minimum wage had a wife a kid and due to bad decision making had a degree that was basically worthless. Working two jobs and for all intents and purposes living in a cardboard box.
By making a hard choice, go into the Air Force, while in I worked hard and learned everything about anything anybody would teach me. I also looked around the world and came to realize that computers and networks were *very* big and only going to get bigger I started to insert myself into every situation where they needed someone to help out with any computer related. At the same time I started to devour every book on networking that I could get from any library that I had access to. As a result of that I got out 4 years later and worked at a crappy temp job for a few months while shopping out resumes. Since I had very little experince I got a crappy job doing tech support for a medimum sized ISP. While there I continued to work my ass off and learn everything I could and read everything I could get my hands on. As a result of doing that for the past few years I've managed to go from *very* poor to be rather well off. And it was all hard work. I have yet to meet a person, myself included, who does not waste time. Put even a small perctange of that wasted time to work and you can improve your situation. So yes they should work harder and study more.
Cypherpunks: Civil Liberty Through Complex Mathematics. Those who live by the sword die by the arrow.
I read The Fortune Sellers a few years back and it seems to be along the same lines. Basically, people who "predict" the future (be it the stock market, the weather, lifestyle trends, etc.) will highlight the year they got it right and not mention the times they got it wrong. Read it and you'll look at things differently (and never listen to long-range weather forecasts again).
I live ze unknown. I love ze unknown. I am ze unknown.
For instance, if every single person in the world is bullish on the stock market then you can bet that the markets will start to decline immediately because everyone who can get into the market already has, and there are no buyers left. So there are factors that tend to ensure that most people will lose money.
> Provably false. People get hit by trains because they're on the railroad tracks when the train goes by. This has nothing to do with luck, or circumstance. It has to do with bad decision making. Nobody's car ever "stalls out" on the tracks like in the movies. People try to "beat the train"... and fail.
.. do you really think you're going to see "Mans car stalls on tracks. Man gets out of car. Man survives!" in the paper? You only hear about the times when Man bites Dog, not Dog bites Man, even though Dog bites Man is way more common. Its a junk food news society, baby, and the last thing you want to read about in the paper is boring old reality. The benifit you get out of it is that you can skew your perception of reality any which old way you choose; but a wise and compassionate person always errs on the side of doubt rather than assumption.
Way to predictably react to my statement in a way that only serves to re-inforce my point: Your perception is your truth. Check the replies, and you'll note lots of cars stall on tracks. Or get stuck on tracks through no fault of the driver.
You simply choose to err on the side of discrediting your fellow man
"Old man yells at systemd"
Where did he say that? He actually said "Luck, circumstance, and opportunity play huge roles in where you are." Huge roles being the keywords. As someone else who replied to you said, how about being hit by a hijacked airplane while sitting in your office? Or being born as a woman or slave in Sudan? Keep working hard, maybe you won't get beaten as badly.
Random is the New Order.
No I think that it is more complex than that. Yes to some extant I was lucky in that I got into weather which was one of the fields that relied heavily on computers and networks. But of course many other people I worked with did not take advantage of the bit of luck.
I did not mean to present my case as being "simple" far from it and yes there was luck involved. My point in answer to the parent post, and granted I could have explained more, was that working harder, doing something diffrent, and taking advantage of luck when it happens, another example from my case a Cisco gawd who also happened to like me, can and almost always will allow you to improve your situation.
To me the parent seemed to be saying that working harder will not help people who are in very bad situations. I gave my example to show that working harder can help and in most cases will. So I would not call it dump luck because IMO chance favors the prepared.
And I'm not calling them lazy. But you will notice that the event that set everything in motion involved a risk and was in the short term harder than staying put would have been. People need to have the courage to change and then take advantage of the "luck" those changes can produce.
I think most people are taught that they can't improve themselves and therefore don't even try. By pointing out that in many cases you can make your luck I hope to change that perception and to make the point that no matter how bad off you are or how hard you are working at the moment you can improve.
Cypherpunks: Civil Liberty Through Complex Mathematics. Those who live by the sword die by the arrow.
> I think most people are taught that they can't improve themselves and therefore don't even try.
I agree with most of what you say.
I think my main beef was people always blaming the person for being taught that there was no use for trying to succeed. For me, thats something to empathize about, to try and help them out of. It's not a good reason in and of itself as for why we shouldn't care for their wellbeing. However, you seem to recognize this; I just suspect that many dont.
"Old man yells at systemd"
He makes some good points (yes people do overvalue success, yes people in general suck at estimating probability, yes there is uncertainty in any complex system that obscures truth to the point of making it undeciferable to a certain fact, etc...)in the work but it's taken too far. It's sensationalized deconstructionist arguements that are in and of themselves false by their own premises.
Yes an analyst talking about Iraq can't say that the market is loosing 10 points due to concerns but it takes a true idiot to believe that uncertainty over Iraq isn't affecting the market in a negative way and contributing to those 10 points. Quantifying influence is often impossible but recongnizing it is usually not that hard. It's not just about probability but also trends, and uncertainty is a bad trend in finance.
Okay...uneducated, non-trader rant over!
Maybe had none of these things gone wrong we'd have graduated high school valedictorian and now be on our way to the olympics, but as it is misfortune has befallen us so we are a crack addict on welfare. Sure, we could have done some things but how could we be expected to persevere when our life was so hopeless to begin with. It was all dumb luck's fault. The current situation could have been predicted given events.
But isn't that a little fatalistic? If events determine what we do completely then where is free will? To accept that events drive you like a car on a predictable course is to accept your role as a cog in a machine. To say that someone else is driven along a predictable course by events is to say that THEY are a cog in a machine. It is to say that they have no free will, no soul. Respect for people in 'bad situations' demands that you hold them accountable for their actions, and not write off bad behavior as being the fault of their environment. I could say that a child molester is not neccesarily a bad person because they were abused, but if I did, then that would also say that an abused child who DIDN'T abuse children was no better because they had it somehow 'easier' or else they would have done it themselves. Sorry, but spirit counts for something. A strong spirit can overcome or keep trying till it dies.
This is why I hate liberals, because they don't respect people. They don't hold anyone responsible for themselves and would gladly have government assume the responsibilty for the lives of everyone who might have something go wrong in their lives ( and that's everybody ) until there is no more reason to cultivate that spirit which makes us human and which differentiates us from each other. When the worst of us are no worse off than the best of us, what is the human race but biomass?
This is not to say that conservatives are 'all that'. They do dumbly assume that successful people are somehow 'better' than others because of their success. Much IS attributable to dumb luck. Winning big is usually only possible by taking a bad risk that happens to pay off. One horse's ass that won the lottery ( or the stock market, or ran a business ) and made it thinks he's the shiznit, while 99 other lottery ( or whatever ) players are a dollar or more poorer for their effort know they are just not very lucky.
In the end we are all subject to reality. But reality is random and unfair. Conservatives need to not always judge a book( or people ) by it's cover and be grateful ( or hopeful ) for their own streak of luck, and liberals need to expect more from people and learn that to be free you can't live in a rubber room, and that making everyone the same exposes all of society to dangerous dumb luck. Only through diversity, by not all doing the same 'approved' thing can we be strong as a people against random chance. Without the freedom to make mistakes and take the consequences of them, nobody succeeds and everybody fails because there is no possibility of success to strive for. There is no change or evolution for anybody until dumb luck ( read reality ) strikes the terrible socialist beast down and frees individuals to strive and succeed or fail naturally once more.
I'm sorry, but that is an incredibly naïve point of view. Look at the whole forest, not just the individual trees.
You fall into the same trap that I hear conservative pundits fall into all the time, of confusing a symptom for a cause. Single parenthood, high-school dropout rates, and drug abuse are symptoms of poverty just as much as they are causes. Perhaps even more so. You cannot say that these people are poor because a lot of them drop out of high school, and just leave it at that. There are reasons why so many of them drop out. Would the same group of people show such high dropout rates if they had been raised in $200,000 homes in the suburbs? Of course not.
Clearly, there are social pressures that the poor have to face which the rich do not. And while it is possible for an individual to overcome these obstacles, it is self-evident that he is more likely to fail than someone who faces fewer obstacles, or none.
Furthermore, this idea that the poor would become better off if they just worked harder is absurd. It may work for a few individuals, but it only works for them because they're the only ones doing it. Generally speaking, there is a finite amount of wealth in the world, so if you believe that the poor could collectively earn more, you have to explain whose pocket it would come out of. No matter how hard you crack the whip.
It's like saying that anyone could grow up to become a gold-medal Olympian. All else being equal, perhaps anyone could. But most of them won't, and it's not necessarily because they didn't work hard. It's simply because there is a limit to the number of available medals.