The Perils of Simplifying Risk To a Single Number

A few weeks back we discussed the perspective that the economic meltdown could be viewed as a global computer crash. In the NYTimes magazine, Joe Nocera writes in much more depth about one aspect of the over-reliance on computer models in the ongoing unpleasantness: the use of a single number to assess risk. Reader theodp writes: "Relying on Value at Risk (VaR) and other mathematical models to manage risk was a no-brainer for the Wall Street crowd, at least until it became obvious that the risks taken by the largest banks and investment firms were so excessive and foolhardy that they threatened to bring down the financial system itself. Nocera explores the age-old debate between those who assert that the best decisions are based on quantification and numbers, and those who base their decisions on more subjective degrees of belief about the uncertain future. Reliance on models created a 'false sense of security among senior managers and watchdogs,' argues Nassim Nicholas Taleb, who likens VaR to 'an air bag that works all the time, except when you have a car accident.'"

Comments

Gladwell's "Blowing Up"

[gambit3]

For an EXCELLENT article about this, read Malcolm Gladwell's "Blowing up", which you can find online for free:

http://www.gladwell.com/2002/2002_04_29_a_blowingup.htm

It's simpler than that

[joss]

Risk models are largely irrelevant because the only risk anyone in the financial sector is really interested in minimizing is the risk that they will get fired. The way to do that is to do almost exactly the same thing as everybody else, no matter how mind blowing stupid it is. Plenty of people realized that banks etc were not nearly as sound as commonly believed years ago. Those that tried to act on this were fired long ago since they weren't making as high a ROI as those willing to invest in dodgy hedgefunds etc. Rational market my ass.

Open Source Risk Modeling

[Doofus]

Far down in the depths of the article, the author points out that JPMorgan open-sourced their risk modeling methodology, which popularized the VaR (Value at Risk) approach used by most of the big financial firms:

What caused VaR to catapult above the risk systems being developed by JPMorgan competitors was what the firm did next: it gave VaR away. In 1993, Guldimann made risk the theme of the firm's annual client conference. Many of the clients were so impressed with the JPMorgan approach that they asked if they could purchase the underlying system. JPMorgan decided it didn't want to get into that business, but proceeded instead to form a small group, RiskMetrics, that would teach the concept to anyone who wanted to learn it, while also posting it on the Internet so that other risk experts could make suggestions to improve it. As Guldimann wrote years later, "Many wondered what the bank was trying to accomplish by giving away 'proprietary' methodologies and lots of data, but not selling any products or services." He continued, "It popularized a methodology and made it a market standard, and it enhanced the image of JPMorgan."

Re:Math?

[El Torico]

After seeing the rampant fraud committed by the global financial elite, I'm very inclined to agree with you. What we need isn't just a number that quantifies risk, but also a number that quantifies trust.

I would pay for a service that tracks every person involved in business that was ever convicted, under indictment, or subject of a complaint. It should also track which firms employed them and where they are working now. It should also cover which "civil servants" were "on watch" at the time.

Taleb doesn't know everything

[Kupfernigk]

Taleb is very arrogant. But he still cannot see beyond his limited perspective as a quant. He is right in arguing that the fundamental error in the model was to assume that the binomial distribution works for everything, but there also seems to have been a "conservation" error - assuming that risk scaled linearly with the axes. Any statistician with experience knows that reliance can only be placed on the outliers of a distribution when there is enough data around those outliers.

As an example, suppose that the distribution suggests the chance of losing 50 million dollars is +3 sigma for some measure. The problem is that there is a subtle effect - say panic, herd effect or some interaction of derivative models - which only becomes significant around the 3 sigma mark. The result could be that the exposure at a 4 sigma event is billions of dollars. A proper risk model would need to take this into account

My conclusion based on what I have read so far is that the physicists (in particular) involved in developing quantitative models would have benefited from a lot more exposure to real world experiment. They would then have had more of a clue about the unreliability of data away from the mean, scatter, and the importance of the fact that in physics subtle errors turn out to be signs that the model is wrong - e.g. relativistic effects only become important at a significant fraction of c.

Model accuracy wasn't the only problem

[EdwinFreed]

A friend of mine is a risk assessment quant who was working at Lehman right up to the point where they declared bankruptcy. I asked him about this article the other day. He said that their models started telling them something was very wrong back in 2007. The problem was that Fuld (the CEO) refused to believe what the models were saying.

The most accurate model in the world won't help if you don't pay atention to the results it produces.

There's also apparently an issue with the classical VaR models depending on transparent pricing, which these real estate instruments lack. So some of the most troublesome assets apparently weren't in the model.

Re: I don't think

[Hemogoblin]

You might think that if you have three investments with a 10% risk of losing £1,000,000 the chances of all three of them losing £1,000,000 is 0.1*0.1*0.1 = 0.001 or 0.1%.

No, no-one who actually calculates and uses VaR thinks that. Anyone who has done any statistics, like all finance quants, will correctly take into account covariances. The actual problem is the interpretation of the "correct" VaR, and relying on it too heavily.

I'll give you the actual definition of VaR. If you calculate the VaR(10 day, 5%) to be $100,000, this means that there is a 5% chance that the loss on your portfolio over a 10 day period will be larger than $100,000, or that your profit will be larger than $100,000 assuming a symmetric distribution. It's when people think "Oh that's great, we can ONLY lose $100,000" when you have a problem. The actual loss could be ANY value larger than $100,000.

It's hardly a perfect statistic, since there are still many assumptions involved. However, it's still a decent estimator and it's better than making a wild guess based on gut feelings. Despite what most people currently believe, a lot of brainpower has gone into developing financial theories and some stuff is pretty damn good. The financial industry deserves some bashing, but it frustrates me when people spread incorrect information; at least complain about the right things.