Read The Seven Deadly Sins of Measurement by Jim Campy
Measuring risk means walking a thin line. Balancing what is highly unlikely from what it totally impossible. Financial institutions need to be prepared for the highly unlikely but must avoid getting sucked into wasting time worrying about the totally impossible.
Here are some sins that are sometimes committed by risk measurers:
1. Downplaying uncertainty. Risk measurement will always become more and more uncertain with increasing size of the potential loss numbers. In other words, the larger the potential loss, the less certain you can be about how certain it might be. Downplaying uncertainty is usually a sin of omission. It is just not mentioned. Risk managers are lured into this sin by the simple fact that the less that they mention uncertainty, the more credibility their work will be given.
2. Comparing incomparables. In many risk measurement efforts, values are developed for a wide variety of risks and then aggregated. Eventually, they are disaggregated and compared. Each of the risk measurements are implicitly treated as if they were all calculated totally consistently. However, in fact, we are usually adding together measurements that were done with totally different amounts of historical data, for markets that have totally different degrees of stability and using tools that have totally different degrees of certitude built into them. In the end, this will encourage decisions to take on whatever risks that we underestimate the most through this process.
3. Validate to Confirmation. When we validate risk models, it is common to stop the validation process when we have evidence that our initial calculation is correct. What that sometimes means is that one validation is attempted and if validation fails, the process is revised and tried again. This is repeated until the tester is either exhausted or gets positive results. We are biased to finding that our risk measurements are correct and are willing to settle for validations that confirm our bias.
4. Selective Parameterization. There are no general rules for parameterization. Generally, someone must choose what set of data is used to develop the risk model parameters. In most cases, this choice determines the answers of the risk measurement. If data from a benign period is used, then the measures of risk will be low. If data from an adverse period is used, then risk measures will be high. Selective paramaterization means that the period is chosen because the experience was good or bad to deliberately influence the outcome.
5. Hiding behind Math. Measuring risk can only mean measuring a future unknown contingency. No amount of fancy math can change that fact. But many who are involved in risk measurement will avoid ever using plain language to talk about what they are doing, preferring to hide in a thicket of mathematical jargon. Real understanding of what one is doing with a risk measurement process includes the ability to say what that entails to someone without an advanced quant degree.
6. Ignoring consequences. There is a stream of thinking that science can be disassociated from its consequences. Whether or not that is true, risk measurement cannot. The person doing the risk measurement must be aware of the consequences of their findings and anticipate what might happen if management truly believes the measurements and acts upon them.
7. Crying Wolf. Risk measurement requires a concentration on the negative side of potential outcomes. Many in risk management keep trying to tie the idea of “risk” to both upsides and downsides. They have it partly right. Risk is a word that means what it means, and the common meaning associated risk with downside potential. However, the risk manager who does not keep in mind that their risk calculations are also associated with potential gains will be thought to be a total Cassandra and will lose all attention. This is one of the reasons why scenario and stress tests are difficult to use. One set of people will prepare the downside story and another set the upside story. Decisions become a tug of war between opposing points of view, when in fact both points of view are correct.
There are doubtless many more possible sins. Feel free to add your favorites in the comments.
But one final thought. Calling it MEASUREMENT might be the greatest sin.