Understanding and Balance
Everything needs to balance. A requirement that management understand the model creates and equal and opposite obligation on the part of the modelers to really explain the assumptions that are embedded in the model and the characteristics that the model will exhibit over time.
This means that the modelers themselves have to actually understand the assumptions of the model – not just the mechanical assumptions that support the mechanical calculations of the model. But the fundamental underlying assumptions about why the sort of model chosen is a reliable way to represent the world.
For example, one of the aspects of models that is often disturbing to senior management is the degree to which the models require recalibration. That need for recalibration is an aspect of the fundamental nature of the model. And I would be willing to guess that few modelers have in their explanation of their model fully described that aspect of their model and explained why it exists and why it is a necessary aspect of the model.
That is just an example. We modelers need to understand all of these fundamental points where models are simply baffling to senior management users and work to overcome the gap between what is being explained and what needs to be explain.
We are focused on the process. Getting the process right. If we choose the right process and follow it correctly, then the result should be correct.
But the explanations that we need are about why the choice of the process made sense in the first place. And more importantly, how, now that we have followed the process for so long that we barely remember why we chose it, do we NOW believe that the result is correct.
What is needed is a validation process that gets to the heart of the fundamental questions about the model that are not yet known! Sound frustrating enough?
The process of using risk models appropriately is an intellectual journey. There is a need to step past the long ingrained approach to projections and models that put models in the place of fortune tellers. The next step is to begin to find value in a what-if exercise. Then there is the giant leap of the stochastic scenario generator. Many major conceptual and practical leaps are needed to move from (a) getting a result that is not reams and reams of complete nonsense to (b) getting a result that gives some insight into the shape of the future to (c) realizing that once you actually have the model right, it starts to work like all of the other models you have ever worked with with vast amount of confirmation of what you already know (now that you have been doing this for a couple of years) along with an occasional insight that was totally unavailable without the model.
But while you have been taking this journey of increasing insight, you cross over and become one of those who you previously thought to talk mostly in riddles and dense jargon.
But to be fully effective, you need to be able to explain all of this to someone who has not taken the journey.
The first step is to understand that in almost all cases they do not give a flip about your model and the journey you went throughto get it to work.
The next step is to realize that they are often grounded in an understanding of the business. For each person in your management team, you need to understand which part of the business that they are grounded in and convince them that the model captures what they understand about the part of the business that they know.
Then you need to satisfy those whse grounding is in the financials. For those folks, we usually do a process called static validation – show that if we set the assumptions of the model to the actual experience of last year, that the model actually reproduces last year’s financial results.
Then you can start to work on an understanding of the variability of the results. Where on the probability spectrum was last year – both for each element and for the aggregate result.
That one is usually troublesome. For 2008, it was particularly troublesome for any firms that owned any equities. Most models would have placed 2008 stock market losses almost totally off the charts.
But in the end, it is better to have the discussion. It will give the management users a healthy skepticism for the model and more of an appreciation for the uses and limitations of the entire modeling methodology.
These discussions should lead to understanding and balance. Enough understanding that there is a balanced view of the model. Not total reliance and not total skepticism.