Model Validation

All the models are wrong, but some are useful. — George Box

What is a Model

According to OCC Bulletin 2011-12¹, a model is defined as a “quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates.” From the definition, one can see that the model risk can arise from: unreasonable assumptions, wrong or inappropriate theories or techniques, and low-quality input data. Different types of models are widely used in financial institutes for various purposes, such as underwriting, investment, stress test. No model is perfect but it is important to manage the model risk to an acceptable range.

What is Model Validation

Model validation is the process of evaluating and testing models to ensure that they are accurate, reliable, and perform as expected. This process involves a thorough examination of the model’s assumptions, data inputs, calculations, and outputs. The goal of model validation is to identify any weaknesses or potential sources of error in the model, and to ensure that the model meets the business needs and is compliant with regulatory requirements. Model validation is an essential step for any business that relies on models to make important decisions.

What should be Covered in a Validation

We believe that a good model validation process should follow a structured and systematic approach to ensure that all aspects of the model are thoroughly evaluated. Here are some key steps that should be included in a good model validation process:

1. Define the scope and objectives of the model: Before beginning the validation process, it’s important to clearly define the scope and objectives of the model. This includes identifying the intended use of the model, the data inputs and outputs, and any assumptions or limitations of the model.

2. Evaluate conceptual soundness: This involves assessing the model design, the main method employed for model estimation and its core assumptions.

3. Evaluate model inputs: The data inputs used in the model should be carefully evaluated to ensure that they are accurate, complete, and appropriate for the model’s intended use. Any data quality issues should be identified and addressed.

4. Test model assumptions: The model’s assumptions should be tested to ensure that they are valid and appropriate for the intended use of the model. This may involve sensitivity analysis and scenario testing to evaluate how the model performs under different conditions.

5. Assess the impact of model limitations: No model is perfect, but it is important to understand the impacts of the limitations.

6. Verify model outputs: The model outputs should be verified to ensure that they are accurate and reliable. This may involve back testing and regular performance monitoring.

7. Validate the model implementation: The implementation should be perfectly aligned with the model estimation. This requires comprehensive tie-out between the model implementation and model outputs.

8. Document the validation process: A detailed report should be created that documents the validation process and findings. This report should include a summary of the validation plan, the testing procedures and results, the types of findings based on the degree of their impacts, and any recommendations for improving the model.

Summary

In this article we discussed the importance of the model validation and our understanding of a good model validation.

Reference

1. Sound Practices for Model Risk Management: Supervisory Guidance on Model Risk Management, April, 2011 ↩