In addition to verifying that these models are meeting requirements and serving their intended purposes, the process of analyzing and validating the consistency and predictive power of risk rating models also can deliver significant added benefits to a lending organization.
In practice, some credit risk rating models are primarily subjective, and thus difficult to validate, while others are statistically or mathematically based. Many organizations today are moving toward employing some type of scorecard methodology, which combines both subjective and statistical components.
An effective credit risk rating model will take into account a variety of factors. Exhibit 1 shows a simple example of how such a scorecard might be structured for commercial lending purposes, assigning point values to various factors that reflect the relative degree of risk posed by a commercial credit customer.
Obviously, this example is illustrative only. Every organization must develop its own rating criteria, along with the relative values assigned to the various factors. The larger point is that the numerical values produced by this type of scorecard will provide a foundation for objective analysis of risk on both an account-level and portfoliowide basis. Model validation is necessary to confirm that scorecard factors are effectively capturing the risk, that methods used to develop models are conceptually sound, and that any underlying data is reasonably sourced and accurate.
After the 2008 financial crisis intensified concerns over model performance, the OCC updated its guidance in OCC Bulletin 2011-12. Working in concert with the Board of Governors of the Federal Reserve System, the more recent guidance spells out the elements of a sound program for effective management of risks associated with the use of quantitative models in bank decision-making. In addition to model validation, the 2011 guidance also addresses the underlying governance components, such as the establishment of prudent model risk management policies, documentation, and ongoing monitoring. This regulatory attention sometimes leads organizations to regard model risk management as primarily a compliance concern. However, regulatory compliance is only a small part of the picture.
For example, credit risk rating models can be used for risk-based pricing, which serve as “guardrails” to help an organization stay within its defined profitability tolerances in the pricing of its products and services. They also can provide a useful foundation for sensitivity analysis by helping management understand how portfolio risk changes when certain borrower profiles shift in either a positive or negative direction.
Credit risk rating models also figure prominently in stress testing. Many model attributes have established statistical relationships with macroeconomic factors, and might be directly affected by the prescribed economic scenarios used in the course of Dodd-Frank Act stress testing (DFAST) and Comprehensive Capital Analysis and Review (CCAR) reporting.
The coming transition to a current expected credit loss (CECL) methodology for calculating impairment adds another potential application of credit risk rating models. The lifetime probability of default (PD) and loss given default estimates that are used in a CECL-based environment might be calculated using similar methodologies deployed for credit risk rating models.
Although a bank could choose to adopt similar risk models that it uses for underwriting decisions to make its CECL loan provisioning calculations, those models must be validated separately for their new purpose in the CECL process. In addition to validating the individual models that are used to evaluate risk in various portfolio elements (such as the commercial and industrial portfolio or commercial real estate portfolio), it is important to also understand how all of these individual models are coming together in the overall calculation of reserves under the respective CECL methodologies.
As Exhibit 2 demonstrates, either type of error presents risk at both the individual loan level and portfoliowide.
If a model predicts a lower risk of default than actually occurs, the bank risks loss of principal, interest, and fees, as well as higher recovery costs and an overstatement of the fair value of the portfolio. On the other hand, predicting a higher risk of default could lead to noncompetitive bidding, loss of potential profits, and an understatement of fair value.
The model validation process and associated activities should be designed to understand the accuracy of the model to appropriately capture borrower risk. The validation process must be independent, comprehensive, and ongoing, and should be applied to all models, whether internally developed or purchased from a third-party provider.
The most effective model validation programs generally demonstrate the following three important characteristics:
Power can be presented graphically in the form of power curves, such as the example shown in Exhibit 3.
To interpret Exhibit 3, assume a credit risk rating model was used to rate 100 borrowers. If 10 borrowers actually defaulted, a perfect model would have assigned those 10 borrowers a PD that identified them as the riskiest loans in the population. By inference, the 90 borrowers who did not default would have had a lower PD than the borrowers who defaulted. On the other hand, a random model – comparable to flipping a coin – would have accurately predicted the outcomes only 50 percent of the time.
The power curves of various models are displayed by plotting the percentage of the borrowers that defaulted against the rank ordering of the PD assigned to the borrowers. The effectiveness of the models then are compared with the expected 50 percent performance of a random model. The closer the model’s curve is plotted to the perfect model, where the area between the model curve and the random curve is maximized, the stronger the power of the model. In Exhibit 3, the “good” and “better” ratings are subjective – every organization would need to define its acceptable levels of model performance as part of its model validation process.
Ultimately, the goal is to blend the qualitative art of lending with the science of a mathematical model. In this way, validation can be used as a feedback mechanism that lays the groundwork for continued improvement, not simply as a “check the box” requirement for regulatory purposes.
The root causes of many data problems can be traced to poor management of credit information, which generally reflects weak governance. This commonly manifests itself in the use of a large number of disparate systems to capture credit portfolio data, with only limited documentation of the methods that are used to move this data into the models themselves.
Another common issue is a lack of sufficient default experience – that is, the bank has only a limited history of defaulting (or bad) loans to use in developing its credit risk models. Obviously, this is a good problem to have, but it can make model development and validation more difficult. In some instances, this challenge can be addressed by applying the expertise of the lending group and by employing certain statistical analysis tools to adjust portfolio segmentation and rating scales.
For example, analyzing the distribution of risk rating scores across the portfolio and comparing this distribution against industry norms could provide insights into the relative stability of the portfolio. Similarly, a univariate analysis that compares credit scores or some other scorecard variable against the actual default rate can provide better understanding of the validity of the various scorecard elements.
Model validation also can be made more difficult for banks that use models developed by third-party vendors if those vendors are unwilling to share needed information or documentation due to their proprietary nature. Banks should make best efforts for their vendor contracts to require model providers to support and assist in the performance of needed validation activities, such as supporting Q&A sessions with validators. To the extent possible, banks also should request vendors to provide descriptive statistics of the underlying data that was used to develop the models, and to provide comprehensive documentation that meets the documentation standards that would apply to internally developed models.
By anticipating and addressing validation challenges as those just described, banks can make significant progress toward developing validation processes that assess both the power and accuracy of credit risk rating models. In doing so, they have the opportunity to move beyond a compliance-oriented approach and instead can begin to take advantage of the added benefits that effective credit risk rating model validation processes can provide.