Credit risk rating model validation: getting full value from an important process

By Michael J. Budinger; Ryan A. Michalik, CFA; and Scott C. Miller
Credit risk rating model validation
Models predicting the likelihood of credit default are essential tools that contribute to many areas of credit risk management – from underwriting to portfolio management to capital allocation. Validating the integrity of these risk rating models is an essential step to take before deploying them for any business use.

In addition to verifying that these models are meeting requirements and serving their intended purposes, the process of analyzing and validating the consistency and predictive power of risk rating models also can deliver significant added benefits to a lending organization.

Credit risk rating models: the fundamentals

In principle, risk rating models are designed to help assess the likelihood of default. These expected outcomes typically are presented on a numerical or quantitative scale, which enables a bank to rank and compare the relative risks associated with various borrowers. Assessing risk at the loan level provides an opportunity to aggregate risk at the portfolio level and can help quantify the risks based on the type of loan, geographic location or region, industry sector, or other variables.

In practice, some credit risk rating models are primarily subjective, and thus difficult to validate, while others are statistically or mathematically based. Many organizations today are moving toward employing some type of scorecard methodology, which combines both subjective and statistical components.

An effective credit risk rating model will take into account a variety of factors. Exhibit 1 shows a simple example of how such a scorecard might be structured for commercial lending purposes, assigning point values to various factors that reflect the relative degree of risk posed by a commercial credit customer.

Obviously, this example is illustrative only. Every organization must develop its own rating criteria, along with the relative values assigned to the various factors. The larger point is that the numerical values produced by this type of scorecard will provide a foundation for objective analysis of risk on both an account-level and portfoliowide basis. Model validation is necessary to confirm that scorecard factors are effectively capturing the risk, that methods used to develop models are conceptually sound, and that any underlying data is reasonably sourced and accurate.

Credit Risk Rating Model Validation: Getting Full Value From an Important Tool

Credit risk rating model applications

The use of credit risk rating models – and the broader subjects of model risk management and model validation – are not entirely new topics of concern within the financial services industry. As the use of internally developed capital models proliferated in the 1990s, users soon recognized the need for a sound and consistent method of validating these models’ performances. To address this need, in 2000, the Office of the Comptroller of the Currency (OCC) issued OCC Bulletin 2000-16, which outlined basic model validation principles and the agency’s expectations for a sound model validation process.

After the 2008 financial crisis intensified concerns over model performance, the OCC updated its guidance in OCC Bulletin 2011-12. Working in concert with the Board of Governors of the Federal Reserve System, the more recent guidance spells out the elements of a sound program for effective management of risks associated with the use of quantitative models in bank decision-making. In addition to model validation, the 2011 guidance also addresses the underlying governance components, such as the establishment of prudent model risk management policies, documentation, and ongoing monitoring. This regulatory attention sometimes leads organizations to regard model risk management as primarily a compliance concern. However, regulatory compliance is only a small part of the picture.

For example, credit risk rating models can be used for risk-based pricing, which serve as “guardrails” to help an organization stay within its defined profitability tolerances in the pricing of its products and services. They also can provide a useful foundation for sensitivity analysis by helping management understand how portfolio risk changes when certain borrower profiles shift in either a positive or negative direction.

Credit risk rating models also figure prominently in stress testing. Many model attributes have established statistical relationships with macroeconomic factors, and might be directly affected by the prescribed economic scenarios used in the course of Dodd-Frank Act stress testing (DFAST) and Comprehensive Capital Analysis and Review (CCAR) reporting.

The coming transition to a current expected credit loss (CECL) methodology for calculating impairment adds another potential application of credit risk rating models. The lifetime probability of default (PD) and loss given default estimates that are used in a CECL-based environment might be calculated using similar methodologies deployed for credit risk rating models.

Although a bank could choose to adopt similar risk models that it uses for underwriting decisions to make its CECL loan provisioning calculations, those models must be validated separately for their new purpose in the CECL process. In addition to validating the individual models that are used to evaluate risk in various portfolio elements (such as the commercial and industrial portfolio or commercial real estate portfolio), it is important to also understand how all of these individual models are coming together in the overall calculation of reserves under the respective CECL methodologies.

Model validation: an essential function

The purpose of model validation is to assess if a model is performing as intended and in a way that is acceptable for use. In the case of credit risk rating models, this means comparing the risk ratings produced by the model with actual outcomes. While no model will ever be perfect, the goal is to limit the overestimation or underestimation of the likelihood of default.

As Exhibit 2 demonstrates, either type of error presents risk at both the individual loan level and portfoliowide.

Credit Risk Rating Model Validation: Getting Full Value From an Important Tool

If a model predicts a lower risk of default than actually occurs, the bank risks loss of principal, interest, and fees, as well as higher recovery costs and an overstatement of the fair value of the portfolio. On the other hand, predicting a higher risk of default could lead to noncompetitive bidding, loss of potential profits, and an understatement of fair value.

The model validation process and associated activities should be designed to understand the accuracy of the model to appropriately capture borrower risk. The validation process must be independent, comprehensive, and ongoing, and should be applied to all models, whether internally developed or purchased from a third-party provider.

The most effective model validation programs generally demonstrate the following three important characteristics:

  1. Independent. Validation should be performed by an independent staff with appropriate incentives, competence, and influence.
  2. Comprehensive. All components of the process – including conceptual design, input, processing, output, and reporting – should be subject to validation.
  3. Ongoing. Validation activities should continue on an ongoing basis at reasonable time intervals once a model has gone into use and when significant changes have been made to a model.

Model validation: a closer look

Generally, there are two considerations for evaluating credit risk rating models – power and accuracy. Power refers to a model’s capability to differentiate defaulting obligors from nondefaulting obligors. Accuracy, or calibration, refers to the precision of the credit model in estimating the probability of default compared to the actual default rate.

Power can be presented graphically in the form of power curves, such as the example shown in Exhibit 3.

Credit Risk Rating Model Validation: Getting Full Value From an Important Tool

To interpret Exhibit 3, assume a credit risk rating model was used to rate 100 borrowers. If 10 borrowers actually defaulted, a perfect model would have assigned those 10 borrowers a PD that identified them as the riskiest loans in the population. By inference, the 90 borrowers who did not default would have had a lower PD than the borrowers who defaulted. On the other hand, a random model – comparable to flipping a coin – would have accurately predicted the outcomes only 50 percent of the time.

The power curves of various models are displayed by plotting the percentage of the borrowers that defaulted against the rank ordering of the PD assigned to the borrowers. The effectiveness of the models then are compared with the expected 50 percent performance of a random model. The closer the model’s curve is plotted to the perfect model, where the area between the model curve and the random curve is maximized, the stronger the power of the model. In Exhibit 3, the “good” and “better” ratings are subjective – every organization would need to define its acceptable levels of model performance as part of its model validation process.

Ultimately, the goal is to blend the qualitative art of lending with the science of a mathematical model. In this way, validation can be used as a feedback mechanism that lays the groundwork for continued improvement, not simply as a “check the box” requirement for regulatory purposes.

Overcoming common validation difficulties

The validation process is expected to come with its share of challenges, among which are understanding how model development treats missing data and recognizing the potential limitation of poor data quality. Data accuracy tends to receive considerable attention from those responsible for validating model performance – and rightly so. But other data-related questions also must be considered as part of the data challenge. Examples include fully documenting the sources of data, determining whether the data captures and reflects various economic cycles, and illustrating how the models factor in missing values in certain critical data fields.

The root causes of many data problems can be traced to poor management of credit information, which generally reflects weak governance. This commonly manifests itself in the use of a large number of disparate systems to capture credit portfolio data, with only limited documentation of the methods that are used to move this data into the models themselves.

Another common issue is a lack of sufficient default experience – that is, the bank has only a limited history of defaulting (or bad) loans to use in developing its credit risk models. Obviously, this is a good problem to have, but it can make model development and validation more difficult. In some instances, this challenge can be addressed by applying the expertise of the lending group and by employing certain statistical analysis tools to adjust portfolio segmentation and rating scales.

For example, analyzing the distribution of risk rating scores across the portfolio and comparing this distribution against industry norms could provide insights into the relative stability of the portfolio. Similarly, a univariate analysis that compares credit scores or some other scorecard variable against the actual default rate can provide better understanding of the validity of the various scorecard elements.

Model validation also can be made more difficult for banks that use models developed by third-party vendors if those vendors are unwilling to share needed information or documentation due to their proprietary nature. Banks should make best efforts for their vendor contracts to require model providers to support and assist in the performance of needed validation activities, such as supporting Q&A sessions with validators. To the extent possible, banks also should request vendors to provide descriptive statistics of the underlying data that was used to develop the models, and to provide comprehensive documentation that meets the documentation standards that would apply to internally developed models.

By anticipating and addressing validation challenges as those just described, banks can make significant progress toward developing validation processes that assess both the power and accuracy of credit risk rating models. In doing so, they have the opportunity to move beyond a compliance-oriented approach and instead can begin to take advantage of the added benefits that effective credit risk rating model validation processes can provide.

Contact us

Michael Budinger - Large
Michael Budinger
Ryan Michalik
Ryan Michalik
Scott Miller