Although Accounting Standards Update (ASU) No. 2016-13, “Financial Instruments – Credit Losses (Topic 326): Measurement of Credit Losses on Financial Instruments,” does not take effect for calendar year-end U.S. Securities and Exchange Commission (SEC) filers until 2020 and for other calendar year-end entities until 2021, early adoption is permitted for all companies in 2019. Many SEC filers will develop their new models throughout 2018, with plans for robust validation and testing to occur in early to mid-2019. It is wise to establish controls and conduct robust testing and validation over the course of several quarters of performance under the new methodology before releasing any estimates or contemplating early adoption. Regardless of when a company plans to adopt the guidance or release preliminary estimates to the market, the time to begin plotting the course to timely compliance is now, in part to allow adequate time for model validation before implementation.
Estimating Credit Losses Under the New Standard
ASU 2016-13 replaces the incurred loss model for estimating credit losses with the current expected credit loss (CECL) model. The incurred loss model generally considers only past events and current conditions. The CECL model, however, requires the immediate recording of the full amount of credit losses expected over the life of a loan, rather than waiting until the losses are deemed probable. The amount recorded will be based on relevant information about past events (including historical experience), current conditions, and “reasonable and supportable” forecasts that affect collectability.
The ASU does not impose a specific technique for estimating credit losses. Companies can exercise judgment in determining the appropriate model for their circumstances. Considering the significance of the models and the impact on financial statements and capital position, management should consider subjecting the methodology to a rigorous validation process before providing estimates or early adopting. These expectations are particularly true in light of the Supervisory Guidance on Model Risk Management1 issued by the Office of the Comptroller of the Currency and the Federal Reserve Board in April 20112 and adopted by the Federal Deposit Insurance Corporation (FDIC)3, primarily for FDIC-supervised banks with assets greater than $1 billion.
Model Risk Management Expectations
The guidance identifies the three elements a sound model risk management program should have to effectively manage the risks associated with the use of quantitative models in bank decision-making:
- Model development, implementation, and use
- Model validation
- Model governance
Because CECL-based ALLL calculations will be highly dependent on models, the expectations set forth in the guidance are certain to come up in regulatory dialogue and examinations. In addition, auditors may ask about validations and the observations derived from the validation exercise.
The CECL model development process is likely to be lengthy for many banks and financial services companies. As such, banks should consider how they approach the validation process and timing. One consideration for CECL model validation would be to insert “tollgates” (or natural break points) at significant steps during the development process to incrementally validate, as opposed to holding off until the end of development to uncover problems that otherwise could have been remediated earlier on in the process.
The tollgate process requires that validation procedures, findings, and recommendations be well documented throughout the validation process in order to maintain independence and to not unduly influence the model development.
Preparing for Validation
Validation refers to the set of activities performed by independent staff to verify that a model is performing as expected, commensurate with the conceptual design and use. The process considers all components of a model, including the conceptual design, input, processing, outputs, and reporting.
Validation is also applicable to vendor-developed models. Although many vendor models maintain proprietary information, the validation requirement does not go away. It is of utmost importance for an organization to establish why a certain vendor model is applicable for use. Often, vendor models are used because an organization lacks robust internal data history or the resources to develop models in-house. When they use a vendor model, organizations still have to consider the relevancy of the model to their portfolios and profile. For example, if a vendor model contains data for residential mortgages with a maximum loan-to-value ratio of 80 percent, but the organization originates mortgages that surpass that threshold, then the model might underpredict expected losses (in the absence of a model setting adjustment). Banks that engage vendors to provide or develop models should consider requiring the vendor to provide model documentation and to support the model validation process by addressing validator questions and requests. These requirements can be incorporated into vendor contracts as part of the third-party risk management process.
Documentation
Clear and comprehensive model documentation is critical to providing internal and external parties – including model validators – an understanding of the final model. This includes thoroughly outlining all model assumptions and limitations, including potential impacts. Limited or incomplete documentation of key model components, such as the selected methodology, portfolio segmentation, data sources, and model testing, will hinder the validation process, particularly if the documentation is in draft form. Comprehensive documentation includes providing access to relevant data sets, modeling spreadsheets or code, CECL committee presentations, meeting minutes, and reports (including reports to management) leading up to the adoption of the model.
Documentation also should include any developmental testing compiled to challenge the modeled output, including internal benchmarks. It also might include comparing modeled output to forecasts produced by challenger models. For example, at a bank that plans to adopt a probability of default/loss given default (PD/LGD) model, development might have constructed a vintage-based challenger model. While many organizations might not have the bandwidth or resources to construct challenger models, the essence of this exercise is to provide reasonable comparison of the two forecasts, taking a different approach to modeling.
A broad range of audiences (including internal bank resources, auditors, and regulators), with varying degrees of technical competence, likely will review model development documentation. When possible, more technical documentation elements should be described in simple terms. Many organizations that have incorporated disciplines from the model risk management guidance into their risk management programs have model documentation templates already in use. These templates provide a consistent and repeatable method to thoroughly document a model according to an established set of internal model risk management standards.
Portfolio Segmentation
Portfolio segmentation is more important than ever. Before developing new models or revising existing models for CECL purposes, it’s prudent to take a step back and consider the major types of risks that can affect portfolio performance. The various risk factors within a portfolio can be segmented in a number of ways, but banks and financial services companies must be able to support their segmentation choices. They might find it necessary to segment their portfolios differently than they have under the current incurred loss model.
The validation team will want to see an analysis of loan concentrations within the respective loan portfolios and will assess whether those concentrations have been substantiated thoroughly, and whether other subsegments exist that could have been developed further. For example, given their unique sources of repayment, loan term, and collateral types, commercial real estate (CRE) construction and income-producing properties within a CRE segment might warrant two different segments, rather than being lumped together in a single CRE segment. As part of the analysis to determine segmentation and subsegmentation, development might consider reviewing for the availability of data, underlying borrower risk characteristics, and loan performance to further understand portfolio segmentation candidates.
Data
Banks and financial services companies likely will seek to use new data sources to support their CECL models. The output of these models, of course, is intended for financial statement reporting, so the model development process should include strong data governance. Data consideration includes:
- Reconciling balances, charge-offs, and recoveries to the general ledger
- Documenting data lineage from source system to the model
- Testing of controls related to date by internal audit
- Supporting third-party data as appropriate for the bank or bank portfolio
In addition, the data often resides in disparate sources. For example, information for loan balances, interest rates, and payment history generally is found in the core system, but information on credit attributes like current property values, credit scores, and loan purpose usually is not, instead residing in a different system or database. Merging all of this information in a data warehouse will make it possible to obtain a comprehensive view. A bank will need to exercise strong governance over data aggregation and risk reporting, though, because the information from outside the core accounting system might not have been previously subject to audit or review procedures.
From a validation perspective, it is essential for an organization to clearly establish and document its use of model data, the flow of data from various sources, and the internal controls put in place to have confidence in the reliability of the information.
Selecting a Model Validator
During validation, banks and financial services companies are expected to demonstrate clearly through their documentation that they arrived at their adopted models using a prescriptive and well-thought-out process that considered alternatives and exhibited the appropriate levels of model governance. A robust governance process might, for example, challenge specific model variables or drivers to confirm that they are indicators of loan and portfolio behavior and make business sense. An effective challenge process to select model drivers can be done in conjunction with model development and the applicable business unit. Additionally, forecasts produced by development should not be taken at face value. Effective challenge does not occur when model development is done in isolation. Similarly to how many organizations evaluate the development of stress-test models and forecasts under the Dodd-Frank Wall Street Reform and Consumer Protection Act, regulators want to see a process with appropriate levels of oversight and challenge among all relevant bank parties.
It’s essential, therefore, that validation is conducted by an independent party who was not involved in model development but who possesses the necessary skills and experience to effectively challenge model development. Some banks have formal model risk management departments, but the staff in those departments do not necessarily have the requisite validation experience or thorough knowledge of the new CECL standard. Moreover, if the selected model employs quantitative techniques, the validation team will need experience in statistics and quantitative concepts. The validation team also must be well-versed in the particular subject area (for example, credit risk or various loan products). As a result, some banks will look outside of their organizations and use third-party consultants to conduct or manage their validation processes.
Audits Alone Won’t Cut It
Validation not only is a regulatory expectation but is a key risk management practice that banks and financial services companies should consider prior to placing any model into production. It might be tempting to rely solely on an audit to challenge a CECL model, but proper validation will challenge a model in a multitude of ways that differ from the intent of an audit for financial statement purposes.
1 OCC Bulletin 2011-12, April 4, 2011, https://www.occ.gov/news-issuances/bulletins/2011/bulletin-2011-12.html
2 The guidance specifically states: "Statement of applicability to institutions under $1 billion in total assets: It is not expected that this guidance will pertain to FDIC-supervised institutions with under $1 billion in total assets unless the institution's model use is significant, complex, or poses elevated risk to the institution." https://www.federalreserve.gov/supervisionreg/srletters/sr1107.htm
3 FDIC Financial Institution Letter 22-2017, June 7, 2017, https://www.fdic.gov/news/news/financial/2017/fil17022.html
