To help manage model costs and achieve a better return on their investment in econometric forecasting, many banks are finding ways to apply such modeling to purposes beyond compliance, while also turning to advanced technology solutions such as machine learning to make model development more efficient and accurate.
DFAST Model Usage: The Scope of the Challenge
The use of econometric forecast models has become a major component of most banks’ DFAST compliance efforts. A typical DFAST submission requires approximately 55 forecasted items on the balance sheet and about 43 items on the income statement. The forecasted items include projected loan balances and charge-offs across various segments of the bank’s portfolio, along with numerous measures of asset quality, the projected allowance for loan and lease losses (ALLL), other real estate owned (OREO), net interest income, various retail deposit categories, and numerous other performance metrics.
Each of these items must be forecast for each of the three Federal Reserve (Fed) macroeconomic scenarios – baseline, adverse, and severely adverse. What’s more, if the bank is owned by a separate bank holding company, the bank and holding company must prepare separate stress-testing submissions. In other words, a typical DFAST submission will involve the preparation of nearly 600 forecasted items – and some submissions require many more, as banks consider various other measures in preparing their forecasts.
To develop these forecasts, a typical DFAST program will employ a number of separate econometric-driven models. Many banks use their asset-liability management (ALM) and interest rate risk models for some DFAST compliance purposes, but the total number of models employed can range from as few as eight to as many as 70, though 25 to 30 is typical.
DFAST Model Costs: A Significant Issue
Developing, applying, and maintaining financial models is a highly specialized and technically demanding process – which means it also can be quite costly. Industry experience suggests model costs typically account for more than 50 percent of the total costs associated with DFAST compliance.
Unfortunately, banks often have only a limited grasp of the actual costs involved. For example, in a recent online webinar on DFAST model usage, bank executives were asked to compare the model-related costs their banks incurred in relation to their total DFAST compliance costs. Forty-two percent of the participating executives said they were unable to estimate model costs at all. Of those who did make an estimate, only 11 percent realized that model development and maintenance were likely to account for more than 50 percent of their total DFAST-related costs.
Exhibit 1: Estimated Modeling-Related Costs
Source: Crowe Online Survey
One factor that affects these costs is the bank’s choice of how the various models are obtained – that is, whether they are internally developed or obtained from outside sources. Other components of DFAST model costs include the costs of:
- Specialized quantitative and financial modeling personnel
- Tools and specialized technology applications
- Data subscriptions (such as S&P Global Market Intelligence, Bloomberg LP, or Moody's Analytics Inc.)
- Acquiring, managing, and maintaining other data sources
- Updating the data to reflect new Fed macroeconomic variables
- Updating models if historical data is restated
- Model stability testing as new data becomes available
- Recalibrating the model, comparing the model results, and updating the documentation
Beyond Compliance: Adding Value Through Expanded Model Use
Because of the costs involved, many banks try to limit their use of models. Unfortunately, this practice can also limit a bank’s opportunity to achieve added value through the effective use of financial modeling. Despite their costs, econometric models can simplify and bring efficiency to the compliance process, which often requires a significant time commitment from senior management. The time savings by senior management can help offset model costs somewhat.
Model-related costs can be offset even further by using models for other purposes beyond DFAST compliance. For example, many models that are developed for use in stress-testing submissions also can be used to assess the capital impact of a merger or acquisition. Recognizing this, banks can use financial modeling to support regional expansion strategies and portfolio acquisitions.
Models also can play a useful role in risk management, identifying changes in profile and risk that might otherwise not be obvious. Modeling also has potential value in the budgeting process, especially in developing multiple scenario-based budgets. Given the substantial investment involved in developing and maintaining models, deriving value beyond compliance is desirable.
Managing Model Costs: Using Technology to Improve Efficiency
Another technique many of today’s largest and most technologically mature financial organizations use to moderate the costs associated with financial modeling is to improve the overall efficiency of the effort through the effective use of technology. Advances such as machine learning and automation can help make model development and maintenance more efficient, while at the same time improving the accuracy and usefulness of the models to which they are applied.
Machine learning uses mathematical algorithms to identify patterns and relationships that would be difficult for humans to recognize or time-consuming for humans to find. It can be especially useful in improving accuracy by removing outliers and anomalies, particularly if the data contains patterns that are difficult to see.
The goal is to create a repeatable process that will produce accurate forecasts with limited human interaction, as depicted in Exhibit 2.
Exhibit 2: Machine Learning Cycle Source: Crowe Analysis
The process begins with data gathering, where subject-matter expertise is applied in the form of scenarios, bank selections, and various overrides. After this step, there is very limited human interaction in the machine-learning process. Instead, automated processes clean the data and determine which variables will drive the results. Model selection also can be automated based on credibility theory, which is then applied to produce forecasts for all applicable forecast items.
The final step in the process is to monitor results. At this point subject-matter experts again interact with the process to determine if the models are behaving as intended and that the results make sense.
In addition to enabling potential cost savings and improvements in modeling accuracy, machine learning offers other benefits. For example:
- Automation of various components allows for more reliable results.
- Efficient documentation and automatic model diagnostics can help prevent significant audit costs due to repeat work.
- Deliverables are consistent and repeatable, rather than being a one-off approach.
- Significant time reduction in the development process allows for analysts to focus time and resources on other areas.
- The refined process flow can allow for similar development in other modeling areas including ALLL, asset quality, deposit forecasts, and performance benchmarking.
Model Management Best Practices
In addition to making wise use of technology, other attributes of successful model usage include a multiyear commitment and plan, strong model risk management techniques, and effective data management and project management practices. The most successful applications of effective modeling also are characterized by strong involvement and active engagement by the individual lines of business, which can help support additional model usage – beyond compliance – to further offset costs and extend the model's value to the business.
Such practices, coupled with active challenge practices by senior management, can help banks manage model costs more effectively, achieve a better return on their investment in econometric forecasts, and make the overall model development process more efficient and accurate.