Model risk management: Evolving challenges, advancing technology

By Michael J. Budinger, CAMS, and Ryan Michalik, CFA
| 6/19/2020
Model risk management: Evolving challenges, advancing technology

In recent years, model risk management (MRM) has become an increasingly critical priority for banks and other financial services organizations. At the same time, as the use of models expands, having a robust, ongoing monitoring framework becomes that much more important.

As MRM challenges continue to evolve, many banks are applying important new tools and practices that can help them manage and monitor this important risk function more effectively.

The current state of model risk management

The practice of model risk management has undergone some significant changes in the years since the Federal Reserve and the Office of the Comptroller of the Currency issued “Supervisory Guidance on Model Risk Management” (SR 11-7, OCC 2011-12). At the time that the interagency guidance was first published, the use of sophisticated financial models was concentrated primarily in large institutions, particularly those with more than $10 billion in assets, which were subject to heightened regulatory scrutiny. 

In the ensuing years, however, attention to model risk management became much more widespread as model usage continued to increase in banks of all sizes for a variety of purposes including compliance with rapidly changing regulatory and financial reporting requirements. When the Federal Deposit Insurance Corporation adopted the interagency guidance in 2017, a wider universe of banks recognized the need to improve their MRM capabilities. Advances in models such as incorporation of advanced statistics and machine learning also spurred greater need for efficient management of the associated model risks.

Today, the maturity levels of MRM programs vary considerably across the industry. Many banks with less mature programs are still identifying and refining their model inventories or working their way through the first or second round of validations of key models – a function typically outsourced or supplemented by support from internal audit. In larger, more mature programs, however, management’s focus is shifting toward gaining more value from the effort, realizing efficiencies, developing a holistic view of the organization’s model risk, and enhancing internal resources in order to build more sustainable MRM practices. 

Evolving model risk management practices

At a fundamental level, model risk management encompasses a range of activities within three overarching functions:

  1. Model development, implementation, and use. This function encompasses a clear understanding of the model’s purposes, support for the theory and design of the model, rigorous assessment and preparation of data, a disciplined approach to implementing the model in the bank’s environment, comprehensive testing, and challenge by model users.
  2. Model validation. This function consists of a set of processes and activities, conducted by a party that is independent from model development, which are intended to verify that models are performing as expected.
  3. Governance, policies, and controls. A governance framework provides the necessary policies, procedures, and management oversight to verify that the MRM program is appropriately managing model risk, in alignment with the extent and sophistication of the bank’s usage of models.

MRM practices within these functions have continued to evolve to enhance the MRM function. They include:

  • Ongoing monitoring
    • A standardized manner for model owners to document an ongoing monitoring plan
    • Tracking of the execution of ongoing monitoring activities by the MRM function
    • Visibility to model performance issues and an understanding by model risk managers of the impact to interconnected models
    • Aggregated reporting on ongoing monitoring compliance and model performance
  • Change management
    • A standardized method for documenting changes
    • A clear definition of what constitutes a material change
    • Notification of material changes to model risk management
  • Nonmodel tools
    • Identification of nonmodel applications or end-user computing systems (EUCs) and inclusion in the model inventory
    • Established policies and procedures to provide oversight
  • Model validation
    • A standardized model validation report
    • An established set of risk-based activities to conduct between full validations
    • A centralized manner for tracking validation findings and timelines for remediation

Advancing the MRM function to drive business value while realizing efficiencies can be challenging. A successful program requires that MRM teams be able to address both rising volumes of model usage and evolving requirements and expectations by model users and regulators alike.

Addressing the challenge of ongoing monitoring

Among the various challenges that MRM teams must address, issues associated with ongoing monitoring often are among their most immediate concerns. For example, when hundreds of bank executives participating in a recent Crowe webinar were asked to identify their top MRM-related challenges, their number-one concern involved working with model owners to conduct such monitoring. Nearly half (45.5%) said they expect this to be one of their top challenges in the coming year (Exhibit 1).

Exhibit 1: Model risk management challenges

Participants were able to select multiple responses.

As they work to address the challenges associated with implementing an ongoing monitoring program, those responsible for model risk management generally need to focus their efforts in four broad areas of concern:

1. Standardization and documentation. One of the first challenges to address in many organizations is the lack of a standardized process for model owners to use in conducting and documenting their ongoing monitoring. Model owners often lack a standardized manner to document the activities to monitor, how frequently to conduct that monitoring, and what thresholds or metrics they should use to assess model performance. Fortunately, there are technology solutions that can help facilitate these standardization and documentation activities by model owners. Exhibit 2 illustrates how one such solution provides a standardized documentation format.

Exhibit 2 : Ongoing monitoring plan documentation

Source: Crowe Model Risk Manager

2. Tracking and visibility. Model risk managers often lack basic visibility into whether and how model owners are conducting their ongoing monitoring, both in terms of the timing and sufficiency of their testing activities. Here again, technology can help both model owners and risk managers alike by providing real-time tracking of when testing is completed, and when deadlines are either met or missed. Exhibit 3 shows an example of how such information can be conveyed in a risk management solution. Integrated solutions also can send reminders to model owners about upcoming testing deadlines.

Exhibit 3: Ongoing monitoring completion tracking

Source: Crowe Model Risk Manager

3. Understanding and evaluating performance. Another critical pain point for model risk managers is the need for greater transparency and insight into the performance of the various models within their organizations. Beyond verifying compliance with the prescribed testing regimens, managers also need some form of aggregate reporting that enables them to identify which models are failing and what performance patterns can be discerned over time.

4. Identifying linkages and common sources. Almost every financial model in a bank will share at least some common data sources with other models. In addition, the outputs from one model often feed into several others. Being able to identify such linkages, interconnections, and shared data sources, as shown in Exhibit 4, is critical to both identifying model risk and improving model performance over time.

Exhibit 4: Model mapping

Source: Crowe Model Risk Manager

Although ongoing monitoring of model performance and testing is still an ad hoc process in many organizations, a growing number of banks are taking active steps to address these concerns. As they do, they inevitably seek out technology solutions that can help them in this effort.

Using technology to advance model risk management

Ironically, management and monitoring of MRM programs is still a manual process in many institutions. In the webinar mentioned earlier, participating executives were asked to identify which enabling technology solutions their organizations use to manage their MRM programs. More than half (51.7%) said they relied on a primarily manual approach, using off-the-shelf tools such email, spreadsheets, and basic office software programs (Exhibit 5).

Exhibit 5: Enabling technologies

The paradox of using conventional, manual management tools to oversee a highly advanced, technology-driven process is noteworthy. But beyond being incongruous, manual management of MRM programs is inefficient, and it exposes the organization to additional risk. Manual approaches can lack clearly defined user roles and accountabilities. They also lack the ability to build a complete, trustworthy audit trail. 

To address these shortcomings, those charged with managing the overall MRM effort generally have several alternatives available to them. In some cases, they can adapt certain systems the organization already has in place, such as existing databases, business intelligence analytics, or business process management tools. These tools support more robust reporting and visualization than manual processes, and they can help automate some of the process steps. Unfortunately, the effort required to pull together these disparate systems can be significant, requiring heavy involvement by already-burdened IT resources, along with significant time to develop and implement.

Another approach is to adapt existing governance, risk, and compliance (GRC) platforms to incorporate a designated MRM component. Taking advantage of systems that are already in use can help facilitate and standardize reporting and provide support for the general enterprise risk management use case. On the other hand, systems that are not purpose-built for model risk management often require significant reconfiguration, and even then, they might lack some necessary MRM functionality. Competing priorities across the enterprise also might mean the MRM use case is not fully supported.

Purpose-built vendor solutions developed specifically for MRM applications can overcome these disadvantages, enabling faster compliance and streamlined workflow management and automation of critical processes. Such solutions establish a centralized repository for model information and program documentation and also deliver improved database, reporting, and analytics capabilities to help all stakeholders better understand and manage model risk. Exhibit 6 illustrates how an effective solution can provide a real-time, dashboard view of an organization’s total model inventory. 

Exhibit 6: Model inventory dashboard

Source: Crowe Model Risk Manager

When selecting such vendor solutions, organizations should take care that the software they are considering is not merely a repurposed risk management or customer management system that lacks MRM-specific functionality. In addition, they should verify that the solution they choose is easily configurable to meet their specific needs.

Ideally, the system used to manage the MRM program should provide an enterprisewide view of all models and nonmodel EUC information, with full governance reporting and audit trails. It also should support automation or streamlined workflow of all critical MRM-related activities such as inventory and risk assessments, annual census, validation, development and implementation, and monitoring. An effective solution also will provide insights into model operation, risk, and interdependencies, with the ability to adjust dynamically to the bank’s current and future MRM requirements. 

As model risk management continues to grow in importance, and as the scope of MRM team responsibilities continues to broaden, an increasing number of organizations are recognizing the value of taking full advantage of the available technology while continuing to apply sound practices for this critical risk management function. In addition to supporting compliance with evolving regulatory and financial reporting requirements, effective MRM tools can enable more effective oversight of both model-related and nonmodel activities.

Contact us

Michael Budinger - Large
Michael Budinger
Principal
Ryan Michalik
Ryan Michalik