Establishing a robust, well-documented framework for ongoing monitoring is critical to identifying and preventing model deterioration. A framework that identifies issues will serve your organization more proactively than one that only reacts once an issue has been discovered through a longer period of use.
Although ongoing monitoring should be established during model development, many organizations lack the framework and understanding to establish a comprehensive plan. In a 2020 Crowe survey, 45.5% of webinar participants stated that their top model risk management (MRM) challenge is working with model owners to conduct ongoing monitoring.
The consequences of an inadequate or overlooked ongoing monitoring framework often lurk in the future. External evaluations might trigger flags, but the most direct damage can come when models are allowed to run for significant periods without their issues being identified. This oversight could lead to extensive backtracking, corrections and – if not caught in time – poorly informed business decisions.
What belongs within a comprehensive ongoing monitoring framework? Those elements might depend on the characteristics of each model, but the following considerations should be top of mind.
The actual product of an organization’s models should receive consistent testing through applicable means such as backtesting, benchmarking, sensitivity testing, data quality testing, vendor’s testing, review of changes, model processing testing, and assessment of internal and external changes.
This form of testing can take different approaches depending on the type of model. A loan default model might directly compare actual defaults to the model’s predicted likelihood, while an anti-money laundering model might compare the number of false positives provided to a predetermined threshold.
Ongoing monitoring policies and procedures that are vague, difficult to execute, or not enforced will poorly serve model users. Everyone involved in the model process should have comprehensive, accessible answers to pertinent questions regarding monitoring activities, including:
Ongoing monitoring evaluations and recommendations are already part of our standard model validation process, but we also offer direct services to establish or bolster your ongoing monitoring framework.
We can build policies, procedures, and templates designed to align teams and introduce technology that strengthens ownership, timeliness, and communication.
Crowe specialists can help you put your plan together