Point of View

Predictive analytics model validation: Building a model validation group to better mitigate risk

  • Facebook
  • Twitter
  • Linkedin
  • Email

Banks are increasingly turning to predictive analytics as a key component of their risk management strategies, especially in the wake of Basel II. However, predictive models bring their own set of risks that call for specialized risk management techniques, usually delivered by a corporate model validation group (MVG). Although many institutions have set up these groups, they often lack the structure and resources to deliver on anything more than the bare regulatory requirements. By taking a more comprehensive approach to developing the MVG, banks can further mitigate the risk of steep regulatory penalties and even capture significant business value beyond the compliance function. The following overview outlines practices that can foster the creation of an operationally independent and effective Center of Excellence (CoE) for model validation.

Creating the right model validation group

Establishing an independent and highly qualified MVG with the authority to delve into every aspect of developing and using a model across the enterprise is key to effectively managing risk.

With advances in computing technology, predictive analytics has become a universally accepted tool for most types of financial decision making. Banks use predictive models for many purposes, and in recent years, the Basel II requirements have given banks added impetus. All too often, however, the complexity involved in modeling a wide range of real-world risks across multiple business lines has promoted organizational silos and inadequate visibility into underlying methodologies. With even more regulation now on the horizon, model governance has emerged as a critical challenge, but it also offers opportunity. Experience has shown that a well-designed model validation function can not only facilitate compliance but also contribute broader value to the business.

The idea behind recent regulatory initiatives is to promote a more risk-sensitive capital framework by providing banks with incentives for implementing good risk management practices. Basel II, for example, gives banks greater freedom to monitor and quantify credit risk, operational risk, and market risk in specific ways. This less prescriptive approach has triggered an avalanche of new mathematical and statistical procedures. However, this growing flexibility has its own challenge: “model risk” stemming from assumptions, biases, and errors in the models themselves.

Take a copy for yourself

Download PDF

Regulators are aware of this, and they’ve set out two possible mitigation strategies. Banks can allocate capital reserves to absorb losses caused by the flawed use of models. Given the scarcity and cost of capital required, this is not a popular choice 1 . The second option is to establish an independent mechanism for mitigating model risk through an independent model validation function, sometimes known as an MVG.

The MVG should provide assurance to senior management that the bank’s predictive models are performing within established performance thresholds. This means not only validating the performance of each model but also assessing the relevance of model specifications and the methods used to report and interpret model outputs. However, not all MVGs are created equal. The best in class are those governed by a robust and effective policy framework with qualified resources and comprehensive execution standards to ensure the ongoing applicability of all models against their intended purposes.

The roadmap for success

Success in managing model risk depends on how the MVG is empowered and the extent to which the model is embedded in the corporate culture. Many banks are tempted to settle for a minimalist approach focused on merely meeting regulatory requirements; as a result, the banks fail to achieve the benefits of robust enterprise-wide risk management, and they do not recognize the growing business value of predictive models. To that end, outlined below are the six best practices for envisioning the ideal model validation group.

1) Independence
The ideal MVG is empowered by senior management as an independent team with unfettered access to every predictive model used in the business, regardless of who owns the model or what purpose it serves. Of course, regulatory authorities require the separation of model development and validation. However, the most effective teams have even greater scope, with clear lines of authority that avoid any real or perceived conflict of interest. Ideally, this team reports directly to the chief risk officer (or equivalent) with formal policy guidance from the Risk Management Committee.

2) Competencies
The MVG must possess knowledge and competencies that provide a balance of technical expertise, organizational awareness, and business judgment. Team members must demonstrate authoritative knowledge of mathematical and statistical methods. The team must also have a broad understanding of the business context ranging from regulatory requirements to the enterprises’ strategic objectives. Strong leadership and collaboration pull these individual competencies together to deliver outstanding team results in diverse business settings.

3) Framework
The MVG operates within a structured analytical and policy framework so that every aspect of each model is covered, from design to implementation and ongoing use. The framework establishes an evaluation cycle (typically annual) and directs team efforts toward the central question of whether models are performing within their established performance thresholds. The framework also ensures that each aspect of every model is systematically addressed, including factors such as conceptual soundness, consistency with regulatory guidance, alignment with industry practices, and relevance for the intended business purpose. The ideal framework also incorporates ongoing performance monitoring of each model. This includes assessment of data sources used as inputs as well as outcome analysis to test model outputs against real-world outcomes. Results falling outside established performance thresholds are then flagged for further investigation leading to possible model recalibration or redevelopment.

4) Comprehensiveness
The ideal model validation approach encompasses the full range of KPIs relevant to the business. In other words, the model should not only meet its established technical requirements but also contribute to continuous improvement of the business as a whole. Certainly, the team must be free to determine whether a model is adequate for its intended purpose. However, the team is also alert to the possibility of improvements or expansions to address additional business objectives.

5) Model inventories
The MVG should go beyond evaluating individual models to create an enterprise-wide inventory to facilitate model validation workflow. The inventory tracks when and by whom each model was validated and establishes scheduling and work assignments for the next validation. The outcomes of previous validation processes are described, along with the position of each model in its expected lifecycle.

6) Reporting
The reporting strategy should link critical findings with the specific actions required. Senior executives need to know how much confidence they can place in their overall modeling strategy and in each model. Model owners and other actors need to know how they can improve the performance of their models. Although this might suggest a customized approach for each report, comprehensiveness is still critical to meet the documentation requirements of the firm’s model validation policy. Thus, the ideal situation is to use a reporting template that calls to mind which elements are mandatory and which can be shaped toward specific needs. To carry this out effectively, MVG team members must use tact and judgment to avoid or resolve potential conflicts with model owners or developers.

Toward the implementation of MVG

Every organization is unique and has its own requirements when it comes to determining the level and complexity of the model validation requirements. Developing a best-in-class MVG is not a trivial task given the scope of change that must occur without disrupting ongoing business processes. What’s more, robust model risk management is generally only part of a broader shift toward a more comprehensive risk management culture. Since the policy and operational implications can be profound, external assistance may be required to create sustainable improvements, especially if the right resources and knowledge base are not readily available inside the organization. The impact can be significant, and durable.

Best practices for model validation teams

  • An independent governance structure, adequate authority, and clear policies
  • A collaborative team with a range of competencies and experience
  • A comprehensive framework of validation processes
  • A full toolbox of validation techniques
  • A complete inventory of all models used in the business
  • A strong reporting culture

This has been authored by Animesh Mandal, Senior Manger, Retail Risk and Collections Practice at Genpact.

Visit our analytics page

Learn More
  1. A review of 2011 annual reports from 15 global banks revealed only two had reported a separate capital allowance for model risk.