Models serve many purposes and support various decisions across an organization. A model is a mathematical representation of an entity system given certain operational, financial, compliance, and/or economic conditions that aims to quantify past, present, or future outcomes to provide decision-making information. Models typically are used to predict future results or to allow an entity to perform analysis within the mathematical model to determine the impacts of different drivers or variables on model output. Models can be simple calculations in an Excel spreadsheet with a small table of variable inputs, or they can be highly complex mathematical and statistical computations with a web of interrelated models using sophisticated software on a dedicated server.
Model governance provides oversight and control to minimize model risk, establishes policy to protect the integrity of the model output used in decision-making, prioritizes and authorizes changes to models used by the organization, and facilitates the sharing of information across the organization regarding the use and limitations of the models to improve transparency.
Before internal audit can evaluate the model governance structure and effectiveness, it needs to gain an understanding of the models that are used within the organization. This can be time-consuming. Documentation is valuable to any process, but it is difficult to find in practice. Internal audit may have to work with management to develop an initial listing that can be used to identify and assess risks and determine the audit scope. The list of models should include:
- Name for the model.
- A brief description of the model’s purpose and use.
- Key model personnel: model owner, developer, tester/validator, production operator, and users.
- Frequency of model output reporting.
- The software and platform used for the model.
- The latest version of the model being used.
- The model risk rating.
The model owner should maintain more detailed information for each model regarding inputs, assumptions, methodologies, process documentation with risks and controls identified, data flow diagrams, items excluded from the model, approximations or assumptions used in the model, model limitations, manual outside adjustments to the model, and software and hardware used by the model.
The model risk rating should be based on probability and impact and be consistent with other risk rating structures used within the organization. When determining the model risk rating, internal audit should consider several risk drivers (along with other relevant criteria based on the industry or business), including: financial statement impact of results, level of model dependency in making business decisions, regulatory requirements, complexity of calculations and the extraction/transferring/loading of inputs, degree of interdependencies among models, subjectivity of assumptions or inputs, experience level of the personnel involved, historical experience of issues, effectiveness of controls, and degree of incentive compensation that may be tied to performance or output.
Once the listing of models is compiled, risk rated, and agreed upon by key stakeholders, internal audit can perform an assessment of model governance focusing on the high-risk models as a starting point. All high-risk rated models should be within the purview of a model governance committee.
The scope of responsibilities of a model governance committee is subject to debate and tends to be the victim of scope creep given the volume of risks associated with models. “Model Governance Committee Responsibilities,” below, provides a comprehensive listing of items to be considered in determining the scope of a committee. There may be other responsibilities specific to an organization or evolving risks.
The structure and oversight of the model governance committee should be tailored to the specific needs and level of maturity of the organization:
- The committee should report to the board directly, or indirectly via another committee.
- Membership should include a variety of senior-level model stakeholders.
- Responsibilities should be clearly defined for committee members and those involved in the modeling process.
- Committee decisions should be clearly documented with supporting rationale in committee minutes.
- A communication process should be in place to notify those who are responsible for any follow-up actions, noting anyone who should be consulted or informed.
Having a model governance committee centralizes the identification of, and response to, model risks, which typically improves communication across stakeholders, builds consensus around decisions, establishes controls, and enables management action given the diversity of committee membership. The focus on model risks by regulators and external auditors has been increasing. Having a committee that receives and generates appropriate documentation makes it much easier to address those concerns.
Model Governance Committee Responsibilities
Potential responsibilities may be completed by the committee, management or a project team with committee oversight, or some combination thereof. Responsibilities will vary but could include:
- Develop, approve, and communicate model policy, standards, and procedures.
- Plan resources and prioritize tasks when there are competing priorities or dependencies.
- Review and approve technical papers from subject-matter experts regarding gray areas or where there is disagreement on model approaches.
- Prioritize and approve model changes, including tolerance and materiality levels for approvals needed for model changes.
- Review and approve risk control matrices for material models. Also, have insight into control issues that impact the model, including general IT and application controls over inputs, processes, and outputs.
- Monitor compliance issues that impact the model and approve management actions to remediate issues.
- Oversee model data quality — integrity; outliers; timeliness and availability; security; and extraction, transfer, and loading.
- Oversee model validation — static and dynamic testing, sensitivity analysis, analytics, user acceptance testing, analysis and quantification of changes, and identification of risk-based deep dives into current models on an ad hoc, periodic, or rotational basis.
- Provide an objective, robust check and challenge process on model results.
- Approve outside-the-model adjustments and rationale for use.
- Maintain a list of known model limitations and implications for use.
- Approve the timing of model releases to production.
- Coordinate the reporting calendar and use of model results.
- Identify stress and scenario testing for the models and determine management actions.
- Provide a consistent, common communication point to address questions and drive improvement.