Within the seven analytic control domains — skills, business use, data, model selection, model validation, output, and maintenance controls – there are specific controls that internal auditors can test. Auditors can use the Integrated Model Review Methodology to test all the controls listed in the tables below. With the Stand Alone Model Methodology, only data controls, model selection controls, and model validation controls are tested.
Skills Controls (AC 1) — Provide assurance that data analytics personnel are sufficiently trained in relevant analytics methods.
|AC 1.1 Analytic Competency||Analytics staff have appropriate baseline educational qualifications and/or sufficient experience in data science and data analytics for their role and level. |
- Review employee files and assess level of analytic competency.
- Compare education/ experience at each level with established corporate competency levels, if available.
|AC 1.2 Skills Maintenance||The analytics department has sufficient training budget and ensures employees' skills are kept up to date. |
- Calculate training spend per employee per year and compare with established norms/benchmarks.
|AC 1.3 Skills Deployment||Employees with the appropriate skills and knowledge are assigned to projects that suit their competencies.|
- Cross reference results of AC 1.1 with type of analytic model under consideration and assess appropriateness of fit (e.g., a junior analyst should not be the sole resource assigned to a high value/high visibility model).
Business Use Controls (AC 2) — Provide assurance that the model addresses the intended business objective.
|AC 2.1 Business Objectives||Analytics personnel understand business requirements and have documented success criteria. |
- Review documentation detailing the analytic model success criteria and ensure that business owners and analytics personnel have signed off on the criteria.
- If that documentation is unavailable, interview business owners and analytics personnel separately and assess the level of agreement between business owners' expectations and analytics personnel's understanding of the business problem and success criteria.
|AC 2.2 Economic Justification||Use of analytic models is an appropriate and cost-effective method to address the business problem. |
- Obtain cost-benefit analysis from business owners.
- If document is unavailable, conduct cost-benefit analysis.
|AC 2.3 Model Applicability||The type of model selected is appropriate for the business situation and objective. |
- Review the list of analytic model and assess choice of model in relation to the business problem.
- If model choice seems inappropriate, obtain rationale from analytics personnel.
Data Controls (AC 3) — Provide assurance on the integrity of data entered into an analytic model.
|AC 3.1 Data Acquisition||Data sources are reliable and appropriate for the model under review. |
- Review data source and assess level of quality control and reliability (i.e., open source data vs. purchased data).
- Verify accuracy of data by corroborating a sample with other sources, if available (e.g., economic indicator data from a private data provider can be validated against Bank of Canada website data).
- Interview analytics personnel to determine reasons that the data set was selected for this model.
|AC 3.2 Data Manipulation and Completeness|
Model input data has been cleaned and post-verified.
Input data transformations are documented and post-verified.
Merged/aggregated data has been post-verified.
- Obtain documentation relating to all procedures used to clean raw data. Obtain evidence of post verification (i.e., key summary statistics are the same pre and post cleaning).
- Obtain documentation relating to any variable transformation procedures. Assess rationale and reasonability of data transformations.
- Obtain documentation related to data joins/links and verify procedures used to ensure data integrity.
Model Selection Controls (AC 4) — Provide a means to ensure model selection is appropriate and reasonable to provide decision support.
|AC 4.1 Variable Selection||Rationale for variable inclusion in the model is justified and documented. |
- Obtain documentation regarding variable selection process.
- If documentation is not available, interview analytics personnel regarding variable selection process and rationale.
- Confirm with business owners that variables selected for inclusion in the model are appropriate.
|AC 4.2 Model Choice||The choice of model is appropriate to the business problem and justified. |
- Obtain documentation regarding model selection process.
- If documentation is not available, interview analytics personnel regarding model selection process and rationale.
|AC 4.3 Model Assumptions||Assumptions around model selection (i.e., market conditions, proxy variables) are reasonable and supported by evidence and/or business owners. |
- Obtain documentation regarding model assumptions.
- If documentation is not available, interview analytics personnel regarding model assumptions and rationale.
- Verify that business owners understand and agree with analytics personnel's assumptions.
Model Validation Controls (AC 5) — Address what is done to ensure that the model output is reasonable and accurately reflects the underlying nature of the input data.
|AC 5.1 Model Assessment||Model performance has been assessed using statistically valid/supportable method. |
- Obtain test strategy document.
- If document is not available, interview analytics personnel regarding model testing methods.
- Verify that models have been evaluated.
|AC 5.2 Model Ranking||Models have been ranked according to performance measures. |
- Have analytics personnel run models in the presence of internal audit and observe performance statistics. Analytics personnel should explain the nature of model performance measures to internal audit.
Output Controls (AC 6) — Provide assurance that the model output is presented and used in an appropriate and justified manner to ensure it remains consistent and correct. Includes end-user controls.
|AC 6.1 Model Results||Model results have been evaluated against business success criteria. |
- Compare model results with stated goals (see AC 2.1).
|AC 6.2 Parsimony||Model results are understandable and justifiable to business owners. |
- Interview business owners to determine if model results meet expectations and intended use.
- Have business owners explain high-level model methodology and rationale for selecting the chosen model among competing models.
Maintenance Controls (AC 7) — Address the need to reevaluate and refresh analytic models periodically to ensure they are still relevant in the current environment.
|AC 7.1 Model Deployment Controls||A model deployment strategy exists and is being followed. |
- Obtain mode deployment strategy document.
- Interview business owners to determine whether deployment strategy is being followed.
|AC 7.2 Model Refresh Controls||A process exists to periodically review model accuracy and refresh the selected model when required. |
- Obtain model refresh strategy document.
- If document is not available, interview analytics personnel to determine refresh strategy.