The analytics gold rush is on. Organizations around the world are spending considerable money to build or buy analytic models and analytics capability to take advantage of big data, machine learning, and artificial intelligence (AI) technologies. These models have made their way into every aspect of business and are being relied on as decision support — and, in the case of machine learning and AI, actually making the decisions — for issues such as:
- Determining the probability of default for potential borrowers (corporate and individual).
- Evaluating new employees' probability of success and tenure with the organization (from professional athletes to salespeople).
- Forecasting success and return on investment for new marketing initiatives.
- Making product mix and store location decisions.
- And coming soon, making life-and-death decisions in self-driving vehicles.
Today's organizations have billions of dollars riding on the accuracy and performance integrity of analytic models. With model performance becoming a strategic enabler, organizations need to manage the risks associated with analytics.
To effectively manage these risks and move beyond simple financial model or spreadsheet auditing, organizations need a system of controls around analytic model development, application, and maintenance. These analytics controls provide checks and balances around model selection, validation, implementation, and maintenance. Periodic internal audits can help determine whether analytics controls are designed appropriately and operating effectively.
Models and Controls
An analytic model is a mathematical equation that takes in data and produces a calculation such as a score, ranking, classification, or prediction. It is a very specific set of instructions for analyzing data to deliver a particular kind of result — behavior, decision, action, or cause — to support a business process.
The objective of analytics controls is to ensure that:
- Analytics personnel have the appropriate skills and training.
- Input data is appropriate, complete, authorized, and correct.
- Model selection procedures are documented and justified.
- Model validation and testing have been conducted in accordance with scientific principles.
- Outputs are accurate, complete, and being used by the business as intended.
- The model is refreshed and reevaluated periodically.
- The organization maintains a record to track the processing of data from input, to processing, to the eventual output.
There are several types of analytics controls. Skills controls provide assurance that data analytics personnel are competent and sufficiently trained in relevant analytics methods. Business-use controls provide assurance that the model addresses the intended business objective. Data controls are used mainly to check the integrity of data entered into an analytic model. Model selection controls ensure model selection is appropriate and reasonable to provide decision support. Model validation controls address what is done to ensure the model output is reasonable and accurately reflects the underlying nature of the input data. Output controls provide assurance that the model output is presented and used in an appropriate and justified manner to ensure it remains consistent and correct. Maintenance controls address the need to reevaluate and refresh analytic models periodically to ensure they are still relevant in the current environment.
Analytics vs. IT General Controls
Internal auditors need to understand the relationship and difference between analytics controls and IT general controls. Otherwise, an analytics controls review may not be scoped appropriately, negatively impacting the audit's quality and coverage.
IT general controls apply to all systems components, processes, and data present in an organization or systems environment. The objectives of these controls are to ensure the appropriate development and implementation of applications, as well as the integrity of program and data files and of computer operations.
Analytics controls differ from IT general controls because they relate to the methodology and data pertaining to each analytic model. They are specific to each individual application.
Internal auditors must note the degree to which management can rely on analytics controls for risk management. This reliance depends in part on the design and operating effectiveness of the IT general controls. If these controls are not implemented or operating effectively, the effectiveness of analytics controls is greatly diminished. For example, if the IT general controls that monitor program changes are not effective, then unauthorized, unapproved, and untested changes to an analytic model can be introduced to the production environment, thereby compromising the overall integrity of the model.
Analytic Model Categories
There are three main categories of analytic models: descriptive, predictive, and prescriptive. Each category can provide an organization value and strategic insight.
Descriptive These models allow organizations to condense big data into smaller, more digestible pieces of information. Typically, organizations that use analytics meaningfully have mountains of raw data at their disposal. Descriptive analytics enables an organization to summarize that data and determine what really happened. Most analytics in use are descriptive: sales breakdowns, social media likes and followers, ratings, and reviews.
Predictive The next level up in data analysis, predictive analytics uses a variety of statistical, modeling, data mining, and machine learning techniques to study recent and historical data, enabling analysts to identify patterns and correlations in the data. Based on these identified patterns and correlations, analysts can create a model of the future results given selected inputs. For example, based on certain borrower characteristics, a bank may use a predictive model to forecast its amount of loan defaults.
Prescriptive The highest level of analytics, prescriptive analytics recommends one or more courses of action and shows the likely outcome of each decision. Unlike a predictive model, a prescriptive model shows multiple future scenarios based on a decision the organization makes today. Prescriptive analytics requires a predictive model with two additional components: actionable data and a feedback system that tracks the outcome produced by the action taken. An example of prescriptive analytics would be a casino floor product mix optimization model that predicts revenue gains given various game configurations.
Auditors should use risk assessment techniques to identify critical vulnerabilities pertaining to the organization's reporting and operational and compliance requirements when developing the risk assessment review plan. These techniques include the review's nature, timing, and extent; critical business functions supported by analytic models; and the extent of time and resources to be expended on the review.
To add value to organizationwide analytics control risk assessment activities, internal auditors should define the universe of analytic models and supporting technology (modeling software, data services, etc.). They also should summarize the risk and controls using the risk and control matrices documented during the risk assessment process.
Next, internal auditors should define the risk factors associated with each analytic model by answering questions such as:
- Does the model support a regulatory requirement?
- How complex is the model type?
- How effective is the design of analytics controls?
- Is the model prepackaged (off the shelf) and customized or developed in house?
- Does the model support
- more than one critical business process?
- How is the data processed by the model classified (e.g., financial, private, or confidential)?
- How frequently are changes made to the model?
- How complex are those changes?
- What is the model's financial impact?
- How effective are the IT general controls residing within the application (e.g., change management, logical security, and operational controls)
Once they have answered these questions, internal auditors should weigh all risk factors to determine which risks need to be weighed more heavily than others (see "Assessing Model Risk" below). From there, they should determine the right scale for ranking each application control risk by considering qualitative and quantitative scales, such as:
- Low, medium, or high control risk.
- Numeric scales based on qualitative information (e.g., 1=low-impact risk, 5=high-impact risk; 1=strong control, 5=inadequate control).
- Numeric scales based on quantitative information (e.g., 1=less than $50,000 and 5=more than $1 million).
With this information in hand, internal auditors should conduct the risk assessment, rank all risk areas, and evaluate the risk assessment results. Finally, they should create a risk review plan that is based on the risk assessment and ranked risk areas.
Internal auditors should keep in mind that the review's scope, depth, approach, and frequency depend on the results of the risk assessment and the availability of internal audit resources. If the analytics team uses a recognized methodology for model development such as the Cross-Industry Standard Process for Data Mining (CRISP-DM) or some other widely accepted system, then internal auditors should consider auditing to that standard. In addition, some organizations have established a model risk management function. Internal audit can audit that area using similar methodology to that applied to other compliance functions.
For organizations whose analytics teams do not use a prescribed model development methodology, there are two approaches auditors can use to audit analytics controls: the Integrated Model Review Methodology (IMRM) and the Stand Alone Model Methodology (SAMM). These methods apply CRISP-DM principles in an internal audit context.
IMRM This approach can be used to evaluate model risk by examining all the business processes that feed or are dependent on the model being reviewed. When using the IMRM, internal auditors should include within the review's scope all the organization's systems that are involved in the model under review and whether the implementation of the model is consistent with the organization's analytics strategy. In other words, the auditor needs to include within the review's scope the separate processes that make up the different components of the model cycle. The auditor then can identify the inbound and outbound interfaces within the model and complete the scoping activity. For example, when auditors review a marketing campaign response model, they would scope in survey methodology and data collection processes, customer segmentation processes (inputs), and marketing decisions made based on model output.
Using the IMRM approach automatically devotes more audit resources to those analytic models that affect a larger portion of the organization's operations. To use the IMRM effectively, auditors need to understand the business processes surrounding the use of the model being reviewed and how data flows into and out of the model.
SAMM The alternative approach, the SAMM, is used when the auditor wants to review the controls within a single model. The SAMM is useful for new models or when audit resources are limited. Essentially, the auditor is verifying that the model, itself, has appropriate controls and performs the intended function. It does not provide assurance as to whether the organization is using the model output effectively or whether the model inputs are valid. Although SAMM is effectively a subset of the IMRM, internal auditors should clearly specify which methodology they are applying so that management and the audit committee know the extent to which they can rely on the results.
It's Still Internal Auditing
Although many auditors may be unfamiliar with analytic models, machine learning, and AI, the fundamentals of internal auditing remain the same. As with all new technologies and processes that organizations have embraced, internal auditors have a responsibility to learn how analytic models can be useful in their work and adapt their methods to serve their stakeholders.