Here is the biggest secret in internal audit analytics: The success of new analytics has less to do with what happens in the computer, and more to do with good project definition and management. The internal audit function's analytics expert cannot know everything about all of the organization's data and everything about all of the organization's business processes. However, that individual can know how to convince the people who understand a specific process and the people who understand the data produced by that process to work with internal audit for the greater good of developing the new analytic test or process.
If developing an analytic is a project, it helps for internal auditors to keep the development in sequential stages: scoping, planning, piloting, deployment, and establishment. There will be some fluidity between stages, but in general, each has its own objectives (see "The Analytics Development Process," below).
Step 1: Scoping
Target: Go or no-go.
So internal audit has an idea for a new analytic test or process. Now what?
The objective of this stage is to understand where this idea came from (background), define its general scope and objective, and decide whether to go forward with the new project based on how it will fit with internal audit's overall analytics program (see "The Analytics Journey: Finding the Right Direction").
Step 2: Planning
Target: Team and tests.
At this stage, the analytics expert leading the project will build a team for the project and agree on what aspects, features, or objectives to pursue and how to do it. The team should comprise:
- The process owner, who understands what to test for and why it is important.
- Someone from the process team, who knows how the test elements are captured in the process, their normal ranges, and the meaning of their deviations.
- The IT person who supports that system and knows — or can find out — what tables and fields capture the information identified by the process team.
Because each of these team members sees the process from a different angle, each will have different, and valuable, ideas about what to look for and how to test for them. Also, they all may appreciate the exposure of working in an interdepartmental effort to address the process owner's concern. The project leader should introduce the members to each other and engage them in brainstorming.
At the end of planning, the team should have a collection of simple tests, which may not mean much on their own but could indicate what the internal auditor is looking for when considered together. For each of the tests, the project leader will have a good idea of what the auditor will test for, how to perform the test, and where the data will come from. It is helpful to use a template to log that information for each of the simple test elements, as shown below.
Using the logs, different team members can understand the test objective and process whenever the test is performed. Also, auditors can use these logs of the test logic to recreate the test in new systems as the program continues to evolve.
Step 3: Piloting
Target: Sample data, draft data model, and draft visualization/reporting.
The extra planning time pays off during the pilot stage. Because the auditor knows what to look for (test) and where it is (data), he or she can quickly move to the math (data model), reporting, and follow-up to validate that the test is providing the expected value. The team members will give the auditor the data and help figure out whether the test is working.
A common question at this stage is whether or not the internal auditor needs direct access to the data during the pilot. The truth is that having direct access may save time later, but it is a "nice to have" at the pilot stage. After all, the auditor doesn't know if the test will work, so as much as possible he or she should work with existing data in the form that can be provided easily to yield results and refine the approach.
That said, data must be trustworthy and useful for the test, notes web analytics expert Brent Dykes. It helps for auditors to keep a log of the related data sets, along with comments on their usefulness and trustworthiness, as shown below.
The log can help the auditor decide what data needs to be refined, how it should be improved, and why this is important. Being able to articulate this information is invaluable when negotiating direct access with data owners in the future.
Step 4: Deployment
Target: Established data access, integrated data modeling, and visualizations/reporting.
Once the pilot has proved its value, the auditor should formalize the testing process and make it repeatable for periodic reporting. Ideally, this process should be automatic, such as using robotic process automation.
Step 5: Establishment
Target: Documentation, distribution, and schedule.
As the development work is completed, the real work of acting on the new information starts. To declare this project closed and move on to the next one, the auditor needs to:
- Document the testing, including why the auditor is performing the tests, where the data comes from, and what will be done to the data.
- Determine who will receive the results and establish a follow-up process and expectations.
- Set the re-run schedule for these new tests. Does the user need this information daily? Monthly? Must the user be contacted immediately when this happens?
Knowing When Development Is Working
A consistent approach to analytics can show internal audit's stakeholders that they are not wasting their time giving auditors their support. It sets realistic expectations for what will be needed from stakeholders at each stage of the development process and helps keep projects on track. Along with standard templates, it improves the chances that projects are transferable and repeatable.
The development process, or "the way we do things here," is key to consistency and scalability. It can provide a shared language between team members and stakeholders, as well as allow auditors to pursue and track projects running in parallel. It also will enable other auditors to take over part, or all, of a project if help is needed. Although the process is simple, its importance should not be underestimated.