​The Value of QAIP

Fannie Mae’s quality program demonstrates the effectiveness of its internal audit operations in meeting stakeholder expectations.​

Comments Views

​In today’s environment, an effective quality assurance and improvement program (QAIP) is critical to ensuring that internal audit meets the requirements of the audit committee, executive management, and other stakeholders. Internal and external assessments are key parts of the QAIP, and a robust QAIP incorporates many elements that are part of an organization’s day-to-day activities. The IIA Practice Guide, Quality Assurance and Improvement Program, states, “Quality in internal audit begins with the structure and organization of the audit activity. Quality should be built into, and not onto, the way the activity conducts its business — through its internal audit methodology, policies and procedures, and human resource practices.” By embedding quality into processes, rather than treating it as extra work, external quality assessments (EQAs) become a turnkey operation.

Fannie Mae has approximately 106 internal audit employees, performing 95 to 110 audits per year. A professional practice group comprising six full-time employees administers the QAIP. The group also is responsible for internal operations and reporting, and approximately 50 percent of its time is focused on the QAIP.

The QAIP is a regulatory requirement whose scope covers all operations of the internal audit department, including audits, reviews, audit issue follow-up, and special projects. Most of the program’s components have been in place since before 2007; however, heightened requirements for financial services companies and the internal audit profession require Fannie Mae Internal Audit to continually refine and expand the program.

Because of the size and complexity of the enterprise, and to demonstrate continued compliance with The IIA’s International Standards for the Professional Practice of Internal Auditing (Standards), internal audit has had an EQA performed more often than the required five-year period. In the 2014 EQA, internal audit received the highest rating — Generally Conforms — from The IIA’s Quality Services team. Before that, audit’s last EQA was in 2010, and going forward, it plans to have one performed every three years.

Fannie Mae considered several processes when looking at the overall quality of its internal audit function. And while process design may differ from organization to organization, having these processes in place is a key step toward building a high-quality department.

Independence and Objectivity

To strengthen independence, Fannie Mae’s CAE reports directly to the chair of the audit committee and administratively to the CEO. Additionally, the audit committee sets the CAE’s compensation, and the audit department has goals that are completely separate from those of the overall organization.

An independence and objectivity policy provides required actions related to various situations that may lead to potential impairments, including the transfer of the CAE or audit staff from the business units into the internal audit department, cosourcing engagements, an auditor’s personal relationship with a member of the business unit being audited or consideration of employment with a business unit, scope limitations, and consulting/advisory engagements.

All internal transfers to internal audit complete an independence questionnaire to identify areas where there may be a conflict affecting objectivity. If a potential conflict is identified, the auditor is prohibited from participating in audits of that area for 12 months. This is monitored through a potential conflicts log that is reviewed in conjunction with scheduling. A similar independence questionnaire is completed by any cosource or staff augmentation personnel that is considered before bringing the resource on board. As an additional protection, each assurance engagement includes an assessment of the objectivity of the engagement team members. This assessment is documented in the engagement workpapers.

Finally, audit personnel receive annual training on the policy, the Standards related to independence and objectivity, and The IIA’s Code of Ethics. Audit personnel also certify annually their compliance with the Code of Ethics. The results of the objectivity process form the basis for the CAE’s annual confirmation of the independence of the department to the audit committee.

Staff Development

Fannie Mae Internal Audit’s training program starts with an annual competency assessment to provide a structured guide enabling the identification, evaluation, and development of interpersonal, general, and technical capabilities of individual employees. Each category includes multiple competencies with specific measures identified for each competency. Interpersonal competencies include teamwork/collaboration, communication, driving execution (appropriate prioritization and achieving results), and inspiring/motivating. General competencies include critical thinking, business acumen, documentation, and project management. Technical competencies include mortgage business knowledge, enterprise risk management, and cybersecurity.

Each employee performs an annual self-assessment, and managers assess each person on their team. The manager and employee meet to discuss differences in their assessments and any gaps between where the employee was assessed and the expected rating. These gaps are considered while developing the employee’s annual training plan.

When all assessments have been completed, an analysis is performed by the professional practice group to identify competencies where 15 percent or more of employees have gaps in expected and actual assessed competencies. These competency gaps are an input to the annual department training schedule. Through internal development, or identification of an external training course, internal audit seeks to improve the department’s knowledge and skills related to any competency with a significant gap.

A training plan detailing the courses that will satisfy the employee’s 40-hour continuing education requirement and the breakdown of hours among competencies is developed by each employee (and reviewed by his or her manager) in conjunction with the performance management and goal-setting process. The professional practice group develops a training schedule or menu, considering results of the competency assessment, any significant changes in internal audit methodology, risks facing the enterprise, and results of the prior year’s quality assurance (QA) reviews. The plan is revised as needs change throughout the year. The competency assessment and training plan strengthens the performance management process and the QAIP. Additionally, employees appreciate the visibility into expected competencies and capability at each level provided by the competency assessment criteria.

Risk Assessment

Fannie Mae’s risk assessment process includes an annual risk assessment, a re-baseline of the annual risk assessment at midyear, and a continuous risk assessment (CRA), which is formally documented in the quarters during which the annual and re-baseline risk assessments are not performed. The annual and re-baseline risk assessments have various deliverables, including a revised audit plan; whereas the deliverable for the CRA is an updated watch list that includes key risk considerations identified and their impact on internal audit activities (e.g., covered in an existing audit or additional monitoring, or the addition of a new project to the audit plan). The presentation to the audit committee to support approval of the annual audit plan includes a list of key focus areas for the year; charts with project risk and type trends, plan hours by audit area, and plan hours by risk rating; and an audit plan resource analysis. The results of the CRA are not directly shared with the audit committee and management; however, any significant changes to the audit plan as a result of the CRA are.​​​

Methodology Documents

Methodology documents include methodology manuals, internal practice advisories and practice guides, standard audit programs, and required templates, which are incorporated by reference into the methodology manuals. Methodology manuals outline the basic requirements for internal audit’s activities and cover the risk assessment, planning, fieldwork, and reporting phases of audit engagements, as well as audit issue follow-up and quality management. Fannie Mae practice advisories expand on internal audit’s approach and related criteria for specific areas such as sampling, fraud risk assessment, and acceptance of risk, while Fannie Mae practice guides provide step-by-step guidance and may provide detailed procedures. Standard audit programs ensure certain elements of internal audit’s methodology are considered or performed, and required templates such as a risk control matrix, audit report, and management self-identified issues assessment, help ensure consistent application of the methodology.

All documents, including the internal audit charter, are revised, as needed, with updates identified through the QAIP process; new IIA, regulatory, or industry requirements; or requests for additional guidance from the audit teams. Documents are reviewed at least annually to identify required changes.

Ongoing Monitoring

Ongoing monitoring is achieved through continuous monitoring activities, including engagement supervision and feedback, internal audit management reporting, and internal QA reviews of audit and issue follow-up activity.

Engagement Supervision and Feedback In addition to the requirement that all internal audit workpapers have a second level of review, the department has formally documented the required minimum level of review for all audit activity in a matrix that audit staff can easily refer to.

Customer surveys are sent to business leads for all engagements. Surveys have recently been updated to obtain feedback through scoring (poor, fair, good, very good, excellent) rather than comments to facilitate more efficient completion by business unit management and to support reporting on results. Internal audit initially used an external survey site, but it is working to bring surveys in-house to avoid any future security concerns with an external site. The results of the surveys are not reported to the audit committee or senior management, but are used to identify opportunities to improve the audit process or to identify additional training needs.

Formal engagement evaluations are performed for all audit staff spending more than 80 hours on an engagement. This form also was changed recently to be score-based to increase efficiency in completion of the form (meets expectations, does not meet expectations, exceeds expectations).

To ensure consistent performance, internal audit requires customer surveys and engagement evaluations for each engagement. These tools are an
integral part of the audit process.

Internal Audit Management Reporting This reporting is prepared and distributed monthly via a PowerPoint presentation to the CAE and audit leadership. The reports are in a dashboard format, with department averages for comparison. Metrics include:

  • Quality: A year-to-date cumulative average score of all audit and issue follow-up internal QA reviews.
  • Efficiency: Average days between audit announcement and report issuance, audit plan completion, and staff use.
  • Innovation and Capability: Staffing activity, tenure, percentage of certifications, highest degree obtained, average training hours per auditor, and budget to actual comparison.

Additional planned reports include tracking of audit issue follow-up completion. Reports currently are prepared manually, which can be challenging, as it requires additional preparation time and a more detailed review than an automated report. To support the ability to add additional reports, internal audit has changed the production cycle for certain reports from monthly to quarterly. Moreover, internal audit management reports are leveraged for audit committee reporting, with the audit committee receiving certain audit management reports annually (e.g., innovation and capability reports) or bi-monthly (e.g., quality reports).

Internal QA Reviews These reviews have two primary components: audit QA reviews and internal audit issue follow-up QA reviews. The audit QA reviews are cosourced with an external third party to leverage their subject matter expertise and knowledge of best practices. The reviews have contributed to the improved interaction with management as they promote consistent application of the audit methodology and process. Additionally, the reviews provide the external auditors additional assurance on the effectiveness of the internal audit department as an entity-level control. The reviews are performed throughout the year, independent of the periodic self-assessment process.

The professional practices team selects audits to be reviewed by the third party at the beginning of the year, and required templates include a QA checklist. Approximately 25 percent of current-year projects are selected for review. The QA checklist:

  • Is broken down by phase (e.g., plan, fieldwork, report) and further broken down within each phase by key activity (e.g., announcement, risk identification, and walkthroughs in the planning phase).
  • Includes specific criteria for each activity. Each section receives a score, which is totaled to derive an overall score (0 to 100) for the project. The scores are broken down between quality (60 percent) and documentation (40 percent).

Internal audit performs audit issue follow-up (AIF) reviews using a checklist similar to the engagement review checklist, but the focus is on AIF activities. Reviews are done quarterly on a sample of the past quarter’s follow-up activity.

At the conclusion of engagement or AIF QA reviews, the QA checklist, including review comments, is shared with the responsible audit team members for their review. QA “lessons learned” are shared with the CAE, leadership team, and audit staff quarterly, and key observations are incorporated into training materials for future use, or methodology documents are updated to provide additional guidance as necessary.

Periodic Self-assessment Internal audit has recently put in place a self-assessment process to ensure it stays current with the Standards. Internal audit completes self-assessments in those years when an external assessment is not performed. In addition to interviews and surveys of stakeholder groups and review of internal audit activity, internal audit uses checklists developed based on QAIP guidelines promulgated by The IIA. The results of the internal QA reviews are leveraged for the workpaper quality review component.

Audit Committee and Executive Management Reporting Audit committee and executive management reporting are the most time intensive nonaudit-related element of the QAIP. To maximize efficiency in preparation of these reports, they are automated, where possible, and audit committee reporting is leveraged for executive management reports.
Audit committee reporting includes materials for the CAE’s report to the audit committee at each board meeting, and a memo providing key updates during months when the audit committee does not meet. The report includes regular categories:

  • Current internal audit results.
  • Internal audit issue and issue theme trending.
  • Analysis of report ratings year over year. Internal audit has three report ratings: The control rating (satisfactory, needs improvement, unsatisfactory), a management awareness rating (high, medium, low), and a control environment trending rating (improving, unchanged, declining).
  • An update on the status of the internal audit plan.
  • An update on the department’s methodology and QAIP results.
  • A summary of headcount and staffing activity.

Quarterly, internal audit issues a dashboard to each business head with the status of internal audit activity in his or her area. The goal is to provide the executive with a view of what is reported to the audit committee, as it applies to his or her area. The dashboard includes a five-quarter trend analysis of the following for internal audit issues, Sarbanes-Oxley deficiencies, and matters requiring attention:

  • Current inventory of issues by priority.
  • Issue status change.
  • Management self-identified issue percentage year-to-date.
  • Remediation time frames.
  • Internal audit ratings year-to-date.
  • Detailed issue status reports also are provided to the business monthly. These reports include issue description, priority, and status.

Strategic Plan

Internal audit’s strategic plan is updated semi-annually. The first part of the plan outlines at a high level the department’s vision, mission, and core values as well as four to five strategic areas of focus. The second part includes details related to the department’s strategic goals and action plans, including specific action items that will contribute toward achieving the goal and a target date for each action item. The goals and action plans span one to two years (a three-year horizon is recommended; however, Fannie Mae Internal Audit found two years to be more practical). Twice a year, the leadership team reviews the plan and status of the action items, adds new goals and action items as necessary, and changes time lines, if necessary. The plan and status against each goal is shared with internal audit management and the internal audit department at least annually.

QAIP Evolution

While maintaining and continually refining the QAIP is challenging from a resource perspective, internal audit has found that its interaction with management and the audit committee has improved as a result of the refinements. For example, during the 2010 EQA, management noted opportunities for improvement in interactions with internal audit. No such feedback was received in the 2014 assessment, and IIA Quality Services noted that interviews with management indicated internal audit’s role is highly valued.

Fannie Mae’s QAIP continues to evolve with changes in the industry, leadership, and the regulatory landscape. In maintaining or making improvements to the program, the function must evaluate trade-offs between activities to ensure it does not spread itself too thin and lose the very benefits it is trying to reap. Internal audit also must consider which activities can serve multiple purposes, to maximize its time and resources.

Internal Auditor is pleased to provide you an opportunity to share your thoughts about the articles posted on this site. Some comments may be reprinted elsewhere, online or offline. We encourage lively, open discussion and only ask that you refrain from personal comments and remarks that are off topic. Internal Auditor reserves the right to remove comments.



Comment on this article

comments powered by Disqus
  • Galvanize-September-2020-Premium-1
  • FSE-September-2020-Premium-2
  • Auditboard-September-2020-Premium-3