1510 Selection of Performance Audit Topics

Aug-2021

The Auditor General may bring to the attention of Parliament or a territorial legislative assembly those matters that the Auditor General believes to be of interest and significance. This can be done through performance audit reports or other products, including, but not limited to, reviews, studies, or follow‑ups on past recommendations or on the government’s performance results in areas we previously audited. The starting point in selecting products that can add value is deciding what to audit from among the multitude of government activities and programs. The OAG can examine activities or programs within one entity, across several entities, or across all of government and may look at the role of central agencies in regard to the activities or programs to be examined. To assess the state of a program in Canada, the OAG can also undertake audits in cooperation or collaboration with other levels of government, such as provincial auditor general offices. Audits can also examine the implementation of previous audits’ findings and recommendations.

Strategic audit planning: Selection of the audit topics and sectors

Audit selection begins by identifying significant risks—both internal and external—facing government departments and agencies and the government as a whole. The audit team may select audit topics that are important to the achievement of organizational or governmental goals for which the risk is assessed to be high. Areas that are important, but where the risk is assessed to be low, might also be selected because even if the risk is low, any deviation could be important to the entity or to its stakeholders.

Knowledge of business—The audit team must establish and maintain a good understanding of the area or entity objectives, expected results, and accountabilities. The team does some of this on an ongoing basis through its knowledge of business activities, which include meeting periodically with entity officials, reviewing key entity documents, and monitoring parliamentary and legislative committee activity and media reports (see OAG Audit 1505 Acquiring and maintaining knowledge of business for performance audits).

Strategic audit planning—A more intensive exercise to obtain critical, up‑to‑date entity information, known as the strategic audit planning process, takes place for each entity or functional area, or for the whole of government in the territories. The strategic audit planning process promotes a consistent yet flexible approach to multi‑year audit planning that is documented and risk based. It demonstrates that the OAG is fulfilling its responsibilities and exercising independence and objectivity in selecting matters to audit. It ensures that OAG resources are focused on the areas of highest risk by considering each area’s significance and relevance to Parliament or the respective legislature. It also ensures that the audits add value to Parliament or the legislature and Canadians and that the OAG is exercising due diligence in applying the discretion provided in the Auditor General Act for selecting matters for audit. The strategic audit plans may also help the team identify other types of products that could help support accountability, transparency, and improvement of government operations.

This guidance provides suggested steps to follow to identify potential areas to audit. Teams should use their discretion and professional judgment to determine the steps they need to follow to create a meaningful strategic audit plan. The following steps can be considered as part of a tool kit to plan, conduct, and report strategic audit plans (Exhibit 1).

Overview of the strategic audit planning process

Exhibit 1—Key steps of the strategic planning process

Strategic audit planning is the risk-based assessment that the OAG uses to select topics for future audit work. A risk‑based assessment is a process to identify, assess, and prioritize risks, in order to be able to identify areas to audit. The team may assess the importance of, and risks associated with, the entity's or area’s activities by considering factors such as:

• Economic, social, and environmental impact—Programs, activities, or processes affecting a large segment of the population or vulnerable populations, or which impact environmental sustainability, may be considered to be more important to the entity.

• Relevance to stakeholders—The interest shown by the legislature or other governing bodies, by management of the entity or by the public may indicate the importance of the activity to stakeholders.

• Diversity, consistency and clarity of the entity's objectives and goals—Diverse or inconsistent objectives increase the risk that the entity's activities or programs are not operating with due regard to one or more of the principles of economy, efficiency, effectiveness and environment and sustainable development. Entity objectives and goals that are not clearly defined may increase the risk that they will not be achieved because they are not understood by employees.

• Complexity of operations—An increase in the complexity of an entity's operations, through increased variety and type of programs, functions and activities may increase the risk that the entity does not achieve its objectives and goals or that they are not achieved efficiently, economically or with due regard to the environment or sustainability.

• Complexity and quality of management information and control systems—Complex systems may be more difficult to develop, enhance and maintain. When adequate management information systems are not maintained, proper control may not be exercised.

• Impact of environmental or organizational change—Changes in an entity's environment or organization can impact the continuity of operations and the understanding of priorities and processes by employees. This may increase the risk that the entity's goals and objectives will not be achieved. Environmental changes include new government priorities, significant budget amendments and changes to enabling legislation. Organizational changes include changes in leadership, reorganization, new initiatives and staff turnover.

• Financial magnitude and nature of transactions—Large dollar amounts, high transaction volumes and transaction complexity and flow may create increased risks to the entity.

• Management response to previously identified deficiencies—Areas where management has not made adequate improvements to address important issues raised in prior performance audits or other studies may be more important and higher risk.

• Organizational structure—Centralization and decentralization of key activities such as budgeting, payroll, disbursements, human resources management and facilities management each create their own operational risks. Similarly, program delivery through agents carries different risks than those associated with direct program delivery.

• Program delivery method—Programs in the public sector may be delivered by policy instruments such as expenditure, regulation and revenue-raising; may provide goods or services directly or may redistribute income; and may be delivered directly or by using agents. The amount of associated risk may vary depending on the delivery method.

The objective of the strategic audit planning process is to help identify areas of significance or of a nature that should be brought to the attention of the House of Commons. The process allows the teams to efficiently and effectively identify potential audit topics that

• add value, meaning that they focus on important gaps, outcomes, or areas for improvement in federal or territorial programs

• address federal or territorial programs’ effects on individual Canadians and other stakeholders

• can be audited, meaning that

  • the subject matter is capable of consistent measurement or evaluation against the applicable criteria

  • suitable criteria exist

  • the evidence needed to support the conclusion can be obtained

  • the work conducted will lead to recommendations that are reasonable, able to be implemented, and can be directed to an entity that has the responsibility and authority to act on them

Suggested steps for strategic audit planning

Step 1. Structure the strategic audit plan, and determine the level of effort required

The strategic audit planning process has 3 phases—planning, conducting the work, and reporting. The output from this process is a list of potential topics to audit or review. A strategic audit plan can be developed for an entity within a given portfolio or can be based on a sectoral topic. It is to be reviewed and updated periodically as needed, ideally every 3 years. To remain current and relevant, teams will be asked to review annually the list of proposed audit topics that came out of the strategic audit planning and consider emerging risks, new announcements made by government, changes to government programs, and so on. This ensures that the audits that are selected as part of the annual selection process are risk based and relevant.

If teams are updating a strategic audit plan that was previously developed, they should use their professional judgment regarding the level of effort required. All of the following steps might not be necessary if the team is requesting only a few meetings to discuss recent changes or updates to documentation. Depending on the anticipated level of effort, the strategic audit planning team and the principal, who maintains a budget for strategic audit planning work, will determine the budget (hours and resources) for completing the strategic audit plan.

There are three types of strategic audit plans:

• Entity-specific—A strategic audit plan is developed for a single entity, and different areas (programs, processes, and functions) within the entity are identified for audit.

• Sectoral—A strategic audit plan is developed around a subject and includes more than one entity, as different entities share responsibility for the subject, with each entity fulfilling a unique role. Examples of sectoral subjects include information and technology, innovation, environment, and human resources.

• Whole of government (in the case of territories)—Teams responsible for a territorial portfolio will most likely perform a strategic audit plan on activities and programs within the whole of government.

The strategic audit planning team should also determine the tool to use to document the information they gather as they develop the strategic audit plan: audit working paper software or PROxI. While both options are available to teams, the use of PROxI is strongly encouraged to allow for ease of access, sharing of information, and communication of the results of the strategic audit plan.

Note that team members are not required to complete and sign an independence form, as the strategic audit planning process is not an assurance engagement performed in accordance with professional standards.

Step 2. Send notification letter

Strategic audit planning teams are required to send a notification letter to each entity included in strategic audit planning work. The letter advises the entity that strategic audit planning work will commence to assist with the OAG’s selection of future audit topics. The letter also advises on solicitor-client privilege matters and on the OAG’s right to access the information needed to conduct its strategic audit planning work.

Step 3. Gather information, consult, and assess past work

Planning strategic audit planning work involves several steps. Teams are also encouraged to share information with other strategic audit planning teams to improve their understanding and reduce duplication of work.

Identify previous strategic audit plans on the subject matter. Review the work done in any previous strategic audit plans. The information gathered will assist the team with updating an existing strategic audit plan or developing a new strategic audit plan and will ensure that the team has a baseline understanding before meeting with entity officials. The same type of work is required whether a strategic audit plan exists or not. However, if one exists, the same level of effort may not be required, as the team has a base to start from. A list of the strategic audit plans planned and completed is posted on the INTRAnet.

Discuss past audit work with performance audit teams. Discussions with performance audit teams may help the strategic audit planning team better understand what work has been conducted and identify if significant risks from previous audits should be followed up. Teams can further discuss with auditors who have experience on the portfolio or have relevant expertise. Relevant documentation should also be identified and shared between the performance audit team and the strategic audit planning team, especially if the topic is a continuation of a former audit.

Discuss current audit work with financial audit teams assigned to the entity. While collaboration with financial audit teams occurs throughout the year, specific discussions and collaboration with financial audit teams may assist the strategic audit planning team with identifying key risks.

Discuss audit work with special examination teams. Teams should also consider speaking with audit teams responsible for special examinations. Discussions with special examination teams are particularly relevant if a Crown corporation has shared responsibility with an entity on a particular subject.

Discuss risks with internal specialists. Teams should determine which internal specialists could assist in identifying specific entity or sectoral risks that should be considered. Similarly, strategic audit planning teams should share with the relevant internal specialists the risks identified through the strategic audit planning process that are related to the specialist’s area and discuss the assessment of these risks.

Teams should complete the Environment and Sustainable Development Risk Profile for Strategic Audit Planning form and consult with the Internal Specialist—Environment and Sustainable Development. Teams should also consult the audit guidance on the Sustainable Development Goals (SDGs) in order to explore links between SDGs and the entities’ mandate, core activities, and major programs.

Teams should consult the Internal Specialist—Environment and Sustainable Development to discuss any considerations of gender-based analysis plus (GBA+) in the strategic audit planning work.

As needed, teams should involve the data analytics team early to ensure the strategic audit planning team is aware of the questions to ask the entity to assist with understanding the quality of the entity’s data and understanding if and how the data could be used in future audits.

Teams should also involve Legal Services early in the strategic audit planning process. Legal Services can help the team navigate certain constitutional, jurisdictional, or other legal mandate issues before the planning process advances too far.

Consult with other portfolio teams. Consultations and collaboration with other teams can help determine if there are any common or shared topics or risks that may affect the development of the strategic audit plan. Strategic audit planning teams should collaborate, coordinate, and share information to efficiently and effectively identify potential audit topics. Strategic audit planning teams are also encouraged to collaborate when requesting interviews with entity officials.

External collaborations. Teams should consider external collaborations while planning the strategic audit planning work. Examples of such collaborations include consultations with the provincial legislative audit offices or other supreme audit institutions.

Other services. Teams are encouraged to consult other services available at the OAG to help them in their understanding of the government programs and activities. The OAG Knowledge Centre Library can set up media alerts, analyze results, and help conduct various research. Other services, such as the IT Audit team, the Controls Assurance team, Contracting and Procurement, Parliamentary Liaison, and many more, can help teams gain knowledge, skills, and experience on various topics.

Step 4. Conduct document review

The strategic audit planning team should obtain sufficient knowledge of the entity or sector so it can identify the mandate, key activities, and key outcomes and assess any risks (Exhibit 2).

Exhibit 2—Areas for the strategic audit planning team to consider to increase its understanding of the entity or sector

Areas to develop an understanding of	Areas to consider	Examples of documentation to review or information to obtain
Operating environment	political, legal, economic, social, and technological environments (existing and planned) entity priorities, strategic outcomes, programs, expected results other relevant or related entities that may be involved or have competing interests and the nature of these entities and accountabilities (special operating agencies, agencies, departments, and so on) stakeholders (the public, private industry, and interest or advocacy groups) Canadians’ opinions and reactions to the entity’s programming	Departmental Plan (3‑year forward looking) Departmental Results Report (actual performance for most recently completed fiscal year) entity’s corporate risk profile internal audit and evaluation reports organizational chart media alerts created by the Knowledge Centre Library entity mandate letters media reports publications by the entity’s stakeholders Departmental Sustainable Development Strategy gender-based analysis plus social media monitoring walk‑throughs of key systems and processes review of related databases and other repositories of key information
Organizational structure	how the entity is organized, including reporting relationships key relationships among the different divisions within the entity and accountabilities importance and relevance of federal–provincial relations
Parliamentary interest and media attention	key issues of interest to Parliament and those that may be in the media key issues of interest to Canadians	activities conducted and reports produced by relevant parliamentary committees in the Senate and House of Commons Hansard reports that list information discussed by members of Parliament or legislative assemblies social media monitoring

The strategic audit planning team may also find it helpful to consult key experts and stakeholders who have knowledge of the entity or sector to use their expertise to confirm identified risks.

Step 5. Meet with the entity’s internal audit officials

The strategic audit planning team arranges an introductory meeting with the entity’s chief audit executive or equivalent to discuss the strategic audit planning process. The chief audit executive is responsible for the internal audit function, which acts as a liaison with entity officials. The strategic audit planning team can use this opportunity to

• explain the importance of the strategic audit planning work to plan future audits

• discuss the list of entity officials to interview

• request relevant documentation not publicly available

• decide on a working protocol

At meetings with officials of the internal audit function, the strategic audit planning team should also discuss, among other things,

• the activities performed, or to be performed, by the internal audit function

• the knowledge of business work performed by the internal audit function

• the risk and control assessments done

• evidence of non-compliance with relevant ethical requirements or fraud, if any

• other activities or information that could be shared with the strategic audit planning team

Step 6. Interview other entity officials, subject matter experts, and stakeholders

The strategic audit planning team interviews selected entity officials for strategic insight and to provide an opportunity to engage them in the ongoing strategic audit planning work. Other key individuals may be interviewed to gain a comprehensive understanding of the entity and its operations. Consideration should be given to meeting with the following individuals:

• the Deputy Minister and assistant deputy ministers

• Departmental Audit Committee members (in camera meeting)

• selected members of senior management

• operational and program managers

• key regional officials

• experts and stakeholders familiar with the entity and its operations (such as non-governmental organizations, associations, businesses, and professional bodies)

• Treasury Board of Canada officials, especially when multiple entities are involved

• research staff from relevant House of Commons and Senate committees (the strategic audit planning team should consult with OAG Parliamentary Liaison staff to help with scheduling meetings with research staff)

Step 7. Consider risks and controls

The strategic audit planning team should assess the risks of the entity not achieving its objectives. The higher the risk, the more likely there will be a significant impact on the entity’s operations. To have a more complete understanding of risks, the strategic audit planning team can consider the effect controls have on the risks identified and identify the residual risk that remains once controls have been considered (Exhibit 3). If controls are generally in place and working, then the risk may be lowered to an acceptable level and there may not be much value in auditing the subject matter.

Exhibit 3—Consideration of controls to assess the residual risk

Inherent risk – controls = residual risk
Inherent risk—the susceptibility to a significant deviation before consideration of any related controls applied by the entity	Residual risk	Residual risk—the remaining risk after treatment or controls have been put in place
	Controls	Portion of inherent risk mitigated by controls

Step 8. Convene an expert advisory committee meeting, if appropriate

The strategic audit planning team may wish to convene an expert advisory committee meeting to discuss the information obtained and the potential audit topics, including the rationale for identifying each topic. If a meeting is convened, the Auditor General, the assistant auditors general, and the Commissioner of the Environment and Sustainable Development should be included in the discussion to gain a better appreciation of the risks and the importance of the topics identified.

Step 9. Consolidate results

Once the detailed work has been completed, the strategic audit planning team summarizes the results and creates a list of potential audits or other products. The list of potential topics should include products to be conducted over the following several tabling years and identify the tabling period (spring or fall) for each or whether a special report (as specified in section 8 of the Auditor General Act) may be tabled.

In addition to the potential areas to examine, the following information could also be included to support the list:

• entity mandate, priorities, objectives, programs, and services provided

• key risks, challenges, significant changes, or gaps in audit coverage

• stakeholders’ concerns and interests

• importance of auditing the topic (include examples of what is not working well and the impact)

• scope of the audit (what the audit will examine and what will be excluded)

• the audit’s impact on how services are delivered (consider timeliness, accessibility, and consistency)

• applicable SDGs, targets, and indicators to be covered by the audit

• potential key messages

• value added (assurance, advice, information, and other benefits)

• potential collaboration possible with provincial or international audit offices

• possibility of contracting an entire external audit team to perform the audit

To assist the strategic audit planning team with prioritizing the potential audit topics, the team could use a heat map (Exhibit 4). A heat map is a technique used to assess and communicate identified risks based on likelihood of the risk occurring against impact of the risk occurring.

Exhibit 4—A heat map can be used to prioritize the risks

Risk assessment—Risks could be identified as normal, elevated, or high (see the heat map noted below).		Impact: If the risk were to occur…
		Normal—The risk may affect operations but would not jeopardize outcomes and objectives. Limited impact on subject matter results or clients.	Elevated—The risk may jeopardize operations but not strategic subject matter objectives. Moderate impact on subject matter results or clients.	High—The risk may significantly affect the entity’s ability to meet strategic subject matter objectives. Significant impact on subject matter results or clients.
Likelihood— Based on professional judgment, the likelihood is…	High—The risk will probably occur in routine circumstances or is likely to be pervasive.	ELEVATED	ELEVATED TO HIGH	HIGH
	Elevated—It is not unusual for the risk to occur from time to time.	NORMAL TO ELEVATED	ELEVATED	ELEVATED TO HIGH
	Normal—It may occur in rare circumstances.	NORMAL	NORMAL TO ELEVATED	ELEVATED

Follow-up audit work—A strategic audit plan may also cover previously audited topics. Issues are examined again if they continue to be of interest to Parliament or pose a significant risk. The team ensures that enough time has elapsed since the original audit to allow the entity to implement recommendations or address issues. Other related issues that have emerged since the previous audit may also be included if they are significant and of interest to Parliament.

Products—The team should consider the best way to add value in the areas of significance. Consideration should be given to size, scope, and type of products, such as audits, studies, reviews, and updates on performance results in areas we previously audited.

Format—The final strategic audit plan could take a variety of formats, including, but not limited to

• a PowerPoint presentation

• a working paper summarizing the work undertaken and recommendations for future audits

• a list of audit topics (using the Audit Report Submission template)

The completed plan is presented to the assistant auditors general, the Commissioner of the Environment and Sustainable Development, and the Auditor General for review and discussion.

Step 10. Communicate the results of the strategic audit planning work

Once the strategic audit plan has been finalized, it should be presented to the entity’s senior management or departmental audit committee.

Equally importantly, this information should be shared with the performance audit practice and the financial audit practice. This can be done by

• posting the strategic audit plan on the INTRAnet

• presenting the strategic audit plan periodically at meetings held specifically to identify completed strategic audit plans

• presenting the strategic audit plan at performance audit practice exchanges and to the financial practice

Audit report submissions

Audit report submissions are usually developed by the teams as part of their strategic audit plans, which are discussed and approved through the annual performance audit planning process. However, submissions, including audits that are special requests from the Auditor General, can be discussed and approved outside of the annual planning process. These audit report submissions set out

• what the audit will examine and exclude

• why is it important to audit this area and why is it important to audit the topic now

• which of the United Nations’ SDGs and associated targets and indicators will be covered by the audit

• whether the audited program provides services and, if so, how the audit will examine the impact on Canadians

• whether the audit could be done collaboratively with the provinces or using an entirely external audit team (contract out)

• considerations regarding the audit’s sensitivity, complexity, and auditability (OAG Audit 4042 Audit scope and approach)

• what value the audit intends to add

• the estimated total cost of the audit

Performance audit planning process

An annual audit selection process builds on the foundation of strong strategic audit plans and knowledge of business collected by portfolio, as described above. The process described below can be adapted by the Performance Audit Practice Management Committee (PAPMC) and the Performance Audit Practice Oversight Committee (PAPOC) to respond to operational needs and to build on lessons learned from previous processes.

Each spring, engagement leaders are invited to provide audit report submissions to be reviewed, assessed, and selected for audits that will begin the following year. The ideal is to have a rolling 2–3 year audit plan, which is updated each year to confirm those in the plan and to add new ones for the next year. After being recommended by the PAPMC, the Auditor General’s audit reports are approved by the PAPOC. The process and general timing for this process is described in Exhibit 5.

Exhibit 5—Suggested timelines for the annual audit selection process  

Timelines	Steps
January–March	The team complete pre-work as appropriate. This could include strategic audit plans, updates to strategic audit plans, and a government-wide environmental scan.
March–April	The chair of the Performance Audit Practice Management Committee (PAPMC) initiates the audit selection process.
March–April	The PAPMC, and possibly the Director Committee as well, meets with the Auditor General, the Deputy Auditor General, the assistant auditors general, and possibly the Senior Panel to identify ideas and priorities.
April–May	Teams prepare audit report submissions according to the number of audits that the business planning cycle recommends.
May	Audit report submissions are integrated into the decision-making tool (backlog or database).
May–June	Submissions are reviewed and ranked by directors according to the criteria identified below.
June	Submissions are reviewed and ranked by principals according to the criteria identified below.
Summer	The PAPMC meets to identify any changes for the upcoming year and priority audits for the following year.
Summer	The PAPMC approves the audit plan and makes recommendations to the Performance Audit Practice Oversight Committee.
September	The Senior Panel, Auditor General, Deputy Auditor General, assistant auditors general, and Commissioner of the Environment and Sustainable Development review the proposed audit plan.
September	Modifications are made following the review.
October	The Auditor General makes final decisions on updates to audits planned for the upcoming year and new audits for the following year.
November	The audit plan is posted on the Product Tracking INTRAnet page and the Planned Reports Internet page.

The following 8 criteria are key in selecting audits to be undertaken and may be updated or modified from time to time:

1. Significance—How significant is the measure or activity to the achievement of the government’s goals? What does it cost? What is the impact on Canadians? What would be the impact on Canadians if the measure or activity were not implemented?

2. Risk—Is there an elevated risk of not achieving the objectives of the measure or activity due to, for example, poor design, inaccessibility, fraud, overpayment, or other weaknesses?

3. Timeliness/Preparedness—Will the measure or activity improve government preparedness in a timely manner for the future? Is this the best time for this audit? Is there a specific reason it needs to be audited now, which would make the audit untimely if the subject was audited later? For example, is there an audit team with specific knowledge of the subject, which could improve the efficiency of conducting the audit, who will not be available later or whose knowledge will erode? Or is there an event coming up (such as the renewal of an act or program or a new system) that the audit could contribute to if done now, but would be of less value if done later?

4. Auditability—Are the relevant information and entity officials available and accessible? Could the audit be conducted within an appropriate time frame? Would the audit require significant advanced planning (such as collaboration with other audit offices) or special skills or facilities that need to be identified?

5. Populations at risk—To what extent is the measure or activity intended to help populations and people who are at greater risk of serious consequences than the general population?

6. SDGs—To what extent is the measure or activity aligned with or contributing to one or more SDGs or targets?

7. GBA+—Are there likely to be GBA+ implications for the area under audit?

8. Interest—Has the topic been identified by the Auditor General, the Senior Panel, Parliament, or the public (through, for example, sustained media interest or social media scanning)?

A number of factors, in addition to the ones included, above may be considered, including

• report diversity—a range of topics of public and regional interest, with good diversity and subject matter:

  • social and economic issues

  • issues inside and outside Ottawa

  • domestic and international issues

  • entity and multi-entity issues

  • geographically spread issues

• appropriate entity coverage (over‑and under‑coverage)

• the number of audits and the associated hours

PAPOC may approve changes to the current plan throughout the year, such as new audits, audit deferrals, and audit cancellations. Each fall, the OAG’s comptroller prepares its annual budget and financial plan for the next fiscal year based, in part, on the approved performance audit schedule. The OAG’s annual budget provides for the financial and human resources required to conduct the next fiscal year’s audits. During the annual budget exercise, any amendments to the audit schedule, including changes to reporting dates, cancelling audits, and the introduction of new audits, must be approved by the PAPOC and sent to the comptroller. Increases to audit budgets should be vetted by PAPMC prior to approval.