Executive Summary

Based on the work performed in the 2016–17 fiscal year monitoring exercise, we concluded that there was reasonable assurance regarding the following:

• The policies and procedures for the Office of the Auditor General of Canada’s system of quality control were relevant and adequate. We did not identify any deficiencies.
• The system of quality control operated effectively at the Office level. We identified three Category 3 (isolated) deficiencies at the Office level that did not prevent the system of quality control from operating effectively. We did not identify any Category 1 (serious) or Category 2 (systemic, repetitive, or significant) deficiencies.
• The system of quality control operated effectively at the engagement level, as all files reviewed were either compliant or compliant with improvement needed. We identified a number of Category 3 deficiencies, seven Category 2 deficiencies, and no Category 1 deficiencies at the engagement level. The reports the Office issued were appropriate in the circumstances for the engagements the Practice Review and Internal Audit team reviewed.

The scope of the monitoring exercise included assessing the design and implementation of the Office’s system of quality control. Assessing the design addressed how relevant and adequate the system’s policies and procedures were. Assessing the implementation addressed the system’s operational effectiveness.

Table 1 describes the areas for improvement related to the system of quality control for which recommendations were made, including those made in the 2016–17 fiscal year and those that were outstanding from previous years. Table 1 does not repeat observations set out in practice review reports.

The Practice Review and Internal Audit team identified deficiencies (a number of Category 3 deficiencies and seven Category 2 deficiencies), presented findings of non-compliance on specific elements of the system of quality control, and made recommendations related to the operating effectiveness at the engagement level in both the Financial Audit and the Direct Engagement practices. You can find the recommendations in the following reports on the Office’s Internet website:

• Report on a Review of the Direct Engagement Audit Practice—Direct Engagement Audits Completed in the 2016–17 Fiscal Year
• Report on a Review of the Financial Audit Practice—Financial Audits Completed in the 2016–17 Fiscal Year

Findings from reviews of direct engagement audits. The Practice Review and Internal Audit team identified and made recommendations on the following Category 2 (systemic, repetitive, or significant) findings. The paragraph number in parentheses indicates the location of the finding in the Report on a Review of the Direct Engagement Audit Practice—Direct Engagement Audits Completed in the 2016–17 Fiscal Year.

• Independence confirmation—We found that four of the files we reviewed were missing Independence Confirmation forms for individuals who met the definition of an engagement team member. In total, 10 Independence Confirmation forms were missing from these files. In these cases, we have asked the engagement leaders to reopen the audit files to ensure that independence is assessed and documented for each team member. We have asked them to inform the Chief Audit Executive if any conflicts are identified. (Paragraph 23)
• Independence confirmation—We also found that some engagement team members had charged time to engagements before preparing their Independence Confirmation forms. We reviewed 94 completed forms and found that more than one third of engagement team members charged time to engagements before completing their forms. On average, these individuals charged about 15 hours to the engagements before completing the forms. We identified some cases in which the individuals had charged more than 30 hours to the engagements over a period of many months before completing the forms. (Paragraph 24)
• Independence confirmation—We found delays in the engagement leader’s review and approval of the Independence Confirmation forms in five of the six audit files reviewed. We rated these five files as compliant while improvement was needed. A total of 80 Independence Confirmation forms had been prepared for these five files. We noted that in about half of the forms we reviewed, individuals had charged an average of 40 hours to the audit before the engagement leader had reviewed and approved the form. We found seven cases in which more than 100 hours had been charged to the audit before the Independence Confirmation form was reviewed and approved. (Paragraph 29)
• Security of sensitive information—In our Report on a Review of the Direct Engagement Audit Practice—Direct Engagement Audits Completed in the 2015–16 Fiscal Year, we noted that audit staff needed to be made aware of the Office's security policy, and that any document stored in TeamMate should be assessed against the policy and be labelled according to the proper security level. In our review of this year’s files, we note that work still remains to be done in applying the Office’s security policy. Five of the six files reviewed included documents that were not properly labelled in accordance with the Office’s security policy. (Paragraph 33)
• Quality control review—A quality reviewer had been assigned to three of the files selected for our review. In two files, the work performed by the quality reviewers complied with Office policy requirements. In a third file, we found that the engagement quality control review was non-compliant. We could not confirm that all minimum quality reviewer responsibilities had been met. For example, we found no evidence in the audit file that the quality reviewer had reviewed key audit documents, including independence and exceptions reports and the engagement risk assessment. As a result, the assurance report was dated and issued despite an incomplete and not fully documented quality review. (Paragraph 39)

Findings from reviews of financial audits. The Practice Review and Internal Audit team identified and made recommendations on the following Category 2 (systemic, repetitive, or significant) findings. The paragraph number in parentheses indicates the location of the finding in the Report on a Review of the Financial Audit Practice—Financial Audits Completed in the 2016–17 Fiscal Year.

• Independence Confirmation form—We found that the seven files reviewed were not in compliance with one of the requirements of the Office’s policy on independence. We noted that engagement team members had charged time to the audit before completing their Independence Confirmation forms. (Paragraph 20)
• Independence Confirmation form—In performing our reviews, we found delays in the engagement leader’s review and approval of the Independence Confirmation forms. The seven files reviewed were assessed in relation to this requirement as compliant, with improvement needed. Indeed, we noted that for two thirds of the more than 150 Independence Confirmation forms reviewed by the Practice Review and Internal Audit team, individuals had charged, on average, 29 hours to the audit before the engagement leader had reviewed and approved the forms. We found many cases in which more than 40 hours had been charged to the audit before the Independence Confirmation form was reviewed and approved. (Paragraph 27)

Introduction

1. The Canadian Standard on Quality Control 1 (CSQC 1), issued by the Auditing and Assurance Standards Board, requires that a quality control system applicable to all assurance engagements be established and maintained.

2. For the Office of the Auditor General of Canada, this provides reasonable assurance that the Office and its personnel comply with professional standards and applicable legal and regulatory requirements, and that the audit reports the Office issues are appropriate in the circumstances.

3. Monitoring compliance with system of quality control policies and procedures is meant to evaluate

• whether the Office observes professional standards and applicable legal and regulatory requirements;
• whether the Office has appropriately designed and effectively implemented the system of quality control; and
• whether the Office has properly applied its system of quality control policies and procedures, so that reports the Office issued are appropriate in the circumstances.

4. The Office is required to communicate the results of the monitoring process annually to the Auditor General and to management, and to recommend appropriate remedial action where necessary. This report fulfills that requirement.

5. The Office’s monitoring process is divided into two distinct parts:

• Annual monitoring. This is a yearly evaluation of the design and operation of the Office’s system of quality control policies and procedures but excludes the inspection of specific engagement files.
• Practice review. This is a cyclical inspection of completed engagement files. A completed assurance engagement is inspected for each engagement leader at least once in every four-year period. Practice review results are taken into consideration in the annual monitoring exercise and are reported separately.

6. The annual monitoring process assesses deficiencies found according to the following categories:

• Category 1—Serious. These matters require immediate corrective action to comply with professional standards and legal and regulatory requirements.
• Category 2—Systemic, repetitive, or significant. These matters require prompt corrective action or changes in the Office’s policies and procedures.
• Category 3—Isolated. These matters require consideration but do not show that the Office’s system of quality control is deficient or that the engagement reports the Office issued were inappropriate.

7. Category 1 and 2 deficiencies reflect major weaknesses that could prevent the Office from achieving the CSQC 1 objectives.

8. This report reflects the two distinct parts of the monitoring process:

• a detailed report on the Office’s annual monitoring, and
• the summary results from the Report on a Review of the Direct Engagement Audit Practice—Direct Engagement Audits Completed in the 2016–17 Fiscal Year, and the Report on a Review of the Financial Audit Practice—Financial Audits Completed in the 2016–17 Fiscal Year.

9. The monitoring exercise covered the period from 1 April 2016 to 31 March 2017. We completed the monitoring work on 30 November 2017.

10. The audit team consisted of the following members:

• Assistant Auditor General: Stuart Barr
• Principal: Gregg Ruthman
• Director: Caroline Jean

Findings, Recommendations, and Responses

Design of the system of quality control—Adequacy and relevance

11. Overall, we found that the Office of the Auditor General of Canada’s design of the system of quality control was adequate and relevant in meeting the requirements of the Canadian Standard on Quality Control 1 (CSQC 1).

Required elements of the Canadian Standard on Quality Control

12. We found that the Office’s system of quality control included policies and procedures that addressed the six required elements of the CSQC 1.

13. We reviewed the results of previous assessments from the 2015–16 monitoring process, the Office’s audit methodology and system of quality control policies and procedures, and a detailed crosswalk that maps the CSQC 1 requirements to the system design.

Documentation and communication of policies and procedures

14. We found that the Office documented its system of quality control policies and procedures and communicated them to staff.

15. We reviewed how the Office documented and communicated the system of quality control (that is, how available and accessible it was) to ensure that it included a description of its policies and procedures and the objectives they are designed to achieve.

16. The Office documented and communicated the system of quality control policies and procedures mainly through its two product-line manuals (financial audit and direct engagement), procedure libraries for each product line, and the INTRAnet.

17. The manuals describe the system of quality control policies and procedures and the objectives they are designed to achieve.

18. The system of quality control manual clearly states that

• each auditor has a personal responsibility for quality and is expected to comply with the policies and procedures,
• audit teams are responsible for carrying out quality control procedures that apply to the assurance engagement, and
• audit teams are responsible for providing the Office with relevant information to ensure that the system of quality control functions properly.

Monitoring and maintenance

19. We found that the system of quality control was up to date. The Office has developed a process for monitoring and maintaining the system’s methodology, training, tools, and support.

20. We reviewed new developments in professional standards and regulatory and legal requirements, as well as improvements, updates, and corrections to the Office’s existing system of quality control policies and procedures. We reviewed whether the policies and procedures reflected, where appropriate, these required changes.

21. The Office updated its methodology on both an annual basis for major updates and on an ad hoc basis for more urgent updates.

22. Competent staff were responsible and accountable for monitoring and maintaining methodology. Staff performed these activities in a timely manner and monitored and took corrective action as needed.

23. The Office has a process for monitoring observations from provincial institute practice inspections and the Canadian Public Accountability Board. This process determines whether there are opportunities to improve the system of quality control to help its practitioners avoid the issues observed in other firms.

24. The Office’s INTRAnet shared announcements of methodology changes (methodology updates, standards interpretations, and notices), providing targeted communications to auditors and a historical reference of the changes made to audit methodology.

25. The Office developed a publication model to make it easier to maintain audit methodology. This model clearly defined the roles and responsibilities for monitoring and maintaining methodology, ensuring the accuracy and integrity of published system of quality control policies and procedures.

26. The Annual Audit Practice Team is responsible for a weekly process that monitors for and identifies upcoming changes proposed by Canadian and international standard setters.

27. The Office entered into a strategic alliance with PricewaterhouseCoopers whereby the Office had rights to PricewaterhouseCoopers’ audit methodology and updates. As a result, PricewaterhouseCoopers updated the Office on the changes it had made to its audit methodologies. The Office used this information to update its own system of quality control policies and procedures.

28. The Office monitors the activities of standard-setting bodies relevant to legislative auditing. This results in formal and informal discussions with practice teams as well as the monitoring of applicable professional standards. Several members of management participated in the activities of standard-setting bodies.

29. The Office consults Legal Services annually on changes to legislation and regulations that affect the methodology.

30. Engagement leaders monitored changes in enabling legislation and the operational laws and regulations of the entities that they audited. The assistant auditors general and the principals responsible for entities developed strategic relationships with senior members within the portfolio of entities that they audited.

Responses to recommendations made in previous monitoring reports

31. We found that the Office responded to the recommendations made in previous monitoring reports.

32. We reviewed responses to recommendations made in previous monitoring reports.

33. Previous monitoring reports. The Monitoring Report on the System of Quality Control—2015–16 Fiscal Year identified no serious deficiencies in the system of quality control. The report did note three isolated issues. These issues were determined not to affect the effectiveness of the system’s operation; these issues were as follows:

• The People Management Framework and related Human Resources policies and procedures did not have a formal recruitment strategy.
• The Office did not have a formal process or procedures in place to select and appoint the internal specialists.
• The Office did not maintain summary information on the nature and extent of consultations with internal specialists.

34. The Office acted on these issues as follows:

• In spring 2017, the Office implemented its Resourcing Strategy and posted it on the INTRAnet.
• In fall 2016, the Assistant Auditor General of Audit Services implemented the use of criteria for selecting and appointing internal specialists.
• In spring 2017, the Office communicated requirements and guidelines for documenting summary information on the nature and extent of consultations with internal specialists.

Operational effectiveness of the system of quality control at the Office level

35. Overall, we found that the Office of the Auditor General of Canada’s system of quality control operated effectively at the Office level.

Internal culture of quality

36. We found that the Office promoted an internal culture of quality through clear, consistent, and frequent messages and rewarded high-quality work.

37. We reviewed actions and messages that emphasize the requirement to perform work that complies with professional standards and to issue reports that are appropriate in the circumstances. We also assessed whether the Office provided enough resources to develop, document, and support its system of quality control policies and procedures.

38. Senior management actions and messages. The Office clearly states and communicates its vision and values, as well as its Code of Values, Ethics and Professional Conduct. Awards programs recognize staff members who promote the Office’s values, including product management and quality. The Office’s orientation training program includes an e-learning session to provide participants with a better understanding of the Office’s purpose, culture, and role in government. This e-learning is mandatory for all new hires and ensures that the culture of quality is made clear to all staff. Sharing the results of practice review activities with staff, including recommendations, helps promote a culture of quality and continuous improvement. During the period under review, the assistant auditors general of the audit practices and the Assistant Auditor General of Audit Services reviewed the 2015–16 fiscal year results. The Chief Audit Executive presented the results at meetings of the principals and directors of financial audits and direct engagements.

39. The Office’s appraisal, hiring, promotion, and compensation processes require a demonstration of the Office’s competency models, which include the quality of work expected.

40. Senior management responsibilities for quality. The Office clearly assigns the roles and responsibilities for the elements of the system of quality control to senior management, who have the appropriate authority to fulfill their related duties.

41. The Auditor General assumes ultimate responsibility for the Office’s system of quality control. The Auditor General appoints the Assistant Auditor General of Audit Services, who is assigned operational responsibility and has an appropriate combination of education, professional qualifications, experience, and skills to fulfill this function’s duties. The Assistant Auditor General has the necessary authority to fulfill these responsibilities.

42. Sufficient resources to support the system of quality control. The Office has enough resources to develop, document, and support the system of quality control. This includes the resources and processes for monitoring new developments in professional standards and integrating the identified changes into the audit methodology in a way that ensures consistency and completeness. Audit Services is the operational centre for the system of quality control and has resources from two product-line practice teams—the Annual Audit Practice Team and the Direct Engagement Practice Team. The practice teams conduct the following activities:

• monitor for new developments in professional standards, laws, and regulations;
• coordinate a common look and feel for methodology;
• provide quality assurance and advice; and
• ensure the accuracy and integrity of published methodology, including TeamMate procedures.

Relevant ethical requirements

43. We found that all staff who were required to complete an annual Conflict of Interest Declaration form for the 2016–17 fiscal year did so.

44. We found that the Internal Specialist, Values and Ethics, assessed exception reports initiated in the 2016–17 fiscal year and applied appropriate safeguards where necessary.

45. We found that the Office had an annual process for evaluating and managing rotation requirements.

46. We reviewed the processes for annual confidential declarations, the identification of threats to independence in exception reports, and job rotation analysis and actions.

47. Annual confidential declarations. To demonstrate their understanding of these fundamental principles and their compliance with Office protocols, employees must read, understand, and adhere to the Office’s Code of Values, Ethics and Professional Conduct. Adhering to ethical requirements includes signing an annual Conflict of Interest Declaration form (annual confidential declaration) and an Independence Confirmation form before beginning work on any assurance engagement. If employees identify threats to compliance with ethical requirements or independence, they must complete an Exception Report to help resolve the threat.

48. The Principal of Human Resources emails conflict of interest requirements annually to staff, and the Office maintains an automated mandatory annual process that requires staff to confirm their compliance with independence policies and procedures. The system sends the request to all users and tracks the progress from the request initiation, to the printing, and to the delivery to Human Resources and, ultimately, to Information and Records Management. The Principal of Human Resources generates reports that track the progress and completion rate. The system automatically sends reminders to staff who have not completed the declaration. For the 2016–17 fiscal year, all staff members who were required to complete an annual declaration did so and in a timely manner. However, the Practice Review and Internal Audit team identified that engagement-level confirmations of independence had in many cases not been prepared or reviewed before work on the related assurance engagements began.

49. Exception reports. Staff members are required to promptly notify the Office of any circumstances or relationships that create threats to independence. If the threat is significant, the employee is required to initiate an Exception Report, which identifies the threat and documents its impact and the appropriate action required to eliminate the threat or reduce it to an acceptable level. The Internal Specialist, Values and Ethics, reviews the report objectively and assesses the proposed safeguards, which may include additional actions to reduce the threat to an acceptable level. These safeguards reflect the individual’s level of influence on an audit and may include the following:

• increasing the level of supervision of the individual on the audit,
• segregating the individual from certain audit lines of enquiry, or
• removing the individual from the audit.

50. The Internal Specialist, Values and Ethics, assessed all exception reports initiated in the 2016–17 fiscal year and applied appropriate safeguards where required.

51. Job rotation. The Office’s objectivity may be threatened or appear to be threatened if senior personnel and quality reviewers, where applicable, continue to work with the same entity for a prolonged period. Staff rotation is often achieved through promotion or staff turnover; however, the responsibilities of engagement leaders with signing authority are less likely to change unless a policy requires staff rotation. The Office job rotation policies require that, each year, Audit Services identify engagement leaders requiring job rotation to the assistant auditors general of the applicable audit practice for consideration and approval. Exceptions to job rotations require approval and are granted only if appropriate safeguards exist.

52. We found that the Office performed an appropriate job rotation analysis. For the situations where extensions were required, the Office approved exceptions appropriately.

Acceptance and continuance requirements

53. We found that the Office had processes to ensure that audit staff adhered to the principles of acceptance and continuance and applied them to all assurance engagements.

54. We reviewed Executive Committee records of decisions and interviewed Legal Services to determine whether audit staff had followed acceptance and continuance processes at the Office level, and whether the Office had identified and resolved any threats of familiarity with an entity.

55. For a legislative audit office such as the Office of the Auditor General of Canada, many assurance engagements are required by legislation; the Office conducts other engagements at its discretion. For discretionary audits, the Office refers all requests for appointment by order-in-council or under the Financial Administration Act to Legal Services to determine whether the Office has the authority to conduct the engagement.

56. At the engagement level, for both discretionary and statutory audits, engagement leaders perform and document acceptance procedures for all new engagements. If the Office decides it needs to withdraw from an engagement, where that option is available, the engagement leader prepares a briefing note and presents it to the Executive Committee for review. Legal Services may analyze whether there is a professional, legal, or regulatory requirement to remain as auditor or whether the Office should report the withdrawal, cancellation, or postponement, and the justification for that decision, to others outside the Office.

57. We found that, during the 2016–17 fiscal year, the Office completed the required Office-level acceptance and continuance procedures and reviewed all threats or acceptance and continuance actions.

Staff sufficiency, competence, capabilities, and commitment to ethical principles

58. We found that the Office assessed the competencies and capabilities it required at the practice level. It developed a global staffing profile and a People Management Framework, which helped to assess competence and the commitment to ethical principles. It also implemented a Resourcing Strategy shortly after the period covered by this report. However, we found that the Office did not regularly update its reviews of the sufficiency of personnel and the appropriateness of key assumptions used.

59. We reviewed documentation and carried out interviews on the following:

• assessing staffing needs,
• hiring and promotion,
• use of specialist skills,
• assigning professional personnel,
• staff training and professional development, and
• performance management.

60. Assessing staffing needs. Through the budget management and approval process, the Office establishes and monitors its financial capacity to respond to its resourcing needs.

61. The Office has an annual process to determine professional staffing needs at the audit professional and management levels. This process considers current and anticipated availability, competencies, and capabilities of resources, as well as the realistic profiles of the practices.

62. In 2014, the Office reviewed its Office-level governance and senior-level functions, so it could redefine these roles and responsibilities to eliminate duplicating functions and increase efficiencies. As part of this review, the audit practices, supported by the Audit Resource Planning and Career Management team, took part in a “realistic profile for audits” initiative. This initiative produced a staff profile for each audit practice. Each profile establishes the number of staff needed at each level (audit professional and management). The profiles are based on key assumptions and are used as a benchmark to forecast resource needs and used in hiring and promotion decisions. After the period covered by this report, the realistic profiles were updated.

63. Although the Office makes significant efforts to assign audit professionals, its reviews of the sufficiency of personnel and the appropriateness of key assumptions used at the Office level are not frequent.

64. Recommendation. Audit practices should perform more frequent reviews of the sufficiency (capacity) of personnel and the appropriateness of key assumptions used.

Management’s response. Agreed. The Financial Audit Practice has recently completed an updated realistic profile to support funding requests of the Office of the Auditor General of Canada. This updated profile reflects a reassessment of products and their budgets, productivity at individual staff levels, and other inputs. The practice has also undertaken to identify and measure key capacity variables, such as overtime, the use of generalist contracts, and on-time and on-budget performance. Periodic reporting of these measures is expected throughout the upcoming financial audit period.

At the end of the current audit period (fall 2018) and annually thereafter, the Financial Audit Practice will review each capacity variable in terms of the relevance of the individual items, the progress in achieving the assumed targets, and the overall impact on the capacity of the practice to meet its assigned deliverables. Based on this assessment, actions will be taken where necessary, including the possibility of revising the underlying assumptions, improving internal processes, or developing appropriate contingency plans.

Similarly, in December 2017, the Direct Engagement Practice completed and approved a realistic (desired) team profile. The outcome of our funding request will determine if this profile needs to be revised. If so, this will be done accordingly. In addition, more work will be needed to analyze the overall breadth of educational background and technical training/expertise within the practice. Some of the analysis will depend on the availability of better information from our Human Resource Information System. This exercise should be completed by March 2019.

65. Hiring and promotion. The Office employed staffing and promotion approaches and tools to assess staff competencies and capabilities.

66. The goal of the Financial Audit Trainee and Performance Audit Trainee programs is to recruit, train, and retain employees with the general competencies required to become good financial and performance auditors. The Financial Audit Trainee program recruits university students from accounting programs to fill the permanent needs in the Office’s financial auditing operations. The Performance Audit Trainee program recruits students who have master’s degrees from Canadian universities in general or specific fields of interest to the practice. The Office has determined that these requirements provide the required general competencies and capabilities.

67. During the course of these two- or three-year programs, trainees must demonstrate that they meet additional specific Office competencies, which include delivering products according to the system of quality control.

68. The Office uses formal staffing posters to advertise positions and structured interview guides with behaviour-based, scenario, and personal experience questions to assess competencies and capabilities. Hiring managers receive coaching prior to participating on selection boards and completing reference checks.

69. The People Management Framework and related Human Resources policies and procedures have included a formal Resourcing Strategy since spring 2017.

70. We found that the process to hire and promote staff was appropriate and operated effectively.

71. Use of specialist skills. The Office used specialists’ skills in the course of its work by means of advisory committees made up of external experts and internal specialists. It identified and reviewed the functional areas for internal specialists. In fall 2016, it implemented the use of criteria to select and appoint internal specialists.

72. Assigning professional personnel. The Office had a process for assigning professional personnel to audit engagements. Directors worked with the Audit Resource Planning and Career Management team to assess and document the assignment of appropriate staff with the necessary competencies to the assurance engagements under their responsibility.

73. The Principal is responsible for validating that the staff mix of the engagement team, specialists, and any audit experts collectively have the appropriate competencies and capabilities to meet the requirements of audit and assurance standards.

74. While principals are responsible for the resource planning of audit teams, the Audit Resource Planning and Career Management team supports them in the following ways:

• managing and monitoring the corporate resource planning tool, Retain, a resource database that contains information by auditor and audit product;
• analyzing resource needs to support assistant auditors general in audit practices, senior Office committees, and audit teams;
• assisting staff transfers and sharing among groups; and
• identifying possible solutions for critical resource requirements.

75. The Audit Resource Planning and Career Management team may be consulted

• by directors or principals when an audit team is looking for a resource or for an assignment or transfer for their auditors, and
• by auditors who are looking for an assignment or a transfer.

76. We found that the process to assign professional personnel was appropriate and operated effectively within each practice.

77. Staff training and professional development. In its learning vision, the Office states that it is committed to building and promoting a learning culture that adds value to its work for Parliament and Canadians and supports the lifelong learning of Office employees.

78. For a few years, the Office has invested heavily in renewing the audit training curriculum, methodology, and tools. The Office used a training needs analysis that assesses training and professional development needs by competency and skill level, developed a professional development business plan to address gaps and opportunities and add to its value proposition, created training and professional development budgets, and dedicated resources to training and professional development.

79. The Office has developed the Leadership Program, which focuses on people management, to meet the professional development needs of the Office’s leaders and assist continuous learning in this area. Leadership is a key component of the system of quality control. The program follows a multi-dimensional approach that includes formal training, interactive knowledge-sharing events, practical tools and resources, and coaching, as well as support services to resolve issues.

80. The Office has a vision for learning that focuses on continuous learning beyond the classroom. One of the key elements of this vision is emphasizing on-the-job coaching and offering on-the-job experiences that are relevant to staff. The Office’s Professional Development team’s role is to give staff the best formal training possible and to help managers provide feedback and coaching as staff experiment with newly acquired skills.

81. The Professional Development team does an annual scan of the training and professional development environment by consulting with management, reviewing training evaluations, and consulting with accounting firms (that is, PricewaterhouseCoopers and Deloitte) on what is happening in the industry. Based on the results, the Professional Development team updates training and professional development initiatives.

82. Performance management. In accordance with professional standards, the Office had an annual performance management system in place that required managing products to a high level of quality.

83. The Office has a process for performance management in place that includes goal setting, competencies, ongoing feedback, assessment processes, corrective actions, training and development, and career planning. All active staff receive mandatory annual performance appraisals.

84. The performance appraisal process includes assessing values and competencies and being required to manage products to a high level of quality according to standards. For any quality-related issues that management identifies, Human Resources helps to remedy the situation through coaching and mentoring, more frequent follow-ups, training, and other appropriate corrective actions.

85. We found that during the 2016–17 fiscal year, the Office completed performance appraisals for active staff. Human resources monitored and followed up on performance appraisals to ensure that they were completed for all active staff.

Complaints and allegations

86. We found that the Office encouraged the reporting of complaints and allegations about the conduct of its work.

87. We reviewed documentation and interviewed Legal Services about complaints and allegations it received about failing to comply with professional standards or the Office’s system of quality control for work the Office performed during the period under review.

88. The Office’s policies—Office of the Auditor General of CanadaOAG Audit 1012 Audit Quality and OAG Audit 1091 Complaints and Allegations—meet the Canadian Standard on Quality Control 1 (CSQC 1) requirements for addressing complaints and allegations. The Office communicates these policies to all employees by means of the INTRAnet.

89. The Office receives external and internal complaints through a public inbox managed by the Communications team. Complaints are then tracked in a database, and the Auditor General or the Chair of the Audit Committee and whoever is appointed as investigator addresses and investigates them. The targeted response time on all issues is 90 days.

90. The Executive Committee receives a quarterly status report on all closed and outstanding complaints and allegations. For the period under review, the Office received one complaint or allegation either internally or externally about how it conducted its audits regarding the system of quality control. The investigation did not identify deficiencies in the design or operation of or non-compliance with the Office’s system of quality control.

Monitoring process

91. We found that the Office communicated the results of the monitoring process.

92. We reviewed the publications of the monitoring reports on the Office’s Internet website and the corporate messages to engagement leaders announcing the completion of the Monitoring Report on the System of Quality Control.

93. The Office’s policy, OAG Audit 1012 Audit Quality, meets the CSQC 1 requirements for communicating the results of the monitoring process. The Office communicates this policy to all employees by means of the INTRAnet.

94. The results of the monitoring process are published on the Office’s Internet website and include the following reports:

• Monitoring Report on the System of Quality Control—2015–16 Fiscal Year
• Report on a Review of the Financial Audit Practice—Financial Audits Completed in the 2015–16 Fiscal Year
• Report on a Review of the Direct Engagement Audit Practice—Direct Engagement Audits Completed in the 2015–16 Fiscal Year

95. The Office communicates the completion of the Monitoring Report on the System of Quality Control to all engagement leaders.

Cases of inappropriate reports and omitted procedures

96. We found that the Office complied with the requirements to address cases in which the results of the monitoring procedures showed that a report may have been inappropriate or that procedures were omitted during the performance of the engagement.

97. We reviewed the Protocol for Practice Reviews and Internal Audits.

98. The Office’s Protocol for Practice Reviews and Internal Audits defines the process in cases where the monitoring procedures’ results show that a report may be inappropriate or that procedures were omitted during the performance of the engagement.

Operational effectiveness of the system of quality control at the engagement level

99. Overall, we found that the Office of the Auditor General of Canada’s system of quality control operated effectively at the engagement level. In the practice reviews of engagements, the auditors’ reports were supported and appropriate.

100. Seven of the deficiencies that the Practice Review and Internal Audit team noted represented systemic, repetitive, or significant deficiencies requiring prompt corrective action.

Consultations with Office specialists

101. We found that audit teams consulted the various Office specialists and documented the extent of their consultations as required by the system of quality control.

102. We reviewed consultation data and details from audit teams of various specialists.

103. The Office’s policy, OAG Audit 3081 Consultations, defines the importance of consultations within the conduct of audits so that the Office can reduce the risk of error and improve how audit teams apply professional judgment. The policy also defines the process for consultations and the requirements for documenting the consultations.

104. We found that audit teams consulted Office specialists when dealing with complex, unusual, or unfamiliar issues.

105. We also found that information on the extent, details, and conclusions of consultations with internal specialists, although available, was not consistent. In spring 2017, the Office communicated requirements and guidelines for documenting summary information on the nature and extent of consultations with internal specialists.

Compliance with professional standards for quality control

106. We found that the Practice Review and Internal Audit team made recommendations about engagement performance and presented findings of non-compliance and compliance with improvement needed on specific elements of the system of quality control in the Financial Audit and the Direct Engagement practices.

107. We found that seven of the deficiencies that the Practice Review and Internal Audit team noted represented systemic, repetitive, or significant deficiencies requiring prompt corrective action.

108. We reviewed practice review audit programs and the reports on the reviews of the practices for the 2016–17 fiscal year to determine whether they complied with the system of quality control at the engagement level. The Practice Review and Internal Audit team reviewed seven financial audit files and six direct engagement files in the following areas:

• supervision and review,
• engagement quality control review,
• differences of opinion,
• engagement documentation, and
• ethics and independence.

109. Supervision and review. Ensuring that team members complete the Office’s assurance engagements to the highest quality requires the Office to adequately supervise team members and to review audit work and documentation. Supervision is important to ensure that engagement teams are organized and that the quality of the work produced during the engagement is monitored for quality. Review is important to ensure that

• team members performed the work according to professional standards,
• the work supports the conclusions reached,
• the evidence obtained is sufficient and appropriate, and
• team members achieved the objectives of the engagement procedures.

110. Engagement quality control review. Quality reviews objectively evaluate the significant judgments the engagement team made and the conclusions reached in formulating the assurance engagement report. The Office assigns quality reviewers to each financial audit of entities that issue or have securities outstanding in public markets. The Office also assigns quality reviewers to other assurance engagements based on the assurance engagement’s level of risk. Quality reviewers have the technical qualifications to perform the role, as well as sufficient and appropriate experience and authority.

111. The Office has a process for selecting and appointing quality reviewers. The Office selects quality reviewers based on the engagement’s level of risk. Audit Services receives risk assessment input from each audit team and prepares a risk assessment for all engagements using selection criteria outlined in the methodology. It is normally recommended that engagements assessed as high risk be selected for a quality review. Low- to medium-risk audits are not normally assigned a quality reviewer.

112. For engagements selected for a quality review, Audit Services appoints a quality reviewer based on specific criteria. Once Audit Services compiles all risk assessments, it consults assistant auditors general on the recommended quality reviewer selection and appointment.

113. Differences of opinion. During the course of an assurance engagement, the team, those consulted about the assurance engagement, and the engagement leader and quality reviewer may have differences of opinion. Audit team members have the right to form their own conclusions on significant matters in the areas of the assurance engagement for which they are responsible and ensure that their views receive adequate consideration. Teams should not date an assurance engagement report until team members resolve all differences of opinion.

114. Engagement documentation. This component addresses the confidentiality, safe custody, integrity, accessibility, retrievability, and retention of engagement documentation and the completion of the final assembly of engagement files on time. Although our review did not observe the completion of the final assembly of engagement files after the 60-day completion date, the Office’s process of finalizing engagement documentation is not properly designed as it may not ensure documentation has been completed on or before the 60-day completion date.

115. Recommendation. The Office should strengthen its controls for engagement documentation to make it possible to verify that the documentation has been completed on or before the 60-day completion date.

Management’s response. Agreed. The Information and Records Management team will coordinate with the appropriate group to ensure that engagement dates are communicated regularly.

116. Ethics and independence. This component addresses the audit staff independence requirement to the engagement.

117. The detailed observations from monitoring the operational effectiveness of the system of quality control at the engagement level can be found in the following reports on the Office’s Internet website:

• Report on a Review of the Direct Engagement Audit Practice—Direct Engagement Audits Completed in the 2016–17 Fiscal Year
• Report on a Review of the Financial Audit Practice—Financial Audits Completed in the 2016–17 Fiscal Year

Assessment and documentation of engagement team competencies

118. We found that the engagement leader assessed and documented engagement team competencies.

119. We reviewed practice review audit programs and the reports on the reviews of the practices for the 2016–17 fiscal year (six direct engagements and seven financial audits) to determine whether team competencies were assessed and documented.

120. Before the planning or survey phase was completed, the engagement leader assessed the team in order to be satisfied that members, specialists, and others collectively had the appropriate competence and capabilities; the engagement leader documented the assessment.

Documentation of engagement team consultations

121. We found that the engagement teams performed and documented appropriate consultation.

122. We conducted a detailed review of randomly selected direct engagement and financial audit files for the 2016–17 fiscal year to determine the nature of the consultations undertaken and whether they were sufficient.

123. Audit teams consult with internal and external specialists and senior Office staff when dealing with difficult or contentious matters or other matters requiring specialized knowledge or experience. Before the date of the assurance report, both the individual seeking consultation and the party consulted agree to the nature and scope of consultations, and the conclusions resulting from them. The teams then carry out the conclusions resulting from consultations.