Executive Summary

The Canadian Standard on Quality Control (CSQC 1) from the Chartered Professional Accountants of Canada (CPA Canada) requires that the Office of the Auditor General of Canada (OAG) establish and maintain a quality control system applicable to all the Office’s assurance engagements. This provides reasonable assurance that the Office and its personnel comply with professional standards and applicable legal and regulatory requirements, and that audit reports issued by the Office are appropriate in the circumstances.

The Office is required to communicate the results of the monitoring process annually to the Auditor General and management, and to provide recommendations for appropriate remedial action where necessary. This report fulfills that requirement.

This monitoring report reviewed the Office’s system of quality control (SoQC) for the period from January to December 2013. The scope of the monitoring exercise included assessing the design (relevance and adequacy) and implementation (operational effectiveness) of the system of quality control. The monitoring work focused on each element of the system, as well as two additional higher risk areas—the assessment of engagement team competencies and the adequacy of consultation by the engagement team.

Overall, based on this evaluation, the Office’s system of quality control is relevant and adequate, and is operating effectively. We did not identify any serious deficiencies.

We did, however, note one area for improvement. This is related to the operational effectiveness of the system of quality control, although this weakness did not affect the adequacy or relevance of the system or the appropriateness of the audit reports issued. Specifically:

• The Office uses specialist skills in the course of its work by means of advisory committees that comprise external experts and internal specialists. The Office has identified and annually reviews the functional areas for internal specialists. While we note that identifying internal specialist areas is done well, we found that no formal training and development are provided for internal specialists to maintain their skills and any related designations.

Management’s response. For 2015–16, the Assistant Auditor General of the Professional Practices Group will have a process in place to identify internal specialist training needs and will provide support to internal specialists to meet those needs.

Introduction

Background

1. The Canadian Standard on Quality Control (CSQC 1) from the Chartered Professional Accountants of Canada (CPA Canada) requires that the Office of the Auditor General of Canada (OAG) establish and maintain a quality control system applicable to all the Office’s assurance engagements, including annual audits, performance audits, and special examinations. The objectives of CSQC 1 are to provide reasonable assurance that the Office and its personnel comply with professional standards and applicable legal and regulatory requirements, and that the audit reports issued are appropriate in the circumstances.

2. A system of quality control consists of policies designed to achieve the objectives established by CSQC 1 and the procedures necessary to implement and monitor compliance with those policies. Table 1 describes each of the elements of CSQC 1.

Focus of monitoring

3. CSQC 1 requires the Office to establish a monitoring process that will assess compliance with its quality assurance manual and professional standards on an ongoing basis, report on whether a quality control system has been appropriately designed and carried out, and provide recommendations to correct the deficiencies identified.

4. The objective of monitoring is to provide reasonable assurance that the policies and procedures that make up the Office’s system of quality control (SoQC) are relevant, adequate, and operating effectively. According to CSQC 1, monitoring is an ongoing consideration and an evaluation of the OAG’s system of quality control (“policy monitoring”). This includes inspecting at least one completed engagement for each engagement leader on a cyclical basis (“completed file monitoring”).

5. Monitoring is conducted as an evaluation for management, and the results of monitoring are reported according to the requirements of CSQC 1. The monitoring report is intended for management purposes. The Office is required to communicate the results of the monitoring process annually to the Auditor General and to management, and to provide recommendations for appropriate remedial action where necessary. This report fulfills this requirement.

6. Monitoring is performed as an active part of the OAG’s system of quality control. The purpose of monitoring compliance with quality control policies and procedures is to evaluate

• adherence to professional standards and applicable legal and regulatory requirements;
• whether the system of quality control has been appropriately designed and effectively carried out; and
• whether the OAG’s quality control policies and procedures have been properly applied, so that reports issued by the OAG are appropriate in the circumstances.

7. The results of monitoring enable management to determine whether key components of internal control continue to function over time, and to identify and communicate internal control deficiencies in a timely manner to those parties responsible for taking corrective action, and to others, as appropriate. The findings and recommendations help to ensure that the Office’s risks are properly managed over time. They can also help to standardize practice among practitioners and ensure that best practices are consistently used.

8. The scope of the monitoring exercise includes assessing the design and implementation of the Office’s system of quality control. Assessing the design addresses how relevant and adequate the SoQC policies and procedures are, while assessing the system’s implementation addresses its operational effectiveness.

9. This monitoring report reviewed the system of quality control for the period from January to December 2013. To gain a more complete understanding of the significant systems and practices we assessed, we also reviewed certain matters that precede the period covered by the review.

10. Our monitoring work focused on each element of the system of quality control, as well as two additional areas considered to be of potential concern—assessing and documenting engagement team competencies, and whether the engagement team consulted adequately. Table 2 provides an overview of monitoring coverage of quality control elements as required by CSQC 1.

11. The Office’s monitoring process is divided into two distinct parts, as required by CSQC 1:

• Annual monitoring (“policy monitoring”)—a yearly evaluation of the Office’s compliance with its system of quality control policies and procedures (the objective of this monitoring exercise) but excluding the inspection of specific engagement files.
• Practice review (“completed file monitoring”)—a cyclical inspection of completed engagement files where at least once in every four-year period a completed assurance engagement is inspected for each engagement leader. Practice review results are taken into consideration in the annual monitoring exercise and are also reported separately.

12. This report includes the summary results from the Practice Review team’s engagement file inspections (that is, “completed file monitoring”). The annual summary report from file inspections will follow the usual separate review and communication process that includes public disclosure. Although CSQC 1 requires that file inspections be part of the overall monitoring process, it does not require that the results of monitoring the system of quality control and the results from engagement-level file inspections be communicated at the same time.

13. The annual monitoring process assesses deficiencies found according to the following categories:

• Category 1—Serious: matters that require immediate corrective action to comply with professional standards and legal and regulatory requirements.
• Category 2—Systemic, repetitive, or significant: matters that require prompt corrective action or changes in the Office’s policies and procedures.
• Category 3—Isolated: matters that require consideration but that do not show that the Office’s SoQC is deficient or that the engagement reports issued are inappropriate.

14. For purposes of this report, Category 1 and 2 deficiencies refer to major weaknesses that could prevent the Office from achieving the objectives of CSQC 1.

Findings and Recommendations

Design of the system of quality control—Adequacy and relevance

15. We found that overall the system of quality control (SoQC) of the Office of the Auditor General (OAG) is adequate and relevant in meeting the requirements of the Canadian Standard on Quality Control (CSQC 1).

16. The Office has established a system of quality control, which comprises policies and procedures that address the six elements required by CSQC 1. The Office documents and communicates the SoQC to staff to ensure quality work is performed consistently. To ensure the system’s continued relevance, the Office has a process in place to monitor and maintain quality control methodology.

The system of quality control addresses all required CSQC 1 elements

17. We found that the Office’s system of quality control includes policies and procedures that address the six required elements of CSQC 1.

18. This finding is important because CSQC 1 requirements are designed to enable the Office to achieve reasonable assurance that

• the OAG and its personnel comply with professional standards and applicable legal and regulatory requirements, and
• reports issued by the OAG are appropriate in the circumstances.

19. Applying these requirements properly is expected to ensure that quality work is performed by the Office and its staff on a consistent basis and according to professional standards.

20. We made no recommendation in this area.

21. The Office conducted extensive work during the Office’s Renewal of Audit Methodology (RAM) project to ensure that all CSQC 1 requirements were addressed, through extensive guidance from PricewaterhouseCoopers (PwC), crosswalks, and consultations. Work done for the 2012 monitoring process validated this finding.

22. We reviewed the results of previous assessments from the 2012 monitoring process and work done under the RAM project, an extensive initiative started in 2009 to align all Office methodology with professional standards.

The Office documents its system of quality control policies and procedures, and communicates them to staff

23. We found that the Office documents its system of quality control policies and procedures, and communicates them to staff.

24. This finding is important because the performance of quality work on a consistent basis requires policies that are documented and communicated and procedures that are supported by Office leaders who are committed to quality. Proper documentation and communication supports the development of an Office culture that fosters a positive attitude toward quality, complies with professional standards, and monitors the Office’s controls.

25. We made no recommendation in this area.

26. The SoQC policies and procedures are documented and communicated mainly through the Office’s three product line manuals (annual audit, performance audit, and special examination), TeamMate libraries for each product line, and the INTRAnet. The three product line manuals, in addition to TeamMate libraries and the INTRAnet, are the complete system of quality control.

27. The manuals include a description of the quality control policies and procedures, and the objectives they are designed to achieve.

28. The SoQC manual clearly states that each auditor has a personal responsibility for quality, and is expected to comply with the policies and procedures. It further states that audit teams are responsible for carrying out quality control procedures that apply to the assurance engagement, and for providing the Office with relevant information to ensure the system of quality control functions properly.

29. We reviewed the Office’s documentation and communication (availability and accessibility) of its system of quality control to ensure that it includes a description of the quality control policies and procedures, and the objectives they are designed to achieve.

The system of quality control is up to date

30. We found that the system of quality control is up to date. The Office has developed a system for monitoring and maintaining the system’s methodology, training, tools, and support.

31. This finding is important because the Office can only have reasonable assurance that its quality control policies and procedures are relevant if they are up to date. CSQC 1 requires that the Office analyze new developments in professional standards and applicable legal and regulatory requirements, and how they are reflected in the OAG’s policies and procedures. Maintaining up-to-date quality control methodology entails monitoring the key sources that affect the methodology and integrating changes identified in a way that ensures consistency and completeness throughout policies and procedures, TeamMate libraries and templates, guidance, and checklists.

32. We made no recommendation in this area.

33. The Office updates its methodology on both an annual basis for major updates, and on an ad hoc basis for more urgent updates. The most recent annual update was released in July 2013, and the fall 2014 update is in progress. Ad hoc updates have been presented throughout the year, as required.

34. The Office has established responsibility and accountability for methodology monitoring and maintenance with competent personnel who perform the activities in a timely manner, monitor, and take corrective action as needed.

35. For the attest product line, a process is in place to monitor observations from provincial institute practice inspections and the Canadian Public Accountability Board to determine if there are opportunities to improve the system of quality control to help our practitioners avoid the issues observed in other firms.

36. The OAG INTRAnet captures and displays announcements of methodology changes (Methodology Updates, Standards Interpretations, and Notices), providing targeted communications to auditors and a historical reference of the nature of the changes made to audit methodology.

37. The Office developed a publication model designed to facilitate the ongoing maintenance of audit methodology. This model clearly defines the roles and responsibilities for monitoring and maintaining methodology, ensuring the accuracy and integrity of published quality control policies and procedures.

38. All changes to methodology are logged by a coordinator, who also retains related information on consultations and approvals.

39. The Annual Audit Practice Team (AAPT) has an established weekly process designed to monitor and identify upcoming changes proposed by Canadian and international standard setters. An upcoming change identified as having an impact on the SoQC is communicated to the central SoQC team.

40. The OAG has entered into a strategic alliance with PwC whereby the OAG has rights to the PwC audit methodology and updates. As a result of this alliance, the OAG receives semi-annual updates reflecting the changes PwC has made to its audit methodologies. The OAG uses this information to update its SoQC policies and procedures.

41. The Office monitors the activities of standard setting bodies for legislative auditing. This results in formal and informal communications with practice teams and monitoring applicable professional standards. Several members of management at the OAG participate in activities of standard setting bodies.

42. Legal Services is consulted annually on changes to legislation and regulations affecting the methodology.

43. Portfolio assistant auditors general and principals monitor changes in enabling legislation and the operational laws and regulations of the entities that they audit. Entity assistant auditors general and principals develop strategic relationships with senior members within the portfolio of entities that they audit. If a department or Crown corporation becomes aware of an initiative to change legislation or regulations that may have an audit impact on methodology, the Office is informed.

44. We reviewed new developments in professional standards and regulatory and legal requirements, as well as improvements, updates, and corrections to the Office’s existing system of quality control policies and procedures. We reviewed whether these required changes were reflected, where appropriate, in the policies and procedures.

The Office responds to recommendations made by previous monitoring and other reports

45. We found that the Office responds to recommendations made by previous monitoring and other reports.

46. This finding is important because identifying any weaknesses in the Office’s system of quality control is crucial for taking timely corrective action.

47. We made no recommendation in this area.

48. Area reviewed: Previous monitoring reports. The 2012 monitoring report identified no “serious” deficiencies in the system of quality control. The report did note four “isolated” issues. These were determined not to impact the effectiveness of the system’s operation. These four areas were as follows:

• Senior personnel in some instances remained on audit engagements for periods exceeding Office policy with no documented approval for these exceptions. These instances have since been addressed, and the Assistant Auditor General of Corporate Services has been assigned responsibility for rotating senior personnel to support timely compliance with rotation requirements.
• A lack of clarity existed about the selection and appointment of internal specialists for all product lines. The Assistant Auditor General of the Professional Practices Group has since been assigned overall responsibility for selecting and appointing internal specialists across all product lines. In 2013, the Executive Committee approved the process for the annual update of internal specialist positions, which includes new principles and criteria. The Committee also approved the 2013 list of internal specialist positions. The Assistant Auditor General of the Professional Practices Group is currently leading the exercise to update the list for 2014, as required.
• Internal allegations or complaints about non-compliance with the system of quality control were directed to the Auditor General. Given the Auditor General’s role in the outcome of audits, this called into question the perception of his ability to exercise his role with an appropriate level of independence. The Executive Committee approved maintaining the current process for complaints and allegations of non-compliance with professional standards, with an additional avenue through the Chair of the OAG Audit Committee. To date, no internal or external complaints or allegations have been raised about compliance with the system of quality control.
• Various lists of the Office’s audit engagements were prepared by different parts of the Office. Because this is key management information, the Office should have one complete and accurate list to ensure consistency. To respond to this, Corporate Services has undertaken the Product Resource Management project. This is designed to streamline the development and maintenance of this management information.

49. The Executive Committee approved the submission to the Auditor General of the 2012 Monitoring Report on the System of Quality Control. The Executive Committee accepted the recommendations in the report and agreed to take the necessary steps to address the issues.

50. Area reviewed: Provincial institute practice inspections. The Office underwent three provincial practice inspections of its financial audit practice in 2013. None of the inspections identified significant implementation issues with the system of quality control.

51. We reviewed observations from provincial institute practice inspections for the attest product line to determine if the system of quality control could be improved and if practitioners could be assisted in avoiding the issues in other firms. We did not make any significant observations.

52. We reviewed responses to previous recommendations from Office monitoring and provincial institute practice inspections, including identifying opportunities to improve the attest audit practice.

Operational effectiveness of the system of quality control at the Office level

53. We found that overall the system of quality control (SoQC) is operating effectively at the Office level. We did note one area for improvement: At the Office level, no formal training and development are provided for internal specialists to maintain their skills and related designations.

54. A system of quality control consists of policies designed to achieve the objectives of the Canadian Standard on Quality Control (CSQC 1), and the procedures needed to implement those policies. Monitoring implementation is important for determining if there are any deficiencies and for recommending appropriate remedial action. Promoting an internal culture of quality so that the Office of the Auditor General (OAG) and its staff understand and comply with relevant requirements helps operational effectiveness at the Office level.

The Office promotes an internal culture of quality

55. The Office promotes an internal culture of quality through clear, consistent, and frequent messages, and rewards high quality work.

56. This finding is important because the culture or tone set by senior management is critical in creating an environment for quality work. Promoting an internal culture of quality establishes this as a core value that every employee is expected to have. Frequent communications to staff reinforce the commitment to quality, as does assigning responsibilities for quality to senior members who have sufficient and appropriate experience and ability, and the necessary authority to assume those responsibilities.

57. We made no recommendation in this area.

58. Area reviewed: Senior management actions and messages. The Office’s vision and values are clearly stated and communicated, as well as the OAG Code of Values, Ethics and Professional Conduct. Awards programs are in place to recognize staff members who promote the values of the Office, including product management and quality. The Office’s orientation training program includes a session to provide participants with a better understanding of the Office’s purpose, culture, and role in government. The course is mandatory for all new hires, ensuring that the culture of quality is made clear to all staff. Sharing the results of practice review activities with staff, including recommendations, contributes to the promotion of a culture of quality and continuous improvement. In 2013, these results were reviewed with the product leaders and the Assistant Auditor General of the Professional Practices Group, and presented by the Chief Audit Executive at meetings of the principals and directors of attest audits and performance audits.

59. The Office appraisal, promotion, and compensation processes require demonstrating that Office quality standards are met, and knowing and applying the system of quality control. Training sessions for independence are also included in the Office’s orientation training.

60. Area reviewed: Senior management responsibilities for quality. The roles and responsibilities for the elements of the SoQC are clearly assigned to senior management, who have the appropriate authority to fulfill their related duties.

61. The Auditor General assumes ultimate responsibility for the Office’s system of quality control, with the Assistant Auditor General of the Professional Practices Group, appointed by the Auditor General, assigned operational responsibility. The Assistant Auditor General of the Professional Practices Group has an appropriate combination of education, professional qualifications, experience, and skills to fulfill the duties of this function. The level of the position provides the Assistant Auditor General with the necessary authority to fulfill his responsibilities.

62. Area reviewed: Sufficient resources to support the system of quality control. The Office has sufficient resources for developing, documenting, and supporting the system of quality control. This includes the resources and processes for monitoring new developments in professional standards and integrating changes identified in the monitoring of audit methodology in a way that ensures consistency and completeness. Specifically, the Professional Practices Group is the operational centre for the system of quality control and has resources from three product-line practice teams—the Annual Audit Practice Team, Performance Audit Practice Team, and Special Examination Practice Team—and the Audit Quality Team. The practice teams conduct the following activities:

• monitor for new developments in professional standards, laws, and regulations;
• coordinate a common look and feel across practice teams;
• provide quality assurance by the practice teams; and
• ensure the accuracy and integrity of published methodology, including TeamMate.

63. We reviewed actions and messages from all levels of management that emphasize the requirement to perform work that complies with professional standards and issue reports that were appropriate in the circumstances. This included reviewing messages in internal documents, training seminars, meetings, formal or informal dialogue, mission statements, briefing memoranda, and appraisal procedures that address performance evaluation, compensation, and promotion to demonstrate the Office’s overriding commitment to quality. We also assessed whether sufficient resources had been provided to develop, document, and support the Office’s quality control policies and procedures.

Office staff understand and comply with relevant ethical requirements

64. We found that all staff required to complete an Annual Confidential Declaration for 2013 did so.

65. We found that exception reports initiated in 2013 were assessed by the Internal Specialist, Values and Ethics, and appropriate safeguards were applied where necessary.

66. This finding is important because Office employees maintain public confidence in the impartiality and objectivity of the Office by avoiding and preventing situations that could give rise to a conflict of interest, or even the appearance of or a potential for a conflict of interest.

67. We made no recommendation in this area.

68. Area reviewed: Annual confidential declarations. To demonstrate their understanding of these fundamental principles and compliance with Office protocols, employees must read, understand, and adhere to the Office’s Code of Values, Ethics and Professional Conduct. Adhering to ethical requirements includes signing an Annual Conflict of Interest Report (“Annual Confidential Declaration”) and assurance engagement reports on independence before beginning work on any assurance engagement. If employees identify threats to compliance with ethical requirements or independence, they must complete an Exception Report to help resolve the threat.

69. Independence requirements are emailed annually to staff and the Office maintains an automated mandatory annual process that requires staff to declare their independence. The system sends the request to all users and tracks progress from the request initiation, to printing, to delivery to Human Resources, and ultimately to Records Management. Reports are generated that track the progress and completion rate. The system automatically sends reminders to staff who have not completed the declaration. For 2013, all staff members who were required to complete an annual declaration did so.

70. Area reviewed: Exception reports. Staff members are required to promptly notify the Office of any circumstances or relationships that create threats to independence. If the threat is considered to be significant, the employee is required to initiate an Exception Report, which identifies the threat, and documents its impact and the appropriate action required to eliminate the threat or reduce it to an acceptable level. The Internal Specialist, Values and Ethics, reviews the report objectively and provides an assessment of the proposed safeguard, which may include additional actions to reduce the threat to an acceptable level. These safeguards reflect the individual’s level of influence on an audit and may include the following:

• increased level of supervision of the individual on the audit,
• segregation of the individual from certain audit lines of enquiry, or
• removal of the individual from the audit.

71. All exception reports initiated in 2013 were assessed by the Internal Specialist, Values and Ethics, and appropriate safeguards were applied where required.

72. Area reviewed: Job rotation. The objectivity of the Office may be threatened or appear to be threatened if senior personnel and quality reviewers, where applicable, continue to work with the same entity for a prolonged time period. Staff rotation is normally accomplished automatically through promotion or staff turnover; however, the responsibilities of senior personnel with signing authority are less likely to change unless a policy requires rotation. The Office job rotation policy requires the Principal of Human Resources to annually identify those senior personnel requiring job rotation for consideration by the Executive Committee. Exceptions to the job rotation policy must be approved by the Executive Committee and granted only if appropriate safeguards exist. Rotations in the regions can present special challenges. As such, they may require more lead time and more consultations among senior management.

73. The 2012 monitoring report found instances of senior staff remaining on audit engagements for periods exceeding Office policy with no documented approval for these exceptions. These exceptions were addressed, and the Assistant Auditor General of Corporate Services was assigned responsibility for the rotation of senior personnel to support timely compliance with rotation requirements.

74. For 2013, there were two instances in which an Assistant Auditor General was assigned to an entity for more than seven years in a senior role. In both of these cases, the Executive Committee approved extensions of their terms and documented existing safeguards in place. This decreased risk to an acceptable level.

75. In one instance a Principal had been assigned to the same portfolio for eight years. It was noted that this Principal was from a regional office and due to retire shortly. There was no record of an approved extension or documentation of existing safeguards. However, this situation was considered to be low risk as plans were in place for another Principal in the regional office to assume these responsibilities.

76. Although professional standards and OAG policy require an annual analysis of job rotation requirements at the team and group levels, preparing a medium-term (two- to three-year) plan at the Office level would support timely job rotation and assist succession planning. Such a plan would include information on whose job responsibilities would need to change, when the rotation would take place, who would be the new senior staff member, the length of the stand-down period, and any other relevant information.

77. We reviewed the process for annual confidential declarations, the identification of threats to independence in exception reports, and job rotation analysis and actions.

The Office fulfills acceptance and continuance requirements

78. We found that the Office has processes in place to ensure that the principles of acceptance and continuance are adhered to and are applied to all of its assurance engagements.

79. This finding is important because acceptance and continuance procedures provide the Office with valuable information for performing risk assessments and carrying out reporting responsibilities.

80. We made no recommendation in this area.

81. As a legislative audit office, many assurance engagements are required by legislation, whereas others are conducted at the discretion of the Office. For discretionary audits, all requests for appointment by order- in-council and/or under the Financial Administration Act are referred to Legal Services to determine whether the Office has the authority to conduct the engagement. The Office decides which audits to perform based on its independence from the entity, so that there is no interference with the Office’s mandate.

82. At the engagement level, for both discretionary and statutory audits, engagement leaders perform and document acceptance procedures for all new engagements. For statutory audits, if it is decided that the Office needs to waive or decline a statutory appointment, the engagement leader prepares a briefing note and presents it to the Executive Committee for approval. Legal Services analyzes whether there is a professional, legal, or regulatory requirement to remain as auditor or whether the Office should report the withdrawal, cancellation, or postponement, and the justification, to others outside the Office.

83. During 2013, there were no acceptance and continuance actions required at the Office level.

84. We reviewed Executive Committee records of decisions, and conducted interviews with Legal Services to determine whether acceptance and continuance processes were followed at the Office level and whether familiarity threats were identified and resolved.

The Office ensures that it has staff with competencies, capabilities, and commitment to quality

85. The Office assesses the competencies and capabilities it requires at the team and group levels. In addition, it is currently developing a global staffing profile as well as a recruitment and promotion plan to assess future staffing needs.

86. The Office uses specialist skills in the course of its work by means of advisory committees that comprise external experts and internal specialists. The Office has identified and annually reviews the functional areas for internal specialists. While we note that identifying internal specialist areas is done well, we found that no formal training and development are provided for internal specialists to maintain their skills and any related designations.

87. The Office has a solid process in place for assigning professional personnel to audit engagements. Principals work with the Audit Resource and Career Planning Management (ARCPM) team and the Office staff scheduler, Retain, to assess and document the assignment of appropriate personnel with the necessary competencies for the assurance engagements under their responsibility.

88. The Office has invested in training and professional development for its product and people management. It has assessed training and professional development needs, developed a professional development business plan to address gaps and opportunities and add to its value proposition, created budgets, and dedicated resources to training and professional development.

89. In accordance with professional standards, the Office has an annual performance management system in place that requires managing products to a high level of quality.

90. These findings are important because personnel are the Office’s most important asset and its biggest cost. The Office can only issue reports that are appropriate in the circumstances if it has staff with the competencies, capabilities, and commitment to quality to perform assurance engagements according to professional standards and ethical requirements.

91. Recommendation. The Office should consider providing formal training and professional development for internal specialists so they can maintain their skills and any related designations.

Management’s response. For 2015–16, the Assistant Auditor General of the Professional Practices Group will have a process in place to identify internal specialist training needs and will provide support to internal specialists to meet those needs.

92. Area reviewed: Assessment of staffing needs. The Office has an annual process in place to determine professional staffing needs at the Financial Audit Trainee (APS) and Performance Audit Trainee (APD) levels. Each year, the ARCPM team reviews the number of APSs and APDs the Office has, analyzes potential departures, meets with assistant auditors general to discuss group requirements for students, and prepares an analysis of APS and APD needs for approval by the Executive Committee.

93. The goal of the APS and APD programs is to recruit, train, and retain employees with the general competencies required to become good financial and performance auditors. The APS program recruits university students from accounting programs to fill the permanent needs in the Office’s financial auditing operations. The APD program recruits students who have master’s degrees from a Canadian university. The Office has determined that these requirements provide the required general competencies and capabilities.

94. During the course of these two-year programs, trainees must demonstrate that they meet additional specific Office competencies, which include delivering products according to the system of quality control.

95. In addition to assessing students’ needs, the ARCPM team is currently looking at group profiles at the Audit Professional (AP) level (AP1–3) and conducting an analysis of AP-level staff to help identify the competencies and capabilities the Office requires at this level.

96. The Office is currently reviewing its Office level governance and senior level functions, so it can redefine these roles and responsibilities to eliminate duplicating functions and increase efficiencies. As part of this review, the ARCPM team is taking part in an initiative called the “realistic profile for audits.” This initiative will establish the Office’s current staff profile on a global level, the profile it requires going forward, and an appropriate recruitment and promotion strategy ensuring that the Office continues to have the competencies and capabilities it requires. This initiative will be reviewed annually for any significant changes.

97. The Office does not currently have a recruitment and retention strategy. The last strategy was developed for the period 2006 to 2009. As part of the Executive Transition Initiative, the Office is planning to develop a new recruitment and retention strategy.

98. Area reviewed: Use of specialist skills. The Office uses specialist skills from both external and internal resources in its work. Audit teams may use audit advisory committees in the course of their audits, which provide advice on the scope and significance of issues, lines of enquiry, and audit approach. Both external and internal committee members are selected on the basis of their skills, expertise, relevant knowledge, and experience. External advisers are generally recognized as leaders in their fields of expertise. Typically, advisory committees include two or three external advisers, who bring different perspectives to the subject matter, and appropriate internal specialists, including those with sign-off responsibilities. Advisory committees usually meet twice: once in the planning phase and once late in the examination phase to discuss an early draft of the report.

99. Two categories of internal resource persons from the Office are also available to audit teams to provide consultation and expert advice: product leaders and internal specialists. Product Leader responsibilities include keeping up to date with standards, policies, and developments related to the audit product lines. The OAG keeps a single list of internal specialists for all product lines, which is updated annually by the Assistant Auditor General of the Professional Practices Group.

100. Further to the 2012 monitoring report finding on the lack of clarity in the selection and appointment of internal specialists for all product lines, the Office has since developed principles and a process to annually update the list of internal specialists, as well as criteria for selecting functional areas, approved by the Executive Committee in 2013. To appoint specific individuals as internal specialists, the current process entails the Assistant Auditor General of the Professional Practices Group negotiating the names of potential internal specialists with individual assistant auditors general. This includes considering such things as related education and experience. However, there are no formal criteria to ensure that the most qualified person is appointed as an internal specialist, such as the requirement for a related designation or education, or training, development, and maintenance of specific competencies, capabilities, or designations. In addition, no formal training and development are provided to internal specialists to maintain their skills and/or related designations.

101. Audit teams who require additional specific expertise may hire consultants with the required expertise on an audit-by-audit basis.

102. Area reviewed: Assignment of professional personnel. The Office has a process in place for the assignment of professional personnel. According to the Chartered Professional Accountants of Canada’s Quality Assurance Manual, working with Human Resources and a staff scheduler, the engagement leader (or Principal) is responsible for assigning suitably qualified personnel to each engagement who collectively have the appropriate competence and capabilities to perform the work required.

103. OAG policy requires that before the planning/survey phase of an assurance engagement is completed, the Principal must assess the engagement team to be satisfied that the team, the specialists, and any auditor’s experts who are not part of the audit team, collectively have the appropriate competence and capabilities, and assign roles and responsibilities. This is documented using the Engagement Team Competency and Resource Assessment form, which captures the assessment and strategies needed to ensure that the Principal is assigning appropriate personnel with the necessary competencies and that there are adequate resources and time available for the assurance engagement under their responsibility.

104. While principals are responsible for the resource planning of audit teams, the ARCPM team supports them in the following ways:

• managing and monitoring the corporate resource planning tool, Retain, a resource database that contains information by auditor and by audit product;
• conducting resource needs analyses to support product leaders, senior Office committees, and audit teams;
• assisting staff transfer and sharing among groups; and
• identifying possible solutions for critical resource requirements.

105. The ARCPM team may be consulted

• by resource managers or principals when an audit team is looking for a resource or for an assignment or transfer for their auditors, and
• by auditors who are looking for an assignment or a transfer.

106. Area reviewed: Training and professional development. In its learning vision the Office states that it is committed to building and promoting a learning culture that adds value to its work for Parliament and the Canadian people and supports the lifelong learning of Office employees.

107. For the past couple of years, the OAG has invested heavily in renewing the audit training curriculum, methodology, and tools. The Office has assessed training and professional development needs by competency and skill level through a training needs analysis, developed a professional development business plan to address gaps and opportunities and add to its value proposition, created training and professional development budgets, and dedicated resources to training and professional development.

108. As part of this, the Office has developed the Leadership Program, which focuses on people management, to meet the professional development needs of the Office’s leaders and assist continuous learning in this area. Leadership is a key component of the system of quality control. The program follows a multi-dimensional approach that includes formal training, interactive knowledge-sharing events, practical tools and resources, and coaching, as well as support services to resolve issues.

109. The Office has developed a new vision for learning focused on continuous learning beyond the classroom. One of the key elements of this vision is emphasizing on-the-job coaching and offering staff relevant on-the-job experiences. The role of the Office’s Professional Development (PD) team is to provide staff with the best formal training possible and to offer support to managers by helping them provide feedback and coaching as staff experiment with newly acquired skills. In the fall of 2013, the Executive Committee endorsed the new vision to reinforce the OAG’s culture of continuous learning.

110. PD does an annual scan of the training and professional development environment by consulting with product leaders, reviewing training evaluations, and consulting with accounting firms (that is, PricewaterhouseCoopers, Deloitte) on what is happening in the industry. PD updates training and professional development initiatives based on the results.

111. Area reviewed: Performance management. The Office has in place a process for performance management, which includes goal setting, competencies, ongoing feedback, assessment processes, corrective actions, training and development, and career planning. Mandatory annual performance appraisals are conducted on all active staff.

112. The performance appraisal process includes assessing values and competencies, and the requirement to manage products to a high level of quality according to standards. For any quality-related issues identified by management, Human Resources assists in remedying the situation through coaching and mentoring, more frequent follow-ups, training, and other appropriate corrective actions.

113. During 2013, performance appraisals were completed for all active staff. One hundred percent of staff who received performance pay had a performance appraisal that supported this.

114. We reviewed documentation and conducted interviews on the following: recruitment (assessment of staffing needs); use of specialist skills; assigning of professional personnel (reviewed the mandate of the ARCPM team); staff training and professional development; and performance management.

The Office encourages the reporting of complaints and allegations regarding the conduct of its work

115. The Office encourages the reporting of complaints and allegations regarding the conduct of its work.

116. This finding is important because it demonstrates that the Office deals appropriately with complaints and allegations if the work it performs fails to comply with professional standards and applicable legal and regulatory requirements, and allegations of non-compliance with the system of quality control are made.

117. We made no recommendation in this area.

118. The Office’s policies, OAG Audit 1012 Audit Quality, and OAG Audit 1091 Complaints and Allegations, meet the requirements of CSQC 1 for addressing complaints and allegations. The Office communicates these policies to all employees via the INTRAnet.

119. External complaints are received via a public inbox managed by the Communications team, and internal complaints are received by the Auditor General directly. Complaints are then tracked in a database, where they are addressed and investigated by the Auditor General and whomever the Auditor General appoints as investigator. Targeted response time on all issues is 90 days.

120. The Executive Committee receives a quarterly status report on all closed and outstanding complaints and allegations. For 2013, the Office received no complaints or allegations either internally or externally about the conduct of its audits in regard to the system of quality control.

121. The 2012 monitoring report observed that the current process for directing internal allegations or complaints about non-compliance to the Auditor General could call into question the perception of his ability to exercise his role with an appropriate level of independence, given his involvement in the outcome of audits. The Executive Committee approved adding the Chair of the OAG Audit Committee to receive complaints or allegations.

122. We reviewed documentation and conducted interviews with Legal Services about complaints and allegations received during 2013 on work performed by the Office about failing to comply with professional standards and/or the Office’s system of quality control.

Operational effectiveness of the system of quality control at the engagement level

123. We found that overall the system of quality control is operating effectively at the engagement level. In the practice reviews of all engagements, the auditor’s reports were supported and appropriate.

124. A system of quality control consists of policies designed to achieve the objectives of the Canadian Standard on Quality Control (CSQC 1), as well as the procedures necessary to implement those policies. Monitoring implementation at the engagement level is important for determining if there are any deficiencies and recommending appropriate remedial action. Operational effectiveness at the engagement level entails examining whether the Office of the Auditor General (OAG) is carrying out its responsibilities by assessing whether its assurance engagements comply with Office policies and standards.

Engagements comply with professional standards

125. We found that overall the Office’s engagements comply with professional standards. We noted one area for improvement: in some instances practitioner review was late and lacked documentation about the nature and extent of the review. However, no systemic issue was identified that would support a recommendation to the Office

126. This finding is important because the Office ensures the quality of its audit work by following professional auditing standards. Assessing compliance with professional standards helps the Office determine whether it is carrying out its responsibilities. It also contributes to continuous improvement by creating an opportunity for audit teams and the Office to learn from experience.

127. We made no recommendation in this area.

128. Area reviewed: Supervision and review. Ensuring that the Office’s assurance engagements are completed to the highest quality requires team members to be adequately supervised, and audit work and documentation to be reviewed. Supervision is important to ensure that engagement teams are organized and the quality of the work produced during the engagement is monitored for quality. Review is important to ensure that work has been performed according to professional standards, that it supports the conclusions reached, that the evidence obtained is sufficient and appropriate, and that the objectives of the engagement procedures have been achieved.

129. Supervision: In the practice reviews conducted for 2013 of seven direct reporting engagements and seven attest audits, all practitioners exercised appropriate levels of supervision.

130. Review: All seven direct reporting engagements that had practice reviews demonstrated that the review of audit work was performed on the basis of more experienced team members reviewing the work of less experienced team members. In five of the seven files, the engagement leader ensured that detailed file reviews were conducted on a timely basis at appropriate stages during the engagement and before the date of the assurance engagement report. One case required more documentation to clearly demonstrate the level and extent of review. In two files, reviews were not timely, not all high risk areas were reviewed, and the review was not completed before the date of the engagement report.

131. All seven attest audits that had practice reviews demonstrated that the review of audit work was performed by more experienced team members reviewing the work of less experienced team members. In two files, senior management had not performed a timely review at the planning phase of the audit, and in one file, the high risk sections of the file lacked evidence of the practitioner review. In all cases, the review was completed before the date of the engagement report.

132. Area reviewed: Engagement quality control review. Quality reviews objectively evaluate the significant judgments made by the engagement team and the conclusions reached in formulating the assurance engagement report. Quality reviewers are assigned to each annual audit of entities that issue or have securities outstanding in public markets. They are also assigned to other assurance engagements based on the level of risk associated with the assurance engagement. Quality reviewers have the technical qualifications to perform the role, as well as sufficient and appropriate experience and authority.

133. The Office has a process in place to select and appoint quality reviewers. The selection of quality reviewers is based on the level of risk associated with the engagement to comply with professional standards. The Professional Practices Group receives risk assessment input from each audit team and prepares a risk assessment for all engagements using selection criteria outlined in the methodology. It is normally recommended that engagements assessed as high risk be selected for a quality review. Low to medium risk audits are not normally assigned a quality reviewer. A quality review is required for financial audits of organizations that issue or have securities outstanding in public markets.

134. For engagements selected for a quality review, a quality reviewer is then appointed based on specific criteria. The Professional Practices Group consults the assistant auditors general annually to review the list of audit engagements for their group and the availability of their senior personnel for quality review assignments. Once all risk assessments are compiled by the Professional Practices Group, the product leaders are consulted on the recommended quality reviewer selection and appointment.

135. Three of the seven direct reporting engagement files reviewed had been assigned a quality reviewer. In one file, the audit report was dated before the quality reviewer’s final review. In another file, the quality reviewer was not engaged by the practitioner in a manner that allowed timely review during various stages of the audit. In these instances, the shortcoming was communicated to the practitioners.

136. One of the seven engagements of the attest audits reviewed was assigned a quality reviewer. In this case, the quality reviewer conducted the review in a timely manner at appropriate stages of the audit, and the review was completed before the date of the report.

137. Area reviewed: Differences of opinion. During the course of an assurance engagement, differences of opinion may arise within the team, with those consulted about the assurance engagement, and between the engagement leader and quality reviewer. Audit team members have the right to form their own conclusions on significant matters in the areas of the assurance engagement for which they are responsible, and ensure that their views receive adequate consideration. An assurance engagement report should not be dated until team members have resolved all differences of opinion.

138. No differences of opinion were noted for any of the direct reporting engagement or attest audit files reviewed.

139. Area reviewed: Engagement documentation. This component addresses the confidentiality, safe custody, integrity, accessibility, retrievability, and retention of engagement documentation and the completion of the final assembly of engagement files on time.

140. In the practices reviewed, all direct reporting engagement and attest audit files complied with engagement documentation requirements: complete and final audit file documentation for performance audits was assembled and finalized within 60 days of the date of tabling and within 60 calendar days after the date of the assurance engagement report (financial audit and special examinations); engagement documentation was retained and stored in TeamMate; and protected and classified information was labelled, copied, stored, transmitted, and disposed of according to OAG security requirements.

141. We conducted a detailed review of 2013–14 practice review files to determine whether they complied with the system of quality control at the engagement level. Specifically, we reviewed seven attest audit files and seven direct reporting engagement files in the following areas:

• supervision and review,
• engagement quality control review,
• differences of opinion, and
• engagement documentation.

Engagement team competencies are assessed and documented

142. We found that engagement team competencies are being assessed and documented.

143. This finding is important because it is critical to ensure that the engagement team has the appropriate competence and capabilities to conduct the engagement. Engagement leaders need to document their assessments and strategies to ensure that they are assigning appropriate personnel who have the necessary competencies and that there are adequate resources and time available for the assurance engagement under their responsibility.

144. We made no recommendation in this area.

145. Area reviewed: Engagement team competencies. In all cases of the seven direct reporting engagements reviewed, before the planning/survey phase was completed, the engagement leader assessed the team to be satisfied that members, specialists, and others collectively had the appropriate competence and capabilities, and documented the assessment. However, in one instance several team members were not included in the assessment of competencies.

146. In all cases of the seven attest audits reviewed, before the planning/survey phase was completed, the engagement leader assessed the team to be satisfied that members, specialists, and others collectively had the appropriate competence and capabilities, and documented the assessment.

147. We conducted a detailed review of 2013–14 practice review files (seven direct reporting engagements and seven attest audits) to determine the nature of the assessment of team competencies and whether it was sufficient.

Adequate consultation is undertaken and documented by engagement teams

148. We found that overall the engagement teams provided adequate consultation that was documented.

149. This finding is important because consultation is critical during the course of an assurance engagement to help promote quality and improve professional judgment, as well as reduce the risk of error. A key aspect of an assurance engagement is the formal and informal consultation that takes place within audit teams, between audit teams, and between audit teams and the Office’s internal specialists or others.

150. We made no recommendation in this area.

151. Area reviewed: Consultation. In all cases of the direct reporting engagements reviewed, audit teams consulted with internal and external specialists and senior Office staff when dealing with difficult or contentious matters or other matters requiring specialized knowledge or experience. In five cases, the nature and scope of, and the conclusions resulting from, consultations were documented and agreed to by both the individual seeking consultation and the party consulted before the date of the assurance report. In one instance, consultations were documented but not agreed to. In another instance, the consultations were not documented. In six of the seven files, the conclusions resulting from consultations were implemented. In one instance, due to the lack of documentation, the practice reviewers could not determine if the conclusions were implemented.

152. In the practice reviews of the seven attest audits, all files showed

• the audit team consulted with internal and external specialists and senior Office staff when dealing with difficult or contentious matters or other matters requiring specialized knowledge or experience;
• the nature and scope of, and the conclusions resulting from, consultations were documented and agreed to by both the individual seeking consultation and the party consulted before the date of the assurance report; and
• the conclusions resulting from consultations were implemented.

153. We conducted a detailed review of 2013–14 practice review files to determine the nature of the consultations undertaken and whether they were sufficient.

Conclusion

154. Based on the work performed in the 2013 monitoring exercise, we conclude that there is reasonable assurance that the policies and procedures for the Office of the Auditor General’s system of quality control are relevant, adequate, and operating effectively at both the Office and engagement levels, so that reports issued by the Office are appropriate in the circumstances. We did not identify any serious deficiencies.

155. We did, however, note one area for improvement related to the operational effectiveness of the system of quality control, although this weakness did not affect the adequacy or relevance of the system or the appropriateness of the audit reports issued. Specifically:

• Formal training and professional development are not provided for internal specialists to maintain their skills and any related designations.