Monitoring Report on the System of Quality Control—2019–20 Fiscal Year

Report date: December 2021

1. Executive Summary

This report outlines the process for monitoring the Office of the Auditor General of Canada’s (OAG’s) system of quality control and presents the 2019–20 results of the evaluation conducted by the OAG’s Compliance and Monitoring team. The monitoring process is designed to provide the OAG with reasonable assurance that the policies and procedures relating to the system of quality control are relevant, adequate, and operating effectively.

Overall, we found that the design of the OAG’s system of quality control was relevant and adequate and met the requirements of the Canadian Standard on Quality Control 1 (CSQC 1), issued by the Auditing and Assurance Standards Board. We also found that the OAG’s system of quality control operated effectively at the engagement level. In the practice reviews of engagements, the auditor’s reports were supported and appropriate. We did not identify any significant deficiencies.

However, there is an opportunity for the OAG to improve how its human resources policies and procedures provide it with reasonable assurance that it is managing its human resources risk effectively and that it has sufficient and appropriate information at the OAG level to make strategic human resources decisions. Such assurance would demonstrate that the OAG would continue to have sufficient personnel with the competence and capacities to address the challenges of the future. We noted a number of areas where the design and implementation of the human resources element in the OAG’s system of quality control could be improved.

In the last few years, the OAG has reviewed and updated many of its human resources policies and procedures. However, the OAG has not updated the mapping of its work to the CSQC 1 to ensure that it continues to address the requirements for the standard’s human resources element.

Nevertheless, the issues we identified did not affect the appropriateness of the reports issued by engagement leaders in the 2019–20 fiscal year. The Practice Review and Internal Audit (PRIA) team completed its inspections of financial audits and direct engagements for the 2019–20 fiscal year, and those practice reviews did not identify any significant findings at the engagement level.

2. Objective

The OAG must comply with the CSQC 1. This standard requires that a quality control system be established and maintained for all assurance engagements. It also requires the OAG to monitor compliance with quality control policies and procedures and report on its evaluation annually. We communicate the monitoring results to the Auditor General and management and recommend appropriate action where necessary.

The system of quality control is to provide reasonable assurance that the OAG and its personnel comply with professional standards and applicable legal and regulatory requirements, and that the audit reports that the OAG issues are appropriate in the circumstances.

3. Scope

The scope of the monitoring process includes assessing whether the policies and procedures relating to the OAG’s system of quality control are relevant, adequate, and operating effectively. We evaluated the OAG’s policies and procedures relating to the 6 elements of the system of quality control as identified in the CSQC 1 (see the “Elements and requirements of a system of quality control” section below).

The OAG’s monitoring process includes 2 distinct parts:

• Annual monitoring—An annual evaluation of the OAG’s compliance with its quality control policies and procedures that is conducted and reported on by the Compliance and Monitoring team. This work does not include inspecting specific assurance engagement files. 
• Practice review—A cyclical inspection of completed assurance engagement files, including financial statement audits and direct engagements (performance audits and special examinations), that is conducted and reported on separately by the PRIA team. The PRIA team selects a risk-based sample, ensuring that each engagement leader is selected at least once in every 4 years.

Period of review

The monitoring process covered the period from 1 April 2019 to 31 March 2020.

Elements and requirements of a system of quality control

The following chart identifies the 6 elements of a system of quality control and the requirements for policies and procedures within each element, as outlined in the CSQC 1.

Monitoring procedures performed

To evaluate the design of the OAG’s system of quality control, we

• conducted interviews with the management employees who were responsible for areas under the 6 elements
• reviewed previous monitoring process results, the OAG’s audit methodology and system policies, internal risk assessments, and relevant external documentation that may have affected the system’s design 
• interviewed selected staff at all levels, including regional office staff, using a questionnaire customized from one included in the Chartered Professional Accountants of Canada’s Quality Assurance Manual, to ascertain staff’s knowledge of, and compliance with, the OAG’s quality control policies

To evaluate the system of quality control’s operating effectiveness, we tested key controls, including any relevant changes to the system’s design. 

Rating categories

As required by the CSQC 1, we rated any deficiencies found as follows:

• Systemic, repetitive, or significant deficiency—These matters require attention and prompt corrective action to comply with professional standards and legal and regulatory requirements and ensure that the report issued are appropriate in the circumstances.
• Needs improvement—These matters are less significant; improvements are needed but do not indicate that the OAG’s system of quality control is insufficient to provide it with reasonable assurance that the system of quality control complies with professional standards and applicable legal and regulatory requirements, and that the reports issued are appropriate in the circumstances.

4. Findings

Needs improvement

Overall, we found that the design of the OAG’s system of quality control was relevant and adequate and met the requirements of the CSQC 1, issued by the Auditing and Assurance Standards Board. We also found that the OAG’s system of quality control operated effectively at the engagement level. We did not identify any significant deficiencies.

However, we did note that some areas related to the design and implementation of the human resources element of the OAG’s system of quality control could be improved. Although the issues we identified are systemic and require corrective action, they did not affect the appropriateness of the reports issued by engagement leaders in the 2019–20 fiscal year.

Human resources management

The OAG could not demonstrate how its human resources policies and procedures provided it with reasonable assurance that it was managing its human resources risk effectively and that it had sufficient and appropriate information at the OAG level to make strategic human resources decisions. Such assurance would demonstrate that the OAG would continue to have sufficient personnel with the competence and capacities to address the challenges of the future.

More specifically, there was a lack of evidence at the OAG level that

• performance objectives were received by all professional staff and that performance discussions were occurring for all.
• staffing decisions were made according to up-to-date knowledge of the OAG’s current professional staff skills and capacities, the current gaps in competence and capability at the engagement level, and planned future staff needs
• succession planning decisions were made with medium- and long-term perspectives
• mandatory training and designation requirements for professional staff were effectively monitored
• rationales for staff departures were assessed on a timely basis to monitor retention risks

In the last few years, the OAG has reviewed and updated many of its human resources policies and procedures. However, it has not updated the mapping of this work to the CSQC 1 to ensure that it continues to address the standard’s human resources element. Also, the OAG has not identified the controls and related control owners that are needed to monitor the effectiveness of its human resources policies and processes and its compliance with the related CSQC 1 standard’s objective.

Nevertheless, the PRIA team completed its inspections of financial audits and direct engagements for the 2019–20 fiscal year and did not identify any significant findings at the engagement level.

As a result of our review of documentation, interviews with managers involved in human resources management, and meetings with professional staff at all levels, we noted some issues with the quality of human resources information at the OAG level in the following human resources areas.

Performance management

In 2019, the OAG introduced a new performance management approach with a focus on timely feedback and fueling performance in the future. The new approach includes setting objectives, ongoing assessment of competencies and feedback, and providing improvement plans for employees identified as needing support. However, we found that there was no process to consolidate the information obtained at the OAG level, which informs a corporate view of gaps and ensures that the discussions are taking place.

As a result, no information was collected at the OAG level to ascertain whether all professional staff had performance objectives aligned with the corporate values and objectives and had received timely feedback. Setting performance objectives and providing timely feedback enable employee performance and continuous development and assist with the execution of high-quality audits.

The lack of key performance management information at the OAG level can also affect the OAG’s ability to inform performance pay, professional development needs, and promotions.

Identifying staffing needs

The OAG determines staffing needs at the audit professional and management levels by considering resources’ current and anticipated availability, competencies, and capabilities, as well as realistic profiles. The OAG has established staff profiles for each audit practice to determine the number of staff needed at each level to ensure sufficient and competent staff on audit engagements. The profiles were based on key assumptions and were used as a benchmark to forecast resource needs, and in hiring and promotion decisions.

However, we found that the information on the resources’ current competencies and capacities and most team profiles had not been updated in recent years. Outdated profiles increase the risk of not having enough staff with the right skill sets to deliver on our mandate and the risk of making untimely and ineffective staffing decisions.

Succession planning

An international peer review report covering the period of January 2017 to December 2018 on the OAG’s system of quality control included a recommendation related to “identifying, developing, and monitoring a pool of appropriately capacitated professionals and practice leaders.” The OAG is committed, in part, to reviewing and updating the medium- and long-term skill set requirements (know-how, competencies, and attributes) to enable the audit practices to address the challenges of the future. The work in this area is ongoing.

Professional development

We found that the management information at the OAG level was not sufficient and appropriate to monitor compliance with mandatory training and designation requirements for professional staff on a timely basis.

The OAG has an information system (a training dashboard) that is used by supervisors and managers to discuss individual training needs and requirements with individual staff. However, the information collected in the system is granular and is not necessarily reliable at the OAG level to oversee compliance with mandatory training requirements.

In addition, the OAG does not monitor an employee’s adherence to the employee’s professional order or institute requirement, as suggested in the Chartered Professional Accountants of Canada’s work program for ongoing policy monitoring. It is important for the OAG to ascertain on an ongoing basis that professional staff are up to date on their provincial professional development requirements when conducting audits and signing audit opinions.

The OAG’s Policy on Learning and Professional Development, implemented in February 2015, is due to be renewed. Such a review would be an opportunity for the OAG to better define what mandatory training for professional staff is and to clarify what role professional development plays in ensuring that professional staff have mandatory training and designation requirements.

Retention

The Audit Resource Planning and Career Management team interviews all departing staff to identify the main reasons they are leaving the OAG. This process is key to helping understand people management risks (for example, the inability to retain and engage staff) and identify any issues that may be related to staff competencies and capabilities, including ethical requirements. Although the team conducted exit interviews in the 2019–20 fiscal year, it did not report to the practices on the issues it identified until June 2021.

5. Recommendations

The OAG is required to establish and maintain policies and procedures designed to provide it with reasonable assurance that it has sufficient personnel with the appropriate competence, capabilities, and commitment to ethical principles.

To this end, the OAG should

• always keep the mapping of its human resources policies and procedures to the CSQC 1 up to date to ensure that the quality control system addresses the requirements for the human resources element
• clarify the information the OAG needs to collect, use, and report on from its system of quality control at the OAG level to give it the assurance that its human resources systems and practices are operating effectively on an ongoing basis
• review the form and content of documentation detailing the operations of the human resources element to ensure that the OAG demonstrates that it has sufficient personnel with competence, capabilities, and commitment to ethical requirements now and in the future, that human resources risks are effectively managed, and that the expectations of the current quality control standard and the upcoming Canadian Standard on Quality Management are met

Management’s response. Agreed. The Human Resources team will work in consultation both with the managers responsible for implementing the different elements of the revised corporate planning process and with the methodology team responsible for implementing the new CPA Canada standards for quality management by the end of 2022. This work will provide senior management with assurance that human resource policies and procedures are aligned with the new standards’ requirements. The Human Resources team will work with the Finance team to collect sufficient and appropriate information about human resource risks and staffing sufficiency, competence, and capabilities. This information will be reported regularly to senior management.

6. Follow‑up on the 2018–19 Recommendations

In November 2020, the OAG reported on its monitoring of the system of quality control for the 2018–19 fiscal year. In that report, untimely reporting on both the monitoring of the system of quality control and on practice review results was identified as an area for improvement. The OAG began its monitoring process for the 2019–20 fiscal year immediately after the 2018–19 report was finalized. At the same time, the OAG began its monitoring process for the 2020–21 fiscal year to address the untimely reporting. The 2020–21 report is expected to be finalized in spring 2022. The PRIA team also improved the timing of its reporting by using “on‑time” reporting, beginning in the 2020–21 fiscal year.

7. New Quality Management Standards

New quality management standards have been issued internationally and were issued in Canada in May 2021. The OAG has started a project to design a risk assessment process that complies with this standard and other analysis of the implications of these new standards.

8. Compliance and Monitoring Team

This monitoring process was completed by a multidisciplinary team from across the OAG that was led by Annie Leclerc, Principal. She had the overall responsibility for the monitoring process, including conducting the process in accordance with professional standards, applicable legal and regulatory requirements, and the OAG’s policies and system of quality management.