Report on a Review of the Direct Engagement Audit Practice—Direct Engagement Audits Completed in the 2015–16 Fiscal Year

Report on a Review of the Direct Engagement Audit Practice—Direct Engagement Audits Completed in the 2015–16 Fiscal Year

Introduction

1. The Office of the Auditor General of Canada (the Office or OAG) conducts independent audits and studies that provide objective information, advice, and assurance to Parliament, territorial legislatures, boards of Crown corporations, government, and Canadians. The Office carries out three main types of legislative audits: financial audits, performance audits, and special examinations. Performance audits and special examinations are referred to as direct engagements.

2. A performance audit is an independent, objective, and systematic assessment of how well government is managing its activities, responsibilities, and resources. Performance audits contribute to a public service that is effective and a government that is accountable to Parliament and Canadians. Performance audits are planned, performed, and reported in accordance with professional auditing standards and Office policies.

3. Special examinations are a form of performance audit that is conducted within Crown corporations. The Office audits most, but not all, Crown corporations. The scope of special examinations is set out in the Financial Administration Act. A special examination considers whether a Crown corporation’s systems and practices provide reasonable assurance that its assets are safeguarded, its resources are managed economically and efficiently, and its operations are carried out effectively.

4. The mission of the Practice Review and Internal Audit team is to enhance and protect organizational value by providing risk-based and objective assurance, advice, and insight. The team helps the Office accomplish its objectives by offering management recommendations based on the application of a systematic, disciplined approach to evaluating and approving the design and effectiveness of risk management, control, and governance processes.

5. The team helps the Office meet its obligations under the Chartered Professional Accountants of Canada Canadian Standard of Quality Control 1 by conducting inspections to determine the extent to which engagement leaders are complying with professional standards, Office policies, and applicable legislative and regulatory requirements when conducting their audits, and to ensure that audit reports are supported and appropriate.

6. The team also performs its work in accordance with the Office’s most recent Practice Review and Internal Audit Plan, as recommended by the Audit Committee and approved by the Auditor General. The Plan is based on systematic, cyclical monitoring of the work of all engagement leaders in the Office.

7. To ensure that audits meet the standards of Chartered Professional Accountants of Canada, the Office establishes policies and procedures for its work. These are outlined in the Office’s direct engagement audit manual, in its System of Quality Control, and in various other audit tools that guide auditors through the required steps. The four assistant auditors general responsible for direct engagement audits provide leadership and oversight of the direct engagement audit practice in the Office and contribute to the quality of individual audits.

8. This report summarizes the key observations related to the practice reviews of selected direct engagement audits completed in the 2015–16 fiscal year.

Overview

Objective

9. The objective of practice review is to provide the Auditor General with assurance that

• direct engagement audits comply with professional standards, Office policies, and applicable legislative and regulatory requirements; and
• audit reports are supported and appropriate.

Scope and methodology

10. The Practice Review and Internal Audit team conducted practice reviews of six direct engagement audits. Our methodology requires that we review a selection of completed audits on a cyclical basis, including at least one audit for each engagement leader over a four-year monitoring cycle. We used a random sampling approach to select the engagement leaders and their related files.

11. Our reviews included an examination of electronic (TeamMate) files as well as paper files, if applicable. We reviewed documentation related to the planning, examination, and reporting of the audits. We also interviewed quality reviewers, selected audit team members, and other internal specialists, as appropriate.

12. We reviewed all files selected in terms of the System of Quality Control (Appendix A). We focused our work on the selected elements and process controls that we considered to be key or high risk (Appendix B) in the selected audits.

Rating

13. For each audit reviewed, we rated each selected System of Quality Control element and process control as one of the following:

• Compliant. Performance is satisfactory, with minor improvement possible; the audit file is in compliance with General Assurance and Auditing Standards (GAAS) and Office policies in all significant respects.
• Compliant and improvement needed. Improvements are necessary in some areas to fully comply with GAAS and Office policies.
• Non-compliant. Major deficiencies exist; the audit does not comply with GAAS and/or Office policies.

14. After completing each practice review, we concluded whether the audit report was supported and appropriate. We also concluded whether the audit file was compliant overall with GAAS and with Office policies.

Results of the Reviews

Appropriateness of the audit reports

15. Overall, we found that the audit reports were supported and appropriate in all six files reviewed.

Compliance with the System of Quality Control elements and process controls

16. In general, the overall level of compliance with the System of Quality Control elements was good. Two files complied in all material respects with the Office’s direct engagement audit policies and General Assurance and Auditing Standards. The remaining four files were compliant and improvement needed. Please refer to the Observations section for details.

17. It is important to note that our overall conclusion on a specific file is based on the review of all elements of the System of Quality Control. Consequently, it is possible to be non-compliant with one element of the System of Quality Control even though the overall conclusion is compliant with improvement needed.

Observations

Ethics and independence

18. In four of six files, we found that Independence Confirmation forms were prepared and dated based on the date that the auditor joined the team, as opposed to the period covered by the audit. This puts the audit at risk for an independence issue between the start of the period covered by the audit and the time that the auditor joined the audit team. This matter was identified last year in our Summary Report dated July 2015; a recommendation was made and management took appropriate action to address it. Because the four audit files were closed prior to the publication of our July 2015 Summary Report, we will not make a recommendation on this matter again this year.

19. In two of the four noted files, several Office specialists charged time to the audits but had not completed an Independence Confirmation form. Also, in one of these two files, we found that one external specialist who worked as an audit team member had not completed an Independence Confirmation form. This puts the audit at risk for an independence issue. This observation was also found in our Summary Report last year.

20. In one of the four audit files noted, an individual had identified a threat to their independence but did not complete an exception report for the Office’s Internal Specialist, Values and Ethics, to review. Although a conversation with the individual, the engagement leader, and the internal specialist did take place, the internal specialist indicated that he had also expected an exception report.

Human resources—Engagement team: assigning and managing tasks

21. OAG Policy 3061—Engagement team: assigning and managing tasks indicates that “before the completion of the planning/survey phase of an assurance engagement, the engagement leader shall assess the engagement team in order to be satisfied that the engagement team, specialists and any auditor’s experts, collectively have the appropriate competence and capabilities.” [Nov-2011]

22. In one file, the engagement leader used an external specialist as a team member but did not assess the competence of that specialist.

Engagement performance—supervision and review

23. The Direct Engagement Practice Team has developed and periodically updates a checklist for sign-offs in TeamMate so engagement leaders can easily see and understand the minimum expectation regarding sign-offs in the audit files.

24. We found that in three files, the engagement leader did not meet all of the minimum expectations for sign-offs in the files.

Engagement performance—engagement quality control review

25. OAG Policy 3062—Engagement leader responsibilities for audit quality identifies that determining whether an engagement quality control review (EQCR) is required for a direct engagement audit is based on a risk assessment. If, during the audit, the engagement leader determines that the risk associated with the audit has increased, the policy requires him or her to reconsider the need for an EQCR. Office guidance indicates that the engagement leader should consult with the assistant auditor general of the Audit Services Group.

26. We found that, in one file, the engagement leader had determined that the risk associated with the audit had increased. Rather than contacting the assistant auditor general of the Audit Services Group, the engagement leader asked another principal to act as the quality reviewer. The other principal identified and reviewed high-risk evidence, but did not complete any other quality control steps required of an EQCR. We found this component to be non-compliant.

Engagement performance—consultation

27. OAG Policy 3081—Consultations indicates that “the engagement leader shall ensure that the nature and scope of, and conclusions resulting from, consultations are documented and agreed to by both the individual seeking consultation and the party consulted, on or before the date of the assurance report.” [Nov-2011]

28. In an effort to reduce the size of the paper file, one audit team removed documentation related to consultation with an external specialist. Other internal specialists were consulted regarding other issues, and the associated documentation of that consultation was well done. We consider this to be an isolated incident.

Engagement performance—engagement documentation

29. For the current practice review cycle (for both financial audits and direct engagements), we have assessed security of sensitive information as a risk worthy of special attention. The OAG Security Policy states that “the Office is responsible for safeguarding the information and assets that it controls, including sensitive information that it creates and receives.”

30. According to that policy, regardless of storage location (TeamMate or PROxI), all protected audit working papers must be designated as such.

31. In performing our reviews, we found three files with issues related to the security of sensitive issues.

32. In one file, we found that the audit team created over 30 hard-copy files that were largely unnecessary to support the audit. Many of the items included were duplicates of TeamMate working papers or were not directly in support of audit work and audit findings. Almost none of the documents in the paper file were marked as reviewed, as Office audit policy requires. As well, the information retained in this file (both electronic and paper versions) was very sensitive and, in Practice Review and Internal Audit’s opinion, not properly marked or labelled as Protected information. Practice Review and Internal Audit is also of the view that if a privacy or access to information request was received, it could take a considerable amount of effort and time to review a paper file of this nature to properly respond to such a request. In addition, this file contained documents marked as Protected B and one Classified document, but these documents were stored in Protected A folders.

33. In another file, we found that the paper files contained Protected B information that had been stored in Protected A folders when sent for storage to Office records. A third file had information that, based on the audit team’s consultation with Office security, ought to have been identified as Protected B.

34. We concluded that the Engagement Documentation element of the System of Quality Control was non-compliant in the first file and was compliant and improvement needed in the other two files. We do not believe this is a systemic matter for the direct engagement audit practice requiring a change in procedures, but there should be a general reminder to engagement leaders to ensure that the security of sensitive information included in the audit file is considered throughout the audit work and again prior to closing the audit file.

35. Recommendation 1 to the Direct Engagement Audit Practice. Engagement leaders should ensure that audit staff are aware of and are applying the Office’s security policy, and that any document stored in the audit file be assessed against the policy and be labelled according to the proper security level.

Management’s response. Agreed. The Performance Audit Practice Management Committee will periodically invite the Office’s departmental security officer to brief the Performance Audit Principals’ Forum (that is, direct engagement leaders) on the Office’s security policies as they relate to audit documentation and how to comply with them. In addition, as part of this year’s annual methodology update for the Direct Engagement Practice, the Kick-Off Meeting Checklist for direct engagements has been amended to include a discussion of Office policies and requirements related to document security and labelling.

36. Note to the reader. In April 2016, Practice Review and Internal Audit completed its review of the attest practice. In that report (Report on a Review of the Financial Audit Practice—Financial Audits Completed in the 2015–16 Fiscal Year), we noted observations related to the security of sensitive information. At that time, we made recommendations to the Financial Audit Practice, the Annual Audit Practice Team, and the departmental security officer (DSO). For the direct engagement audit practice, we noted that the templates already indicate that once completed, these documents are to be considered Protected A. Therefore, we are of the view that we do not need to make a recommendation requiring changes in methodology. Because the recommendation to the DSO dealt with issues for audit, and was not specific to financial audit, we are not making that same recommendation in this report. In accordance with our discussion with the DSO, we understand that the DSO will ensure that the appropriate action is taken and that all audit practices are considered when looking at sessions, e-learning, and the like. The following recommendations were made:

Office policies on planning—examination approval

37. OAG Policy 4080—Examination approval requires a confirmation that the audit strategy, competencies of the team, and financial resources are appropriate for the audit to be completed within the set timelines. At the time of the audits under review, formal sign-off was required by the assistant auditor general, product leader, and engagement leader. In two of the selected audit files, we found that the examination approval was signed off very late in one case; in the other case, it was not signed off by a senior manager. Practice Review and Internal Audit notes that current Office methodology requires only the engagement leader’s sign-off on this step.

Office policies on planning—audit programs

38. In one audit file, we did not see evidence that the audit programs were reviewed and approved prior to the examination stage. There is a risk that procedures will be performed unnecessarily or that other key steps will be missed. We consider this to be an isolated incident.

Office policies on reporting—date of the report

39. In one file we found that one observation and its subsequent recommendation were somewhat inconsistent with each other. Also in our view, some of the evidence to support a positive observation was not well documented. We consider this to be an isolated incident.

40. OAG Policy 8017—Report content approval and date of the report states the following:

41. During our review, we noted two files with issues related to the date of the audit report. One file was dated prior to the quality reviewer finishing his review. In this case, the quality reviewer continued to review documents after the PX draft was sent. In another file, we noted that the engagement leader documented his review of high-risk areas after the date of the audit report.

42. We consider the issue related to the date of the audit report to be systemic and believe the related Office policy needs to be further explained to engagement leaders.

43. Recommendation 2 to the Direct Engagement Practice Team. The Direct Engagement Practice Team should provide engagement leaders with further guidance and explanation about establishing the date of the audit report.

Management’s response. Agreed. Our revised Canadian Standard for Assurance Engagements (CSAE) 3001–compliant reporting templates for direct engagements that will be issued in July 2016 include specific instructions on this matter. The revised significant judgments template and the direct engagement report assurance template that will be issued in November 2016 as part of the Practice’s annual methodology update will also address this matter. Updates to our professional development courses for performance auditing will include specific instruction on establishing report dates. We will also add an agenda item on this subject for discussion at an upcoming Performance Audit Principals’ Forum meeting in the fall of 2016.

Review of high-risk substantiation

44. Our observations in this area are based on several Office policies.

45. According to OAG Policy 7060—Substantiating the chapter,

46. According to OAG Policy 8019—Submitting the chapter to the entity: PX draft and DM draft (Guidance):

47. The Practice Review and Internal Audit interpretation of “review” is based on OAG Policy 1161—Documenting evidence of the extent of review, which explains that “in documenting the nature, timing, and extent of audit procedures performed, auditors shall record who reviewed the audit work performed and the date and extent of this review.” [Nov-2011]

48. To clarify, it provides further guidance:

49. And finally, according to OAG Policy 3071—Review of audit work and documentation (Guidance): “Evidence of review—When the reviewer has completed the review of each audit area within the file, evidence of the review should be indicated by electronic signature on the working papers and audit procedures summaries.” [Sep-2015]

50. Practice Review and Internal Audit expected that the requirements of policies 7060 and 8019, as described in paragraphs 45 and 46, would be documented in accordance with policy 1161, as described in paragraph 47. This expectation is also consistent with the following extract from TeamMate:

51. We found three audit files where the engagement leader did not document their review of the supporting evidence for high-risk areas in the file. Instead, in two of the files, the engagement leader signed off mostly on the text of the high-risk paragraphs. In a third, the engagement leader had signed off on a PX draft that had been footnoted with links to evidence. In short, the engagement leaders did not directly document their review of the actual evidence (working paper level or source documents).

52. In one of these three files, the documentation of that review was completed one day after the PX draft was sent to the entity. In another of these three files, the engagement leader did not document which paragraphs they considered to be high-risk paragraphs.

53. Practice Review and Internal Audit is concerned that this is a systemic issue. Also, we were unable to rely on one policy to clarify Office expectations and had to rely on three different policies, the PX minimum sign-off checklist, and a TeamMate step to outline that engagement leaders are expected to document the review of high-risk evidence (and not simply sign off on the paragraphs).

54. Recommendation 3 to the Direct Engagement Practice Team. The Direct Engagement Practice Team should update and clarify the expectations for review of high-risk substantiation so that the expectations are clear to engagement leaders.

Management’s response. Agreed. The Direct Engagement Practice Team will put the matter on the agenda for discussion at a Performance Audit Principals’ Forum meeting in fall 2016 as well as on the agenda for the next Practice team information session for practitioners. The Practice team will also review existing guidance and tools that address Engagement Leader review of substantiation for high-risk audit observations and findings with a view to further clarifying the Office’s expectations.

Other office policies—post tabling

55. With the use of the Controlled Document Interface (CODI), paper draft reports will rarely be distributed in the future. However, in one file we found that one paper draft report had been lost by an entity. The audit team did not inform the Office’s departmental security officer of this, as is required by OAG Policy 9020—Management of controlled documents. We consider this to be an isolated incident.

Good practices

56. We found good practices in several audit files. In one case, the audit team created an audit program that clearly and easily identified timelines, hours, and the responsible auditor for completing the work.

57. In another file, at the end of planning at the examination approval stage, the audit team linked at the approval stage all of the relevant references related to examination approval, such as budget and audit risk. Substantiating these items allowed the engagement leader and others who were required to sign off to be fully informed prior to signing off.

58. We noted that one team, after receiving guidance from the Professional Practices Group, gained a better understanding of related Office policies and retained no paper file.

59. Several teams made efficient use of TeamMate to document the request and receipt of information by tracking information requests in the audit step section and receipt in the results field. This approach allows teams to easily track outstanding items without extensive documentation.

Conclusion

60. For all of the direct engagement audit files we reviewed, we concluded that the audit report was supported and appropriate.

61. We concluded that two files were compliant, and four were compliant and improvement needed.

Appendix A—System of Quality Control Elements

Appendix A—text version

This diagram shows three sides of a cube, each side depicting aspects of the System of Quality Control.

The top of the cube shows the objectives of the System of Quality Control:

1. Compliance with professional standards and applicable legal and regulatory requirements; and
2. Reports issued are appropriate in the circumstances.

The right side of the cube shows the two levels of the System Quality Control:

• Firm level (CSQC 1)
• Engagement level (CAS 220 or S5030)

The left side of the cube shows the elements of the System of Quality Control:

• leadership,
• ethics and independence,
• acceptance and continuance,
• human resources,
• engagement performance, and
• monitoring

Appendix B—System of Quality Control Elements and Process Controls Reviewed

Our review covers the following System of Quality Control elements:

• leadership,
• ethics and independence,
• acceptance and continuance,
• human resources, and
• engagement performance.

Leadership. We reviewed whether the engagement leaders ensured that the audits were carried out in compliance with Office policies, professional standards, the System of Quality Control, and applicable legal and regulatory requirements.

Ethics and independence. We reviewed whether the engagement leaders ensured that the independence of all individuals performing audit work, including specialists, had been properly assessed and documented.

Acceptance and continuance. For initial or recurring engagements, we reviewed whether engagement leaders assessed that the team had the necessary competence, capability, time, and resources; that the team complied with relevant ethical requirements; and that it considered management’s integrity.

Human resources. We reviewed whether the engagement leaders assessed the audit team’s adequacy, availability, proficiency, competence, and resources and whether they documented their assessments.

Engagement performance

Within the engagement performance element, we also assessed:

• Supervision and review. We reviewed whether engagement leaders ensured that the audit files had documentation regarding who reviewed the audit work performed, the date, and the extent of the review.
• Consultation. We reviewed whether the engagement leaders ensured that appropriate consultations took place in a timely manner, when required.
• Engagement quality control review. We reviewed whether the quality review was carried out in a timely manner and whether the quality reviewer performed an objective evaluation of the significant judgments made by the team, the conclusions reached in supporting the auditor’s report, and other significant matters.
• Differences of opinion. If differences of opinion occurred, we reviewed whether the engagement leaders followed the Office’s established processes for addressing them.
• Engagement documentation. We reviewed whether engagement leaders properly addressed the confidentiality, safe custody, integrity, accessibility, retrievability, and retention of documentation, and whether the final assembly of the engagement file was completed on a timely basis (that is, the 60-day rule).

Other General Assurance and Auditing Standards requirements and OAG policies

We reviewed whether engagement leaders ensured that the audit was planned, executed, and reported in accordance with General Assurance and Auditing Standards, applicable legislation, and Office policies and procedures.

We also considered whether the Office met its reporting responsibilities by having in place appropriate audit methodology, recommended procedures, and practice aids to support efficient audit approaches and to produce sufficient audit evidence at the appropriate time.