COPYRIGHT NOTICE — This document is intended for internal use. It cannot be distributed to or reproduced by third parties without prior written permission from the Copyright Coordinator for the Office of the Auditor General of Canada. This includes email, fax, mail and hand delivery, or use of any other method of distribution or reproduction. CPA Canada Handbook sections and excerpts are reproduced herein for your non-commercial use with the permission of The Chartered Professional Accountants of Canada (“CPA Canada”). These may not be modified, copied or distributed in any form as this would infringe CPA Canada’s copyright. Reproduced, with permission, from the CPA Canada Handbook, The Chartered Professional Accountants of Canada, Toronto, Canada.
4043 Audit Criteria
Jul-2020
Overview
Auditors need a way to assess whether an entity’s performance in the areas subject to audit meets the audit objective. One of the fundamental elements of the OAG’s approach to direct engagement is the requirement that audits be based on suitable criteria—the standards of performance and control against which auditors assess performance.
CSAE 3001 Requirements
26. In order to establish whether the preconditions for a direct engagement are present, the practitioner shall, on the basis of a preliminary knowledge of the engagement circumstances and discussion with the appropriate party(ies), determine whether: (Ref: Para. A35–A37)
[...]
(b) The engagement exhibits all of the following characteristics:
[...]
(ii) The criteria that the practitioner expects to be applied are suitable for the engagement circumstances, including that they exhibit the following characteristics: (Ref: Para. A43-A48)
- Relevance.
- Completeness.
- Reliability.
- Neutrality.
- Understandability.
(iii) The criteria that the practitioner expects to be applied will be available to the intended users; (Ref: Para. A49-A50)
[...]
31. The practitioner shall seek to obtain from the responsible party, written acknowledgement that the criteria are suitable for the engagement. When such acknowledgement cannot be obtained, the practitioner shall consider the effect, if any, on the practitioner’s work and report.
45. The practitioner shall determine whether the criteria are suitable for the engagement circumstances, including that they exhibit the characteristics identified in paragraph 26(b)(ii).
47. If it is discovered after the engagement has been accepted that one or more of the applicable criteria are unsuitable, the practitioner shall, if practicable, revise the criteria and seek acknowledgement from the responsible party that the revision is appropriate. When such an acknowledgement cannot be obtained, the practitioner shall consider the effect, if any, on the practitioner’s work and report.
73. The assurance report shall include at a minimum the following basic elements:
[...]
(e) Identification or description of the applicable criteria. (Ref: Para. A144-A146, A165)
[...]
CSAE 3001 Application Material
A12. Suitable criteria are required for reasonably consistent measurement or evaluation of an underlying subject matter within the context of professional judgment. Without the frame of reference provided by suitable criteria, any conclusion is open to individual interpretation and misunderstanding. The suitability of criteria is context-sensitive; that is, it is determined in the context of the engagement circumstances. Even for the same underlying subject matter there can be different criteria, which will yield a different measurement or evaluation. For example, a practitioner might select, as one of the criteria for the underlying subject matter of customer satisfaction, the number of customer complaints resolved to the acknowledged satisfaction of the customer; another practitioner might select the number of repeat purchases in the three months following the initial purchase. The suitability of criteria is not affected by the level of assurance; that is, if criteria are unsuitable for a reasonable assurance engagement, they are also unsuitable for a limited assurance engagement, and vice versa. Suitable criteria include, when relevant, criteria for presentation and disclosure.
A35. In a public sector environment, some of the preconditions for an assurance engagement may be assumed to be present, for example:
(a) The roles and responsibilities of public sector audit organizations and the government entities scoped into assurance engagements are assumed to be appropriate because they are generally set out in legislation;
(b) Public sector audit organizations' right of access to the information necessary to perform the engagement is often set out in legislation;
(c) The practitioner's conclusion, in the form appropriate to either a reasonable assurance engagement or a limited assurance engagement, is generally required by legislation to be contained in a written report; and
(d) A rational purpose is generally present because the engagement is set out in legislation.
A36. If suitable criteria are not available for all of the underlying subject matter but the practitioner can identify one or more aspects of the underlying subject matter for which those criteria are suitable, then an assurance engagement can be performed with respect to that aspect of the underlying subject matter in its own right. In such cases, the assurance report may need to clarify that the report does not relate to the original underlying subject matter in its entirety.
Roles and Responsibilities (Ref: Para. 14(n), 14(q), 14(u), 15, 26(a), Appendix 1)
A37. All assurance engagements have at least three parties: the responsible party, the practitioner, and the intended users.
A43. Suitable criteria exhibit the following characteristics:
(a) Relevance: Relevant criteria result in a practitioner’s report that assists decision-making by the intended users.
(b) Completeness: Criteria are complete when they do not omit relevant factors that could reasonably be expected to affect decisions of the intended users. Complete criteria include, where relevant, benchmarks for presentation and disclosure.
(c) Reliability: Reliable criteria allow reasonably consistent measurement or evaluation of the underlying subject matter when used in similar circumstances by different practitioners.
(d) Neutrality: Neutral criteria result in a practitioner’s report that is free from bias as appropriate in the engagement circumstances.
(e) Understandability: Understandable criteria result in a practitioner’s report that can be understood by the intended users.
A44. Vague descriptions of expectations or judgments of an individual’s experiences do not constitute suitable criteria.
A45. The suitability of criteria for a particular engagement depends on whether they reflect the above characteristics. The relative importance of each characteristic to a particular engagement is a matter of professional judgment. Further, criteria may be suitable for a particular set of engagement circumstances, but may not be suitable for a different set of engagement circumstances. For example, reporting to governments or regulators may require the use of a particular set of criteria, but these criteria may not be suitable for a broader group of users.
A46. Criteria can be selected or developed in a variety of ways, for example, they may be:
- Embodied in law or regulation.
- Issued by authorized or recognized bodies of experts that follow a transparent due process.
- Developed collectively by a group that does not follow a transparent due process.
- Published in scholarly journals or books.
- Developed for sale on a proprietary basis.
- Specifically designed for the purpose of measuring or evaluating the underlying subject matter in the particular circumstances of the engagement.
How criteria are developed may affect the work that the practitioner carries out to assess their suitability.
A47. In some cases, law or regulation prescribe the criteria to be used for the engagement. In the absence of indications to the contrary, such criteria are presumed to be suitable, as are criteria issued by authorized or recognized bodies of experts that follow a transparent due process if they are relevant to the intended users’ information needs. Such criteria are known as established criteria. Even when established criteria exist for an underlying subject matter, specific users may agree to other criteria for their specific purposes. For example, various frameworks can be used as established criteria for evaluating the effectiveness of internal control. Specific users may, however, develop a more detailed set of criteria that meet their specific information needs in relation to, for example, prudential supervision. In such cases, the assurance report may note, when it is relevant to the circumstances of the engagement, that the criteria are not embodied in law or regulation, or issued by authorized or recognized bodies of experts that follow a transparent due process.
A48. If criteria are specifically designed for measuring and evaluating the underlying subject matter in the particular circumstances of the engagement, they are not suitable if they result in an assurance report that is misleading to the intended users.
A49. Criteria need to be available to the intended users to allow them to understand how the underlying subject matter has been measured or evaluated. Criteria are made available to the intended users in one or more of the following ways:
(a) Publicly.
(b) Through inclusion in a clear manner in the assurance report (see paragraph A164).
(c) By general understanding, for example, the criterion for measuring time in hours and minutes.
A144. The description of the applicable criteria advises intended users of the framework on which the underlying subject matter is being evaluated, and is particularly important when there are significant differences between various criteria regarding how particular matters may be evaluated.
A145. A description that the underlying subject matter complies with particular applicable criteria is appropriate only if the underlying subject matter complies with all relevant requirements of those applicable criteria that are effective.
A146. A description of the applicable criteria that contains imprecise qualifying or limiting language (for example, “the underlying subject matter is in substantial compliance with the requirements of XYZ”) is not an adequate description as it may mislead users of the practitioner’s report.
A165. The assurance report identifies the applicable criteria against which the underlying subject matter was measured or evaluated so the intended users can understand the basis for the practitioner’s conclusion. The assurance report may include the applicable criteria, or refer to them if they are otherwise available from a readily accessible source. It may be relevant in the circumstances, to disclose:
- The source of the applicable criteria, and whether or not the applicable criteria are embodied in law or regulation, or issued by authorized or recognized bodies of experts that follow a transparent due process; that is, whether they are established criteria in the context of the underlying subject matter (and if they are not, a description of why they are considered suitable).
- A statement that the responsible party agreed with the suitability of criteria or that the acknowledgement has not been obtained.
- Measurement or evaluation methods used when the applicable criteria allow for choice between a number of methods.
- Any significant interpretations made in applying the applicable criteria in the engagement circumstances.
- Whether there have been any changes in the measurement or evaluation methods used.
Financial Administration Act
Section 138(3) Before an examiner commences a special examination, he shall survey the systems and practices of the corporation to be examined and submit a plan for the examination, including a statement of the criteria to be applied in the examination, to the audit committee of the corporation, or, if there is no audit committee, to the board of directors of the corporation.
Section 139(2)(a) The report of an examiner under subsection (1) shall include a statement whether in the examiner’s opinion, with respect to the criteria established pursuant to subsection 138(3), there is reasonable assurance that there are no significant deficiencies in the systems and practices examined.
OAG Policy
Audits shall have suitable criteria against which auditors assess evidence, in order to develop observations and draw conclusions with respect to audit objective(s). [Nov-2015]
Audit teams shall seek entity management’s acknowledgement of the suitability of the audit criteria. When the audit team is unable to obtain such acknowledgment, the engagement leader shall consider the effect, if any, on the audit work and the audit report, and document the assessment. [Nov-2016]
In the audit report, the audit team shall identify the criteria used for the audit and disclose the sources of the criteria. [Nov-2016]
The scope of all special examinations of Crown corporations shall, at a minimum, cover “core” systems and practices which are assessed using the Office’s standard criteria. Based on a risk and control assessment performed in the planning phase, the engagement leader can justify expanding the scope of the special examination beyond the core systems and practices. [Nov-2017]
OAG Guidance
What CSAE 3001 means for developing audit criteria
CSAE 3001 requires that the audit team applies suitable audit criteria that exhibit the following characteristics:
-
Relevance. Relevant criteria result in an audit report that can assist Parliament or territorial legislative assemblies in their decision-making process.
-
Completeness. Criteria are complete when they do not omit relevant factors that could affect decisions of the intended users.
-
Reliability. Reliable criteria allow a relatively consistent measurement or evaluation of the subject matter when used in similar circumstances by different auditors.
-
Neutrality. Neutral criteria are free from bias.
-
Understandability. Understandable criteria result in an audit report that can be understood by the intended users.
The suitability of criteria is context-sensitive and must be appropriate to the characteristics and activities of the audited entity and depends on the circumstances of the audit.
Audit criteria can be selected or developed in a variety of ways:
-
They are often based on laws and regulations because they relate to the entity or the government as a whole; for example, sections of the Immigration and Refugee Protection Act for an audit on selecting foreign workers under Canada’s immigration program.
-
They are frequently derived from central agency or entity policies, directives, guidelines, and plans; for example, Treasury Board policies and Departmental Plans (formerly known as RPP). These types of criteria are developed collectively by a group that does not follow a transparent due process.
-
Criteria can be based on recognized bodies of experts that follow a transparent due process including public consultation and debate; for example, standards established by the International Organization for Standardization.
-
Criteria can also be based on international commitments Canada took; for example, the United Nation’s sustainable development goals.
-
In some cases, criteria are specifically designed for measuring or evaluating the subject matter in the particular circumstances of the engagement.
CSAE 3001 also requires that criteria be made available to the intended users of the audit report to help them understand how the subject matter of the audit has been measured or evaluated. The OAG always discloses the criteria used in its audit reports in the “About the Audit” section of the report, and includes the sources of the criteria.
Developing the criteria
Criteria should be developed for each area to be audited (line of audit inquiry or audit project). There can be one criterion or several. They focus, wherever possible, on the results that the program, operation, system, practice, or control is expected to achieve. When wording criteria, audit team members should:
-
Express each criterion as an expectation statement vis-à-vis the entity(ies) that is derived from the source authority for the criterion.
-
Express the expectation in a way that enables a conclusion to be drawn against it—either expectation met or not met (as with audit objectives).
-
Divide long criteria into two or more, particularly when they have more than one major component or they have different sources; for example,
-
“We expect CRA, CIC, and HRSDC to have regularly measured their service performance to identify service quality issues.”
-
“We expect CRA, CIC, and HRSDC to have reported to Parliament and the public on their service performance.”
-
The assessment of the situation/condition compared to the expectations set out by the criterion results in audit findings. Taken together, the audit criteria (and associated findings) should be sufficient to allow the audit team to form a conclusion against the audit objective(s).
Criteria are a key component of the audit approach (OAG Audit 4042 Audit scope and approach). In order to assess performance expectations, audit teams develop audit questions. For further information, see OAG Audit 4044 Developing the audit strategy: audit logic matrix and OAG Audit 4045 Evidence-gathering methods.
Special examinations
The OAG has developed a set of “core” systems and practices and related standard criteria that must be examined in every special examination. After completing the risk and internal controls assessment for the Crown corporation under audit, the special examination team considers whether to expand the scope of the special examination beyond the required core systems and practices and related standard criteria. The engagement leader must ensure that the audit scope and approach respond to the risks that could prevent the corporation from achieving its statutory control objectives. Core systems and practices and related standard criteria, as well as additional guidance for adding to them based on the risk and controls assessment, can be found in the document, “Special Examination Audit Approach”. In order to ensure the Crown corporation has a clear understanding of the basis upon which they will be examined, the engagement leader may wish to consider including audit questions in the special examination plan and encouraging the corporation’s management to review the source materials of the criteria.
Documenting the assessment of suitability of criteria
During the process of selecting or developing suitable criteria for the audit, audit teams should document significant professional judgements made to assess the suitability of audit criteria (OAG Audit 1143 Documenting significant matters and related significant professional judgments), such as the advice received from the appropriate internal specialist (if any); the decision made to keep or not to keep some selected criteria; the various sources of criteria used or not used; some rationale that explains why the criteria are considered suitable in the context of the audit; etc.
Sources of criteria
The audit team may refer to many different sources when selecting or developing suitable criteria, including
- Laws and regulations governing the operations of the entity;
- Government and board policies;
- Good practices of the sector;
- Decisions made by the legislature or the executive branch of government;
- Key performance indicators used by the entity or the government;
- Standards developed through research or used by professional and/or international organizations;
- Benchmarks of good performance for comparative entities;
- Planning documents, contracts and budgets from the entity;
- Criteria used in similar performance audits; and
- Consultation with subject matter experts.
The criteria and their sources are disclosed in the “About the Audit” section of the report (OAG Audit 7030 Drafting the audit report). These sources determine the amount of effort needed to ensure the suitability of the criteria. When using laws or regulations as criteria, the audit team only need to ensure that they are directly related to the audit objective. The same is true of central agency or entity policies. Although central agency and entity policies are not usually subject to public debate, they are based on consultation within government and are authoritative.
Directives, guidelines, plans, tools, controls, and measures developed by central agencies, Crown corporations, and government departments and agencies are less authoritative. However, they can be used as criteria if the audit team can validate their suitability through sufficient research and validation. The audit team can consult with professional bodies or other organizations carrying out similar activities or operations to test the quality of the standards or to identify best practices.
Criteria developed specifically for the audit include criteria based on performance data from other organizations, inside or outside the federal government, that have
- comparable activities or operations,
- best practices determined through benchmarking or consultation, and
- standards the auditors developed by analyzing a task or activity.
These types of criteria require the most effort by the team to ensure their suitability.
Over the years, the OAG has developed and tested criteria for a large number of departments, Crown corporations, agencies, programs, and operational areas. However, the fact that these criteria have been used in the past does not, by itself, make the criteria authoritative. It is the audit team’s responsibility to reassert the source and suitability of the criteria.
Frequently, a criterion is based on more than one source. Auditors should document how the criterion was derived from multiple sources.
For example, in an audit on selecting foreign workers under Canada’s immigration program, a criterion was developed based on the Federal Accountability Act, sections of the Immigration and Refugee Protection Act, the Treasury Board of Canada Secretariat’s Management Accountability Framework, and other government planning documents.
Audited entity’s acknowledgement of the suitability of criteria
The OAG seeks entity management’s acknowledgement of the suitability of the criteria. For performance audits, criteria are presented to the entity as part of the audit plan summary (OAG Audit 4090 Audit plan summary for performance audits), and for special examinations as part of the special examination plan (OAG Audit 4100 Special examination plan). The entity is given an opportunity to comment on the criteria, and the team may make changes as a result. The deputy head (for performance audits) or the head of the Crown corporation (for special examinations) is asked to acknowledge in writing the suitability of the criteria.
If the team is unable to obtain acknowledgement from the entity’s management that the criteria are suitable, the engagement leader must assess the impact on the audit work and the audit report. A clear case must exist and be documented on why, despite objections by the entity, the engagement leader feels the criteria are suitable in the circumstances. Under no circumstances is the audit to be carried out using criteria that would result in biased or misleading audit results. If there is disagreement with management about the criteria, this should be disclosed in the audit report with an explanation of why the audit team used the criteria despite management’s objection.
As the audit progresses, additional information may result in a criterion not being necessary for achieving the audit objective. In these circumstances, further audit work related to the criteria is not needed; however, the team should document the reason for eliminating a criterion in the audit file and also notify the entity. Eliminated criteria do not appear in the audit report.
Follow-up work and criteria
Follow-up audit work examines the recommendations or significant findings made in previous OAG audits. The previous recommendations or findings serve as audit criteria. Further, commitments made by entities in response to audit recommendations may also serve as criteria. However, if there are redefined or additional issues being considered, the team needs to formulate new criteria. For further information, see OAG Audit 4042 Audit scope and approach, under the “Inclusion of follow-up work” section.