Executive Summary

The objective of the internal audit was to determine whether the Office of the Auditor General of Canada had an adequate and effective management framework for learning, training, and development to ensure that employees received the training and development they needed, when they needed it, to carry out their duties.

The audit covered the period from 1 April 2015 to 31 March 2018.

This internal audit is important because well-trained, knowledgeable, and professional employees contribute to delivering audit products that focus on results for Canadians and comply with professional standards and Office policies.

As a result of this audit, we concluded that the Office had an adequate and effective management framework for learning, training, and development to ensure that employees received the training and development they needed, when they needed it, to carry out their duties. However, the Office could make some improvements.

Management agrees with our recommendations. Its detailed responses follow the recommendations throughout the report.

We requested and received an action plan from management on 3 October 2018.

This internal audit was conducted according to the Institute of Internal Auditors’ International Standards for the Professional Practice of Internal Auditing and in compliance with the Treasury Board’s 2017 Policy on Internal Audit and 2017 Directive on Internal Audit.

In my professional judgment as Chief Audit Executive, sufficient and appropriate audit procedures were conducted and evidence was gathered to support the accuracy of the opinion provided and contained in this report. The opinion is applicable only to the area examined. The evidence was gathered according to professional standards of the Institute of Internal Auditors. The evidence was sufficient to provide management with proof of the opinion derived from the internal audit.

Louise Bertrand
Chief Audit Executive
Office of the Auditor General of Canada

19 October 2018

Introduction

1. This internal audit focused on determining whether the Office of the Auditor General of Canada had an adequate and effective management framework for learning, training, and development to ensure that employees received the training and development they needed, when they needed it, to carry out their duties.

2. This internal audit is important because well-trained, knowledgeable, and professional employees contribute to delivering audit products that focus on results for Canadians and comply with professional standards and Office policies.

3. Investing in learning, training, and development can help with implementing the Office’s strategic objectives that support employee learning and growth, with improving organizational and employee performance, and with increasing employee retention.

4. The Office’s investment in learning, training, and development has increased over the years. In the 2017–18 fiscal year, the Professional Development team managed a non-salary operating budget of $1.2 million.

5. The internal audit team selected the following criteria for assessing the Office’s systems and practices:

• Governance. The Office has established an effective governance framework for learning, training, and development. Roles and responsibilities are formally defined, documented, approved, and communicated. Those charged with oversight responsibilities ensure that accountabilities are being executed as intended.
• Risk management. The Professional Development team has conducted a risk assessment of its learning, training, and development activities that support the Office’s strategic objectives. Mitigation strategies are developed, documented, and monitored.
• Internal controls. Policies, procedures, and control activities are in place, documented, and monitored for the cost-effective design and delivery of learning, training, and development.
• Performance measures and outcomes. Performance measures have been defined, documented, and monitored to assess learning, training, and development activities and expected outcomes in support of relevant Office strategic objectives. These performance measures are revised when needed, and performance outcomes are reported to senior management.

6. Our audit covered the period from 1 April 2015 to 31 March 2018.

7. More details about the audit objective, systems and practices examined, and criteria are in About the Internal Audit at the end of this report.

Findings, Recommendations, and Responses

Governance

The Office had a governance framework in place for learning, training, and development, but some improvements were needed

8. Without good governance, the Office of the Auditor General of Canada might not be able to meet its strategic objective of developing and maintaining a skilled, engaged, and bilingual workforce. Without good oversight, roles and responsibilities for training and development might not be carried out as intended.

9. We examined whether the Office’s Policy on Learning and Professional Development and supporting directives and the Office’s Bilingualism in the Workplace Strategy were aligned with the training and development requirements in the Official Languages Act, the Treasury Board’s Policy on Official Languages, and the Treasury Board’s Directive on Official Languages for People Management.

10. In addition, although the Office is not subject to them, we reviewed the Treasury Board’s 2017 Policy on Learning, Training, and Development and 2017 Directive on the Administration of Required Training, for comparative purposes.

11. We also reviewed the Office’s governance framework to determine whether

• those responsible for oversight ensured that accountabilities were being executed as intended; and
• the Office had formally defined, documented, approved, and communicated roles and responsibilities for learning, training, and development.

12. Furthermore, we examined whether the Professional Development team had conducted a risk assessment of its learning, training, and development activities that support the Office’s strategic objectives of

• being a financially well-managed organization accountable for the use of resources entrusted to it;
• ensuring effective and efficient support services;
• ensuring a culture of empowerment; and
• developing and maintaining a skilled, engaged, and bilingual workforce.

13. Our findings discuss the following topics:

• Alignment with Treasury Board and legislative requirements
• Policies and directives
• Oversight
• Risk identification and mitigation

14. Alignment with Treasury Board and legislative requirements. The Office is subject to the Official Languages Act, the Treasury Board’s Policy on Official Languages, and the Treasury Board’s Directive on Official Languages for People Management.

15. We found that the following Office documents were aligned with the training requirements set out in the Act and in the Treasury Board’s policy and directive:

• Policy on Learning and Professional Development
• Directive 8—Official Languages Training
• Directive on the Identification, Designation and Staffing of Bilingual Positions
• Bilingualism in the Workplace Strategy

16. Also, when we compared the Office’s policy and directives with the Treasury Board’s Policy on Learning, Training and Development and the Treasury Board’s Directive on the Administration of Required Training, we found that they were substantially aligned.

17. Policies and directives. Policies and directives set out the corporate direction and expected outcomes for learning, training, and development, as well as describe the roles, responsibilities, and accountabilities of all stakeholders.

18. We found that the Office set out three key objectives in its Policy on Learning and Professional Development:

• to recognize that continuous learning by employees is essential to the success of the organization and is a shared responsibility,
• to contribute to the creation of a competent workforce, and
• to adopt practices that promote employees’ continuous learning and professional development.

19. We also found that to ensure the implementation of these objectives, the Office included in its policy a number of required responsibilities for the Auditor General and the Executive Committee, management, employees, and the Professional Development team. Listed below are a few examples:

• The Auditor General and the Executive Committee must approve the Policy on Learning and Professional Development and any revisions made to it. They must also allocate resources to meet organizational needs and promote an environment of continuous learning.
• Managers at all levels must comply with the policy and complete the Training and Development Plan with the employee to define training needs.
• Employees must comply with the policy and complete, sign, implement, and review their training and development plans in order to develop or improve their current or desired competencies.
• The Professional Development team must coordinate the application of the policy at the organizational level. The team must also recommend, develop, and implement initiatives and training programs, and ensure that all employees have access to formal learning activities in accordance with the principles of equity and transparency.

20. However, we found that the policy and supporting directives did not refer to the roles and responsibilities of some of the stakeholders who contributed to learning, training, and development, such as

• internal specialists and practice teams who contributed to the design and development of training programs,
• Human Resources staff who provided management coaching and managed a training development fund, and
• administrative directors who were responsible for approving internal training.

21. Also, in our review of job descriptions for Professional Development managers and staff, we found that although there were job descriptions for Professional Development positions, the job description used for the Principal responsible for the professional development function (identified in the job description as Principal of Corporate Services) was dated 1999.

22. Oversight. Oversight ensures that responsibilities are being carried out as intended.

23. We found that the Office’s Policy on Learning and Professional Development did not specify who was responsible for overseeing and for monitoring the application of the policy and supporting directives. As a result, there was no assurance that the responsibilities as defined in the policy were being carried out and that learning plans were being completed as required. We also noted in our review of the directives that the Professional Development team was responsible only for monitoring compliance with required training. We noted, however, that the Assistant Auditor General of Audit Services reviewed and monitored the activities of the Professional Development team through regular meetings.

24. In our review of the Executive Committee’s meeting documents, we found that the Professional Development team provided the Committee with updates on the Office’s compliance with required training, leadership training, and official languages training.

25. The policy has been in effect since 2015, with a review scheduled for every five years. Since the Office’s compliance review project to review internal policies is currently under way, it is our view that the Professional Development team should review its policy now, by working with Legal Services to clarify roles and responsibilities for oversight of the policy and for monitoring the implementation of the policy and its directives, and how this will be conducted.

26. The Professional Development team should work with Legal Services to review its Policy on Learning and Professional Development and supporting directives to clarify the roles and responsibilities of those responsible for overseeing and for monitoring the application of the policy and directives.

Management’s response. Agreed. The Professional Development team will work with Legal Services as part of the second phase of the Office of the Auditor General of Canada’s compliance initiative and will ensure that roles and responsibilities are clear.

27. The Office of the Auditor General of Canada should ensure that there is an up-to-date description for the job held by the Principal responsible for the Professional Development team.

Management’s response. Agreed. Management worked with Human Resources to develop the job description for Audit Services principals. This job description is now up to date.

28. Risk identification and mitigation. Risk assessments and mitigation strategies ensure that the proper controls are in place to reduce risk.

29. We found that the Professional Development team identified a number of risks linked to the Office’s strategic objectives and related to

• the team’s operations and compliance with relevant regulatory instruments,
• the delivery of external training, and
• fraud and wrongdoing.

The team did not determine any of these risks to be high.

30. We also found that the team had identified a number of controls to mitigate risks that were considered to be normal to elevated. However, we also noted that at the time of our audit, the Office required service leaders to document only the controls in place to mitigate high residual risks. Therefore, the team had not yet documented all of their control processes, or the work performed by them, to ensure that key controls were operating effectively.

31. For the management of its external training budget, we found that the team had identified the following:

• the risk of external training not meeting needs, or of the right people not being sent on external training;
• the risk of approving training not considered to be required or reasonable by an objective third party; and
• the risk of staff misrepresenting attendance and expense information, for personal benefit.

32. We found that the team had a process in place for approving external training and that the team’s management of this process had improved from year to year. However, the team had not documented the work performed by them to ensure key controls are operating effectively.

33. We also noted that the Office’s Directive 5—External Training required that employees complete a form to confirm attendance and evaluate the training received. However, we found that many employee attestations were missing from the files we reviewed.

34. According to the Professional Development team, the purpose of the completed form is to record the successful completion of an employee’s external training hours in the Office’s learning management system (OAG Campus). The team did not identify the completion of the form as a control for reducing the risk of an employee not attending a course that was already paid for out of the external training budget. We did not find an internal control in place to mitigate this risk.

35. We noted that the team had identified non-compliance with the Office’s procurement directives as a risk that contributed to the Office’s overall strategic risk. However, the team did not identify the controls in place to mitigate this operational risk in terms of managing contracts valued at $776,000 in the 2016–17 fiscal year. The team also did not identify strategic or operational risks related to

• non-compliance with the Office’s policy and directives (other than for external training); and
• managing the budgets for official languages and internal training, which were of $441,000 and $322,000, respectively, in the 2016–17 fiscal year.

36. As a result, we encourage the team to complete the documentation of its procedures, including key controls and mitigation strategies, and to test these controls for ongoing effectiveness.

37. The Professional Development team should review its risk registry to include risks related to non-compliance with the Office’s Policy on Learning and Professional Development and supporting directives, as well as risks related to the management of financial resources.

Management’s response. Agreed. The Professional Development team updated its 2018–19 risk register to include risks related to non-compliance with the policy and directives and to the mismanagement of financial resources.

Training needs, strategies, and evaluation

The Professional Development team analyzed needs for required training but did not evaluate whether the training helped improve employee performance

38. An effective analysis of training needs can assess the gaps between the knowledge, skills, and behaviours that employees currently have and the knowledge, skills, and behaviours that employees require to improve their performance and meet the strategic objectives of the Office of the Auditor General of Canada.

39. Plans, strategies, and budgets address the training needs identified in the needs analysis.

40. An evaluation of needs assessments and strategies to close the gaps can provide information on how well the Professional Development team is meeting its objectives.

41. We examined the Professional Development team’s

• analyses of gaps and deficiencies in employee and organizational performance, which help to identify training needs;
• annual review of training needs (or environmental scans);
• documentation that explained how the team developed and implemented plans, strategies, budgets, programs, and activities to address the learning, training, and development needs identified; and
• evaluation information used to assess the team’s learning, training, and development activities.

42. Our analysis supporting this finding discusses the following topics:

• Training needs analysis
• Plans, strategies, and budgets
• Evaluation

43. Training needs analysis. A training needs analysis is a process that identifies the gaps and deficiencies in employee and organizational performance and thus helps to identify training needs.

44. The Professional Development team’s 2016–2019 Business Plan states that the team contributes to the Office’s strategic objectives of

• developing and maintaining a skilled, engaged, and bilingual workforce; and
• ensuring a culture of empowerment.

45. We found that the team conducted training needs analyses and environmental scans in support of the Office’s strategy to develop and maintain a skilled, engaged, and bilingual workforce. The analyses and environmental scans identified needs for the following types of training:

• compliance training, such as training on the delegation of authorities;
• technical training, such as training on audit methodology;
• leadership training, such as training on coaching, delegating, and managing performance;
• soft-skills training, such as training on teamwork and time management; and
• official languages training, such as training for second-language testing.

46. In reviewing the 2017–18 environmental scan, we found that the team did not identify training needs associated with the second objective. We noted, however, that the team facilitated empowerment sessions for all employees, even though the team was not the lead for this objective.

47. To help it complete these analyses and scans, the team consulted with the Office’s training specialists, practice teams from Audit Services, and managers from Corporate Services. However, consultations with managers and employees as end users were not part of this process, as the team relied on course evaluations to identify additional needs.

48. We found that the team’s environmental scans did not include the training needs identified in employees’ learning plans. Human Resources informed us that it did not provide the Professional Development team with access to these plans because they were part of the appraisal forms.  Once completed, these forms contained sensitive and protected private information.

49. In addition, we found that the team did not document how and when training needs analyses should be conducted to identify gaps or deficiencies in organizational or employee performance. Furthermore, the team did not consistently document consultations with stakeholders in environmental scans.

50. Though the team communicated the results of its environmental scan for official languages to the relevant service areas, it did not communicate the training needs for other types of training to service areas.

51. Because the team did not have access to employees’ learning plans, it is our view that consulting with managers and employees as end users would provide valuable information for determining training needs. It is also our view that letting management know about training needs and the strategies to address them and requesting management’s feedback would improve the process and help set priorities.

52. Plans, strategies, and budgets. Plans, strategies, and budgets address the training needs identified in training needs analyses.

53. We found that the Professional Development team’s learning, training, and development strategies and plans aligned with priorities and requirements related to the Office’s strategic objective of developing and maintaining a skilled, engaged, and bilingual workforce.

54. We also found that the team’s training strategies sought to offer training in a timely manner by ensuring that employees had access to accounting e-learning and online soft-skills training when they needed it.

55. In addition, we found that the team had developed business cases to support its training expenditures for technical training, soft-skills training, and official languages training.

56. We found, however, that the team did not document how it developed its plans, strategies, and budgets.

57. Evaluation. Evaluations assess how effective plans and strategies were in addressing gaps or deficiencies in employee and organizational performance.

58. We found that the Professional Development team used course evaluations to assess participants’ feedback. The team used the evaluations to determine whether the course objectives were met and whether courses were relevant to employees’ work. The team also used passing test results and attendance records to determine the successful completion of training.

59. However, we found that the team did not conduct evaluations to assess whether

• the Office was achieving the expected outcomes of learning strategies,
• employees were applying training in their jobs,
• employees’ behaviours had changed, and
• employees’ performance had improved.

60. We noted that the team had planned to develop an integrated evaluation strategy in the 2017–18 fiscal year. The team informed us that this strategy had been deferred because of changes in priorities, and as a result, dedicated resources were not in place to work on the strategy.

61. The Professional Development team should document the processes for training needs analysis and environmental scans and for developing plans, strategies, and budgets.

Management’s response. Agreed. The Professional Development team will document the processes for training needs analysis and environmental scans and for developing plans, strategies, and budgets.

62. The Professional Development team should finish developing an integrated evaluation strategy by the 2020–21 fiscal year and report on its results. The strategy should determine how effective the team’s learning strategies are for acquiring technical, leadership, official languages, and soft skills.

Management’s response. Agreed. As planned for in its Business Plan, the Professional Development team will develop an integrated evaluation strategy by the 2020–21 fiscal year and report on its results.

Monitoring and reporting

The Professional Development team reported on performance measures for required and official languages training

63. Investing in learning, training, and development is critical to the Office of the Auditor General of Canada’s successful execution of its mandate, vision, and strategic objectives. Monitoring and reporting on performance measures and activities, such as tracking how funds are allocated and reporting on the results achieved, can help the Office assess the value of its investment.

64. We examined

• performance measures (or indicators) used by the team to report against the Office’s strategic objectives and tested the quality of the information used for reporting purposes, and
• documentation that explained how the team monitored and reported on its activities.

65. Our analysis supporting this finding discusses the following topics:

• Performance measures
• Data quality
• Monitoring
• Reporting

66. Performance measures. Performance measures (or indicators) provide management and the Office’s Executive Committee with information on whether an activity is being delivered efficiently and effectively, and whether expected outcomes are being achieved.

67. In terms of performance measures, the Executive Committee had established a corporate-level performance measurement framework, which included performance indicators for all of the Office’s strategic objectives. We found that the Professional Development team had contributed to the corporate-level performance indicators for the strategic objective of developing and maintaining a skilled, engaged, and bilingual workforce. The team developed training and tracked and reported on

• compliance with required training by providing information on the number and percentage of employees who completed required training, and
• the number and percentage of employees who completed the training that was required to meet the linguistic profiles of their positions.

68. We found that in its performance reports, the Professional Development team had reported on its use of its operating budget (planned versus actual), including an explanation of any variances. In our view, budget variance is a potential indicator that the team might consider using at the operational level to contribute to the Office’s strategic objective of being a financially well-managed organization accountable for the use of resources entrusted to it.

69. We also found that the team had recently conducted its first internal client satisfaction survey in 2017. We encourage the team to perform periodic client surveys as a way of evaluating how effective its learning strategies are. We also encourage the team to develop service standards, as other support services have done. This would contribute at the operational level to the strategic objective of ensuring effective and efficient support services.

70. Data quality. Data quality ensures that the information used for monitoring is comprehensive and reliable.

71. We found that the Professional Development team had a documented process in place to ensure that data entered into the Office’s learning management system (OAG Campus) and monitoring system (Dashboard) was correct.

72. However, in our review of course documentation for required and non-required training, we found some missing information and data errors.

73. In our review of documentation for 16 required training courses delivered in the 2016–17 and 2017–18 fiscal years, we found that 6 courses were missing attendance sheets. Signed attendance sheets document employee attendance and successful completion of courses, information that is then added to the Dashboard for reporting purposes.

74. We found that, despite the absence of attendance sheets, 30 employees were credited as having attended the courses.

75. We also found that for courses that had a signed attendance sheet, 2 employees did not receive credit for completing a required course even though they had signed the attendance sheets.

76. In our review of documentation for 32 training courses (16 required and 16 non-required courses), we found some data entry errors, the most common of which was that employees were not being credited for the courses they attended:

• 2 of 16 (13%) of required training courses sampled had some data entry errors associated with them, and
• 3 of 16 (19%) of non-required training courses had some data entry errors associated with them.

77. Monitoring. Monitoring tracks the delivery of plans, the implementation of strategies, and the allocation of budgets.

78. We found that the Professional Development team monitored

• its learning, training, and development plans, strategies, budgets, activities, and programs;
• attendance at courses, workshops, and other learning, training, and development delivered;
• the cost for internal, external, and language training;
• the average number of training hours per employee group;
• the completion rate of required and official languages training taken; and
• the number of classroom training days.

79. We noted that the team used the results of its performance monitoring to inform its priority setting for the next fiscal year in terms of course offerings. If its strategies fell short of expectations or if priorities were not being realized, it took corrective action for its own programs.

80. Reporting. Reporting shows the results of how plans and strategies address gaps or deficiencies in employee and organizational performance, and account for the resources and budgets used.

81. We found that the Professional Development team communicated the results of its activities through various documents, such as plans, reports, and strategies. The team had planned to conduct a reporting needs project in the 2016–17 and 2017–18 fiscal years to determine whether stakeholders found useful the information that the team provided. However, at the time of our internal audit this project had not yet been done.

82. We found that the Professional Development team had compared its technical course content with that of a major accounting firm. We also noted that the Office had information, which could be used for comparative purposes, on the average number of training days that provincial legislative audit offices provided to their staff. However, we found that the team did not conduct other types of benchmarking against similar-sized organizations, such as the number of resources dedicated to the professional development function or the average cost of training per employee. In our view, financial and non-financial benchmarks could help the team compare resources dedicated to learning, training, and development activities and help it better assess its investment in these activities.

Conclusion

83. We concluded that the Office had an adequate and effective management framework for learning, training, and development to ensure that employees received the training and development they needed, when they needed it, to carry out their duties. However, the Office could make some improvements.