Introduction

Background

6.1 Many Indigenous people face barriers to sustained employment, such as living in isolated communities and having low levels of education. They experience high unemployment rates and low average earnings, and often lack job stability. In 2007, the unemployment rate for Indigenous people was just under 11%. In comparison, it was just under 6% among non-Indigenous Canadians. By 2017, these unemployment rates had increased to just over 11% and just over 6%, respectively. Many Indigenous people need training and support to build the skills they need to find and keep jobs.

6.2 Employment and Social Development Canada. Employment and Social Development Canada funds programs to help Canadians develop skills for employment and improve their workforce mobility. The Department prioritizes skill development for groups that are underrepresented in the workforce, such as Indigenous people.

6.3 The Aboriginal Skills and Employment Training Strategy and the Skills and Partnership Fund. Since the early 1990s, the Department has funded programs to help Indigenous people (First Nations, Métis, and Inuit) improve their skills and find employment. Successive programs have maintained the same focus. In 2010, the Department launched its two current programs: the Aboriginal Skills and Employment Training Strategy and the Skills and Partnership Fund.

6.4 The Strategy provides funding for the delivery of training and employment support to Indigenous clients to increase their participation in the Canadian labour market and ensure that they are engaged in sustainable and meaningful employment. The Fund is intended to complement the Strategy, but is project driven in an effort to take advantage of emerging economic development opportunities in sectors such as natural resources.

6.5 Despite this difference, the Strategy and the Fund have the same overall objective, and the services eligible under each are the same (Exhibit 6.1). We refer to the Strategy and the Fund collectively as “the programs” throughout this report except where specifically named.

6.6 In the early 1990s, the Department began devolving the design and delivery of labour market programs to Indigenous organizations. Under the current programs, the Department does not provide services directly to clients. Instead, it enters into contribution agreements with Indigenous organizations (called “agreement holders” in this report) for up to five years, funding them to provide services to clients. Agreement holders across the country provide services to First Nations, Métis, and Inuit clients in urban, rural, remote, and northern locations. Many of the current agreement holders also provided services to clients under the previous programs.

6.7 Under the Strategy, agreement holders can deliver services directly to clients or enter into sub-agreements with other Indigenous organizations to deliver services on their behalf. Under the Fund, agreement holders may put in place sub-projects or agreements with third parties to help deliver training and employment support to Indigenous clients.

6.8 The agreement holders determine and provide the specific skill development and training services their clients need to enter and stay in the labour force. While some clients need minimal training, others face multiple challenges and may require several services to prepare them for employment. In addition to occupational skills training, services eligible under the programs include client assessments, employment counselling, essential skills development, and wage subsidies to encourage employers to hire clients.

6.9 The Department expects agreement holders to provide services that are based on market demand for specific types of labour. Agreement holders set their own targets, such as the number of clients they expect to serve and the number they expect will find jobs after receiving services. Under the terms of their contribution agreements, agreement holders are required to report to the Department on the success of their services versus these targets.

6.10 Under the programs, the Department approves funding to the agreement holders and provides guidance, information, and training to support them in delivering services to clients. The Department is responsible for monitoring the agreement holders’ use of program funds, for measuring and reporting program results, and for adjusting the programs as necessary.

6.11 The Department also funds provinces and territories to provide labour market programs to their residents. Indigenous people are eligible for these as well. Some other federal organizations offer similar programs, but Employment and Social Development Canada is the major source of government funding for these types of initiatives.

Focus of the audit

6.12 This audit focused on whether Employment and Social Development Canada managed the Aboriginal Skills and Employment Training Strategy and the Skills and Partnership Fund to increase the number of Indigenous people getting jobs and staying employed. Specifically, we analyzed the Department’s actions to implement, monitor, report on, and improve the programs.

6.13 This audit is important because employment is vital to the well-being of people across Canada, including Indigenous people, who face more barriers to employment than the general population does. Given the federal government’s large investment in these programs, it is important to measure the effect they have on the employment of Indigenous people.

6.14 We did not examine the Department’s financial management of the programs. Nor did we examine the First Nations and Inuit Child Care Initiative, which is delivered by the Strategy’s agreement holders but is a separate program. Finally, we did not audit other labour market programs funded by the Department or by other federal organizations.

6.15 More details about the audit objective, scope, approach, and criteria are in About the Audit at the end of this report.

Findings, Recommendations, and Responses

Program results

Overall message

6.16 Overall, we found that Employment and Social Development Canada did not collect the data or define the performance indicators necessary to demonstrate whether the Aboriginal Skills and Employment Training Strategy and the Skills and Partnership Fund were meeting their common overall objective of increasing the number of Indigenous people who had sustainable and meaningful employment. This is despite the fact that the Department has been delivering programs to support Indigenous employment for almost 30 years.

6.17 We also found that the Department allocated funding to agreement holders under the Strategy primarily on the basis of data from 1996 that did not reflect the current needs of the populations served. Moreover, the Department did not reallocate funding to the individual agreement holders who had been more consistently successful in training clients and helping them get jobs.

6.18 These findings matter because unemployment rates for Indigenous people are markedly higher than for the general Canadian population. The Department has allocated significant funds to the programs. Collecting adequate data and defining performance indicators would allow the Department to determine whether the programs are leading to meaningful and sustainable employment and whether changes are needed.

Employment and Social Development Canada did not collect the necessary data or define performance indicators as needed to determine whether its training programs met their overall objective

6.19 We found that Employment and Social Development Canada could demonstrate that the Aboriginal Skills and Employment Training Strategy and the Skills and Partnership Fund had helped some clients find employment. However, we also found that the Department did not have information about the nature of those jobs (for example, whether they were part-time or full-time). It also did not measure how long clients stayed employed.

6.20 Our analysis supporting this finding presents what we examined and discusses the following topics:

• Performance measurement strategies
• Performance indicators and targets
• Reporting program results

6.21 This finding matters because Indigenous people fare poorly in the labour market compared with other Canadians. Measuring and reporting on whether its programs increase the sustainable and meaningful employment of Indigenous people would allow the Department to demonstrate the programs’ value and make improvements where needed.

6.22 Our recommendation in this area of examination appears at paragraph 6.32.

6.23 What we examined. We examined how the Department measured and reported on the results of the programs. We analyzed the programs’ performance indicators and targets. We also assessed the Department’s external reporting on program results. Finally, we assessed how the Department used performance information to improve the programs and their delivery.

6.24 Performance measurement strategies. The Department developed a performance measurement strategy for each of the programs to measure and report on results. However, we found that the Department did not fully implement these strategies. For example, the Department said it would survey agreement holders annually to assess whether it was adequately supporting them to deliver services under the Strategy. It also said it would prepare internal reports annually to inform senior management about the performance of key aspects of the Strategy, such as efforts to help increase the capacity of agreement holders. However, the Department did not fulfill either of these commitments.

6.25 Performance indicators and targets. The Department’s performance measurement strategies for the programs contained many performance indicators: 30 for the Strategy and 10 for the Fund. The Department considered 3 indicators to be particularly important for both programs and required all agreement holders to measure them. These 3 key indicators were

• the number of clients served,
• the number of clients who found employment after receiving services, and
• the number of clients who returned to school after receiving services.

6.26 We found that the Department had targets for the number of clients who found employment after receiving services (Exhibit 6.2). For example, under the Strategy, the Department’s target was for 14,000 to 16,500 clients to find employment each year. However, we found that the Department did not establish targets for nearly half of the remaining performance indicators under the Strategy, including its third key indicator (the number of clients who returned to school). However, the Department did establish targets for the remaining indicators under the Fund.

6.27 The Department based its employment targets on the results of the Indigenous labour market programs it had in place before 2010. However, we found that it did not revisit the targets to ensure that they were still relevant, given changes in the programs and other factors, such as changing economic conditions, between 2010 and 2017.

6.28 Notably, we found that the Department had not clearly defined its employment indicator. It viewed all employment outcomes as the same, regardless of their nature or duration. For example, it counted any employment obtained after receiving services as a successful outcome, whether the work was short-term, seasonal, part-time, or full-time. A short-term job could lead to sustainable employment eventually. However, because the Department did not clearly define employment, it could not use the information that agreement holders reported about this indicator to measure how successful the programs were in helping clients find sustainable, meaningful employment after receiving services.

6.29 Similarly, although the Department defined what it meant for a client to return to school, its guidance to agreement holders indicated that they could count as successful those clients who were already in school when they began receiving services and who simply needed help making a career choice or gaining work experience. As these clients may have remained in school if they had not received services, the Department’s results for the return-to-school indicator were not helpful in demonstrating the difference the services made. Measuring the results this way might also suggest a higher level of success than the programs actually achieved.

6.30 Finally, we found that the Department did not measure its performance against several key targets it had established. For example, in 2011, the Department committed to measuring how long clients who had received services under the Strategy stayed employed after finding work. However, the Department had never measured how long clients of either program remained employed after receiving services, or whether their employment was full-time or part-time.

6.31 Reporting program results. Given the significant limitations in the Department’s implementation of the performance measurement strategies for the programs, the Department could not meaningfully report on their performance. Although the Department publicly reported employment results for most years for the Strategy and for three years for the Fund, the numbers it reported did not indicate whether clients had found sustainable employment.

6.32 Recommendation. Employment and Social Development Canada should, in collaboration with Indigenous organizations, measure and report on whether the Aboriginal Skills and Employment Training Strategy and the Skills and Partnership Fund increase the number of Indigenous people who have sustainable and meaningful employment. It should do so by fully implementing its performance measurement strategies for the programs, including clearly defining performance indicators and targets.

The Department’s response. Agreed. The 2015 Aboriginal Skills and Employment Training Strategy and Skills and Partnership Fund evaluation demonstrated positive short-term labour market outcomes for program participants, such as increased earnings and employment. Although limited by a short post-implementation assessment period, this work benefited from previous Indigenous program evaluations showing continued increased employment and earnings five years after training.

As part of Canada’s commitment to reconciliation and a renewed relationship with Indigenous peoples, Employment and Social Development Canada is working in close collaboration with Indigenous partners to effectively deliver and transform its Indigenous labour market programming to support the strengthening of Indigenous communities.

The Department recognizes the importance of meaningful reporting on results that effectively captures the impact of supports Indigenous organizations provide to clients. Additional performance indicators will allow Employment and Social Development Canada and Indigenous partners to regularly assess program impacts and the longer-term outcomes of the program.

A new performance measurement strategy with strengthened outcomes, indicators, and clearly defined targets will be developed with Indigenous organizations for the new Indigenous Skills and Employment Training Program and implemented in April 2019. This will include enhanced reporting on post-program results using existing information on employment and income.

The strategy for the Skills and Partnership Fund will include defined performance indicators and targets. This will result in better information for all parties.

The Department did not use the data it collected to improve its programs

6.33 We found that the Department did not use the data it collected from agreement holders to improve its programs as needed to help Indigenous people find sustainable and meaningful employment. Nor did the Department ensure that the data the agreement holders provided on whether clients found jobs after receiving services was complete, or confirm the accuracy of most of this data.

6.34 Our analysis supporting this finding presents what we examined and discusses the following topics:

• Data completeness and accuracy
• Data analysis
• Program evaluation

6.35 This finding matters because the Department requires useful data to demonstrate that its investment in the programs is achieving the intended results, and to make changes to improve the programs.

6.36 Our recommendation in this area of examination appears at paragraph 6.45.

6.37 What we examined. We examined the completeness of the data the Department collected from agreement holders and what the Department did to ensure the data’s accuracy. We also examined whether the Department analyzed the data to improve the programs.

6.38 Data completeness and accuracy. Under contribution agreements, the Department required agreement holders to provide quarterly data on clients, including information on their barriers to employment, what services they received under the programs, and whether they subsequently got a job or went back to school.

6.39 We found that the Department did not make sure that all agreement holders provided data on the results their clients had achieved after receiving services. Some agreement holders consistently failed to report their results. Since 2011, four of the agreement holders had reported less than 50% of their results to the Department. Moreover, our analysis of the data indicated that the Department did not know whether 22% of all clients who had received services under the programs since 2010 actually found a job or went back to school.

6.40 We also found that, although the Department had identified problems with data accuracy in the past, it did not confirm the accuracy of most of the data the agreement holders provided about whether clients found jobs. While the Department used Employment Insurance data to verify whether clients were employed, it was able to do so for only about 10% of the programs’ clients. Consequently, the Department had no way of confirming whether most of the programs’ clients were actually successful in finding jobs.

6.41 Data analysis. We also found that the Department did not analyze program data to identify trends, problems, or good practices that could guide agreement holders to improve their services and results. This represents a lost opportunity for the Department.

6.42 Therefore, we analyzed the data to determine whether it contained information that would be useful in managing the program. We identified several issues that, in our view, merit further analysis by the Department. For example, we found that the wage subsidy^{Definition i} was the most expensive service on average, averaging more than $7,000 per use. Between the 2010–11 and 2016–17 fiscal years, the Department spent approximately $130 million on this service. We also found that more than 900 clients received this service more than once. The level of funding for this service and its repeated use by some clients raises questions about whether clients who received this service continued to be employed after the wage subsidy ended, or were more likely to find future employment elsewhere compared with those who did not receive the subsidy. Without knowing this, Employment and Social Development Canada cannot be sure whether the programs are helping clients find sustainable employment or only subsidizing employers without benefiting clients over the longer term.

6.43 Our analysis also uncovered other areas that, in our view, the Department should have investigated:

• Since the 2010–11 fiscal year, about 35,000 clients (16%) received five or more services. This might not necessarily be problematic, as certain clients may face multiple barriers to employment and require multiple services. However, without measuring the results of specific services, the Department could not tell whether these clients were making progress in finding and maintaining employment.
• Clients were more likely to have successful outcomes after receiving certain services. We found that, of the more than 12,000 clients who received apprenticeship training since the 2010–11 fiscal year, 54% subsequently found employment, although the Department had no data to indicate how long the employment lasted. Almost 26,000 clients received academic upgrading services in order to obtain the high school credits or academic qualifications needed to pursue further post-secondary instruction or training. However, only 28% actually pursued further education or training. This suggests that the Department needs to further analyze the relative usefulness of specific services under the programs to determine whether certain ones need to be emphasized.
• In a given year, the proportion of Strategy clients who found employment after receiving services ranged from 1% for some agreement holders to 100% for others. Even though the Department did not confirm most employment outcomes, this suggests that certain agreement holders could benefit from additional support from the Department, while others may have best practices they could share.

This type of analysis could have helped the Department determine whether the services funded by its programs were in fact working and suggested areas where additional analyses were warranted.

6.44 Program evaluation. Treasury Board policy requires the Department to evaluate the programs every five years. The Department reported its evaluation of the Strategy and the Fund in 2015. However, because of the lack of data, the Department used some results from its previous Indigenous training programs to evaluate the results of the current programs. This means that the results of the 2015 evaluation were not a good measure of the current programs’ success in helping clients find sustainable employment. Given what we have found about data verification and completeness, the next evaluation of the programs (scheduled to be completed in 2020) is at risk of having similar problems. The Department has been delivering similar Indigenous labour market programs since the 1990s. Up to now, it has missed the opportunity to collect and analyze the data required to determine whether its programs are achieving their overall objective.

6.45 Recommendation. Employment and Social Development Canada should ensure that it has the information it needs to improve the Aboriginal Skills and Employment Training Strategy and the Skills and Partnership Fund, where necessary, by working in collaboration with agreement holders to identify, collect, confirm, and analyze program data.

The Department’s response. Agreed. Employment and Social Development Canada is committed to producing analysis of medium- to long-term labour market outcomes for Indigenous employment programs, and will continue to work with Indigenous partners to enhance data collection and analysis to guide policy directions and program decisions. Recent exploratory analysis using existing program data indicated positive results for Aboriginal Skills and Employment Training Strategy participants in terms of employment, earnings, and labour market attachment, and demonstrated the capability to replicate this analysis.

The 2015 Aboriginal Skills and Employment Training Strategy and Skills and Partnership Fund evaluation reported that program data can effectively support the analysis of sustainable, long-term outcomes for program participants. Employment and Social Development Canada will use the most recent program data to further assess long-term impacts on participants and support the design and delivery of the new Indigenous Skills and Employment Training Program. The Department will continue to work in collaboration with Indigenous organizations to collect, confirm, and analyze relevant data to improve the program.

As part of the Performance Measurement Strategy for the new Indigenous Skills and Employment Training Program, the Department will work with partners to ensure that relevant and timely data is captured to support reporting on outcomes and impacts for Indigenous labour market programming. This enhanced approach will be implemented by April 2019.

Improvements resulting from this new approach will be applied to the Skills and Partnership Fund as appropriate.

The Department did not allocate Aboriginal Skills and Employment Training Strategy funding on the basis of current needs

6.46 We found that Employment and Social Development Canada allocated funding to agreement holders under the Aboriginal Skills and Employment Training Strategy on the basis of 1996 population and socio-economic data that did not reflect the current needs of the populations served. Also, the Department had not updated the formula it used to allocate funding under the Strategy since the formula was established in 1999. We also found that the Department did not consider individual agreement holders’ past performances when it allocated funding, as a means of redistributing funds to agreement holders that had demonstrated the capacity to achieve better results.

6.47 Our analysis supporting this finding presents what we examined and discusses the following topics:

• Funding formula for the Strategy
• Duplication among programs

6.48 This finding matters because, if the information used in the funding formula for the Strategy is outdated, the Department’s funding allocations will not reflect regional needs, which could limit agreement holders’ ability to succeed.

6.49 Our recommendations in this area of examination appear at paragraphs 6.56 and 6.59.

6.50 What we examined. We examined whether the Department updated the funding formula for the Strategy, as it had committed to doing in 2009. We also assessed how the Department used performance information to adjust funding allocations to improve program results under the Strategy. We examined whether the Department assessed the extent of overlap between its Indigenous labour market programs and programs offered by other federal, provincial, and territorial organizations.

6.51 Funding formula for the Strategy. When the Department introduced the Strategy in 2010, it allocated funds to agreement holders in accordance with the 1999 funding formula that it had used for the program that preceded the Strategy. It referred to this formula as the National Aboriginal Resource Allocation Model. The 1999 formula was needs-based and directed more resources to regions where there were more Indigenous people with greater training needs. The formula used population and socio-economic data from the 1996 Census.

6.52 In 2012, the Department identified several reasons why it should revisit the formula, including

• changes in the Indigenous population since 1996,
• a need to re-examine the variables used in the formula,
• a need to direct funding in a way that was relevant to the Strategy’s priorities, and
• a need to simplify and increase the transparency of the funding calculations.

The Department further stated that, without an up-to-date formula, it lacked a credible basis for its funding decisions for the Strategy.

6.53 We found that the Department tried to update its funding formula in 2014, but could not reach consensus with agreement holders on what changes to make. Any changes to the formula would have increased funding for some agreement holders and decreased funding for others.

6.54 This means that the Department allocated funding to agreement holders as recently as the 2017–18 fiscal year on the basis of data that was more than 20 years old. According to the Department, some regions received a higher percentage of the program funds than current information would support, while others received a lower percentage. For example, the Department calculated that, had it based its funding allocations on information from the 2011 National Household Survey, it would have increased funding in Ontario by 6% and decreased funding in British Columbia and Prince Edward Island by 7% and 47%, respectively.

6.55 The Department also had the authority to adjust funding to agreement holders in response to their past performances. However, we found that the Department did not consider the results achieved by individual agreement holders when deciding how to allocate Strategy funding, either initially in 2010 or during renewals. The funding amounts for agreement holders stayed largely the same year after year, regardless of the results they achieved or of changing demand for workers in the areas they served.

6.56 Recommendation. Employment and Social Development Canada should allocate funding under the Aboriginal Skills and Employment Training Strategy in accordance with current needs and the capacities of individual agreement holders to achieve results. This should include

• updating the funding formula,
• using up-to-date population and socio-economic data, and
• considering past performance of individual agreement holders.

The Department’s response. Agreed. Budget 2018 announced new, incremental funding to reduce the employment and skills gap between Indigenous and non-Indigenous peoples, based on updated socio-economic and demographic data.

As part of the co-development of the new Indigenous Skills and Employment Training Program, the Department will engage with Indigenous partners to determine the approach to allocating new funding, to be implemented by April 2019.

This will be done using a distinctions-based approach, by First Nations, Métis, Inuit, and urban/non-affiliated Indigenous peoples, facilitating an increased role for Indigenous leadership. Funding will also respond to partner requests for building greater capacity for service delivery providers.

Finally, funding allocation will take into account relevant considerations to ensure effectiveness and fairness, including that service providers serve different populations with varying needs, both in terms of the clientele served and differing labour market conditions.

6.57 Duplication among programs. In 2016, the Department identified a risk that services offered under the Strategy and the Fund could overlap with each other, as well as with services under programs offered by other federal, provincial, and territorial organizations that support training for Indigenous people. This could lead to a duplication of services, confusion among agreement holders, and inefficient use of the Department’s resources.

6.58 We found that the Department had not formally assessed the extent to which the Strategy and the Fund duplicated services. This is problematic because the services that agreement holders provided under the Strategy and the Fund were identical and, in some cases, agreement holders under the two programs operated in the same region. However, we found that the Department took initial steps to address the risk of service overlap with other federal departments, provinces, and territories by liaising with them to promote coordination for specific projects. The Department had also begun an analysis of the labour market programs offered by other jurisdictions (to better understand possible risks), but had not yet completed this work.

6.59 Recommendation. Employment and Social Development Canada should conduct the analysis necessary to determine whether its Indigenous labour market programs overlap with each other. The Department should also complete work to determine whether its labour market programs overlap with those offered by other federal, provincial, and territorial organizations. The Department should also take the necessary steps to address any identified areas of overlap.

The Department’s response. Agreed. Employment and Social Development Canada acknowledges that the Aboriginal Skills and Employment Training Strategy and the Skills and Partnership Fund both seek to provide skills development and job training opportunities to Indigenous people. These programs were purposely designed to be complementary, to support Indigenous communities to build human capital to respond to project-based and demand-driven opportunities.

On an ongoing basis, including through the work of established federal, provincial, and territorial forums, the Department monitors labour market programs to ensure complementarity and avoid duplication across programming delivered by other federal departments or by provinces and territories.

As part of its ongoing efforts to ensure complementarity across programming, the Department will continue to undertake analysis of existing programs to determine if there are any areas of duplication and to identify opportunities for improvements before the implementation of the new Indigenous Skills and Employment Training Program in April 2019.

Program management

Overall message

6.60 Overall, we found that Employment and Social Development Canada supported agreement holders by providing them with guidance, information sessions, and administrative direction. It also worked with agreement holders to reduce their administrative burden. However, the Department did not provide agreement holders with sufficient labour market information to help them determine which services they should provide to clients to help them prepare for and find available jobs.

6.61 We also found that the Department did not consistently monitor agreement holders to know whether they fulfilled their obligations under contribution agreements. In particular, the Department did not ensure that agreement holders adequately monitored the other Indigenous organizations with which they had sub-agreements to provide services to clients on their behalf.

6.62 These findings matter because agreement holders need good labour market information to help them determine which services to provide to clients. Moreover, consistent monitoring of agreement holders can help the Department understand how it could further support agreement holders to deliver services.

The Department supported agreement holders to deliver services but did not provide sufficient labour market information

6.63 We found that Employment and Social Development Canada supported agreement holders through regular discussions, information sessions, and help with specific administrative issues. We also found that the Department reduced the burden on agreement holders by lightening some of the Aboriginal Skills and Employment Training Strategy’s administrative requirements. However, we found that the Department did not have the labour market information relevant to Indigenous people that the agreement holders needed, and that its current plan to get that information would take several years to produce results.

6.64 Our analysis supporting this finding presents what we examined and discusses the following topics:

• Guidance to agreement holders
• Labour market information
• Administrative burden

6.65 This finding matters because providing clear guidance to agreement holders may help them deliver services to better meet the programs’ overall objective. Agreement holders need current and detailed labour market information to offer training that prepares their clients for jobs that are in demand. Support from the Department is particularly important for organizations with limited capacity.

6.66 Our recommendations in this area of examination appear at paragraphs 6.73 and 6.76.

6.67 What we examined. We examined whether the Department supported agreement holders by providing guidance, administrative direction, and labour market information.

6.68 Guidance to agreement holders. We found that the Department supported agreement holders by providing formal guidance on the programs and templates to help them meet their reporting requirements. We also found that Department officials worked with agreement holders to help them develop business and operational plans and to help them ensure that funding proposals contained all the required information. Department officials told us that the nature and amount of its support depended on agreement holders’ needs.

6.69 We also found that the Department provided support to agreement holders by holding information sessions on topics such as revised reporting requirements and best practices in program implementation. However, the Department offered these sessions intermittently, and the capacity limitations of some agreement holders suggested that additional sessions would have been helpful to them.

6.70 Labour market information. The Department has known since 2010 that it did not have sufficient labour market information about Indigenous communities. In particular, the Department identified a lack of up-to-date, on-reserve labour market information to support program design, service delivery, and decision making.

6.71 In 2011, the Department began providing agreement holders with national and regional economic information, including unemployment rates and information related to various employment sectors. However, in our opinion, most of this information was too general to be of use to individual agreement holders. Department officials told us that in mid-2017, the Department began providing agreement holders with information from its Temporary Foreign Worker Program about sectors where workers were in demand, so that agreement holders could consider targeting their services to these areas; the Department also included a section for Indigenous job seekers in the federal Job Bank website.

6.72 In the 2016–17 fiscal year, the Department also launched a six-year, $12 million pilot project to train Indigenous organizations to collect data and develop a skills inventory on a small portion of the on-reserve population. The Department told us that as of March 2018, 13 agreement holders under the Strategy had volunteered to participate in the pilot. The project’s timelines mean that it will not be completed until 2022. As a result, it will not produce significant change in the information available for several years.

6.73 Recommendation. Employment and Social Development Canada, in collaboration with Indigenous organizations, should identify and provide labour market information that individual agreement holders need to align their services with demand in the labour market for their particular regions.

The Department’s response. Agreed. The Department recognizes the importance of providing timely, detailed labour market information to Indigenous organizations so that they can design and deliver programming according to their community needs.

One way the Department will improve labour market information sharing with Indigenous partners is by leveraging data from the Job Bank to support training and job placements. The Department is working to link jobs available on the Job Bank directly with Indigenous organizations. This work will be implemented by April 2019.

The Department will further improve available labour market information through an extended survey pilot, beginning in April 2018, to test processes and tools to improve the level of detail and frequency of labour market information for on-reserve communities. The annual results of the survey and skills inventory will be used to inform program design and better tailor client interventions to labour market demand.

6.74 Administrative burden. We noted that the Department did not always make use of the information that it required agreement holders to provide. For example, program officers explained that they reviewed the annual reports that all agreement holders were required to submit under the Strategy, but they were not required to compile this information to find out how the program was working at the regional or national level. In 2011, the Department committed to using the information in these annual reports to assess what was working well in the partnerships that agreement holders formed to deliver services. However, the Department never did this.

6.75 During consultations for the renewal of the Strategy, various Indigenous organizations expressed a desire to reduce the administrative burden imposed by the program. In June 2016, the Department established the Program Delivery Improvement Working Group with members from the Department’s headquarters, regional offices, and agreement holder organizations. The working group’s mandate was to review issues related to the administration and delivery of the Strategy and develop recommendations that the Department could implement for the 2017–18 fiscal year, the third extension year of the program. For that year, the Department reduced the detail required from agreement holders in the plans they submitted to obtain funding, eliminated the requirement for them to submit annual reports, and eliminated the Department’s mid-year monitoring activity.

6.76 Recommendation. Employment and Social Development Canada, in collaboration with Indigenous organizations, should assess how the changes it has made to reporting requirements have affected the agreement holders’ administrative burden and the Department’s ability to manage the programs. It should also make any required adjustments.

The Department’s response. Agreed. Employment and Social Development Canada has already taken concrete steps in consultation with Indigenous organizations to reduce administrative burden. The Department will continue this collaborative effort and will assess the impact of previous changes in order to further improve the delivery of the programs.

As part of co-development of implementation of the new Indigenous Skills and Employment Training Program, Employment and Social Development Canada will work with Indigenous partners to assess further changes to reporting requirements to streamline activity and financial reporting as part of new funding agreements to be implemented in April 2019.

Applicable findings will be considered for the Skills and Partnership Fund.

The Department did not consistently monitor agreement holders

6.77 We found that while Employment and Social Development Canada established a system to monitor agreement holders to ensure they fulfilled their obligations under their contribution agreements, the Department did not consistently meet these monitoring requirements. Moreover, the Department did not use the information gathered during its monitoring activities to determine how it could improve the programs; nor did it adequately address the risks related to the agreement holders’ use of sub-agreements with other organizations to deliver some services on their behalf.

6.78 Our analysis supporting this finding presents what we examined and discusses the following topics:

• Monitoring of agreement holders
• Oversight of sub-agreement holders
• Program officer training

6.79 This finding matters because without sufficient monitoring, the Department cannot acquire the information it needs to adjust the programs or help organizations best serve their clients.

6.80 Our recommendations in this area of examination appear at paragraphs 6.86 and 6.94.

6.81 What we examined. We examined the requirements that the Department established for its monitoring of agreement holders and how it ensured these were met. This included looking at how Department officials determined whether agreement holders were adequately monitoring their sub-agreement holders. We also examined how officials communicated the results of monitoring activities to senior management and the actions taken in response. Some of our findings relate specifically to the examination work we did in the Department’s Western and Territories Region, which had the greatest number of agreement holders.

6.82 Monitoring of agreement holders. The Department’s program officers were required to conduct three types of monitoring activities:

• Financial monitoring—verifies that an agreement holder’s financial records are complete and accurately reflect the terms and conditions of the agreement.
• Activity monitoring—verifies that an agreement holder’s activities are proceeding as stated in the agreement, are addressing any potential difficulties, and are supporting the agreement holder in achieving the results it expects.
• Results monitoring—verifies whether an agreement holder is achieving the desired results from the funding provided by the Department. This type of monitoring looks at the bigger picture and verifies whether the agreement holder is on track to achieve the overall expected results and contribute to the program’s objective. Program officers identify any issues that may affect overall progress and success, such as timelines, resources, activities, and milestones.

The Department assessed the risk related to each contribution agreement to determine the frequency of monitoring required.

6.83 We found that the Department did not always complete the three monitoring activities within the required timelines. As a result, the Department missed opportunities to help agreement holders improve their services in a timely manner. According to the Department’s tracking, its compliance with conducting financial and activity monitoring had improved, but neither program was fully compliant with the monitoring requirements (Exhibit 6.3).

6.84 We also found that the Department did not track whether program officers conducted results-monitoring activities on time. In our opinion, this was problematic, as one purpose of results monitoring was to help the Department determine whether agreement holders were contributing to the programs’ overall objective.

6.85 Furthermore, we found that the Department did not use the information it gathered through monitoring agreement holders to explore ways to improve program delivery or to identify systemic issues requiring attention and best practices to share with Department officials or agreement holders. Yet this was one of the Department’s responsibilities under the programs.

6.86 Recommendation. Employment and Social Development Canada should complete all required monitoring of agreement holders within the required timelines and use the resulting information to adjust the contribution agreements and overall programs as necessary.

The Department’s response. Agreed. The Department currently carries out significant monitoring activities in order to manage agreements and support Indigenous organizations. Information currently gathered from these monitoring activities is used to adjust projects as required in order to achieve the expected results.

The monitoring results will also be used by the Department, in consultation with Indigenous partners, to adjust the program as well as the risk management and monitoring framework.

An updated risk management model will be implemented by April 2019 for the new Indigenous Skills and Employment Training Program. The updated model will consider appropriate risk mitigation strategies, including monitoring, client capacity building, and internal resources.

Applicable findings will be considered for the Skills and Partnership Fund.

6.87 Oversight of sub-agreement holders. Many of the agreement holders used sub-agreements with other organizations to deliver services under the Aboriginal Skills and Employment Training Strategy. Some offered services in remote locations where the agreement holders did not have a presence. As with the agreement holders, sub-agreement holders were required to be Indigenous organizations. As of the 2017–18 fiscal year, 34 agreement holders had entered into a total of 347 sub-agreements. Of these 34 agreement holders, 15 had 10 or more sub-agreements each.

6.88 The Department determined that insufficient monitoring of sub-agreements by program officers was a problem. The Department required program officers to review the clauses in the draft sub-agreements, to keep a signed copy of all sub-agreements on file, to visit each sub-agreement holder at least once, and to ensure that agreement holders were adequately monitoring their sub-agreement holders. However, as of June 2016, the Department’s tracking indicated that only 48% of sub-agreements for delivering services under the Strategy were on file. The Department has not since carried out any additional tracking to determine whether this has improved.

6.89 We examined the Department’s tracking of site visits by program officers to sub-agreement holders in its Western and Territories Region. Regional officials tracked whether the required site visits had occurred. By September 2015, the Department tracked that program officers had visited almost all of the sub-agreement holders. The officials confirmed they had not tracked site visits to sub-agreement holders since.

6.90 We found that the Department had no mechanism by which to learn whether agreement holders had entered into new sub-agreements, whether agreement holders had effectively monitored their sub-agreement holders, or whether program officers had fulfilled all of their related responsibilities, such as visiting any new sub-agreement holders. As a result, the Department did not know whether it was adequately addressing the risks related to the use of sub-agreements, such as insufficient monitoring by the agreement holder.

6.91 We also found that the Department provided a checklist to program officers to help them ensure that sub-agreements included the necessary clauses. Programs officers were to use this checklist to ensure that sub-agreements were consistent with program requirements. However, we noted that the Department had not established a way to find out whether this checklist was consistently used or whether agreement holders addressed any deficiencies noted by program officers.

6.92 These findings are particularly concerning given that some agreement holders with documented capacity challenges also had sub-agreements with other Indigenous organizations to administer. However, under the Department’s risk management approach, there was no requirement for Department staff to visit sub-agreement holders more than once in cases where capacity issues had been identified.

6.93 Finally, we noted that the contribution agreements gave the Department the authority to audit the sub-agreement holders. We found that the Department had not conducted any audits of sub-agreement holders in the Western and Territories Region. Furthermore, Department officials indicated that the Department did not track the monitoring or auditing of sub-agreements.

6.94 Recommendation. Employment and Social Development Canada should more fully assess the risks related to sub-agreement holders, identify a strategy to address any identified risks, and implement the required changes.

The Department’s response. Agreed. Employment and Social Development Canada collaborates with Indigenous organizations in the design and delivery of labour market programs based on the understanding that they are best placed to meet the unique needs of their communities and clients. Supporting Indigenous organizations in developing their capacity to manage the agreements, including sub-agreements, is a priority.

Employment and Social Development Canada recognizes the importance of assessing risk factors, including the use of sub-agreements, and of putting in place supports and mitigation strategies to assist Indigenous organizations in achieving results. Monitoring is undertaken in accordance with the current Aboriginal Skills and Employment Training Strategy Program agreement requirements.

Joint work is under way with Indigenous organizations to understand the challenges associated with the use of sub-agreements and to explore roles and accountabilities in their management. The results from this work will inform the development of a revised risk model for the new Indigenous Skills and Employment Training Program that will ensure clear accountabilities for both Employment and Social Development Canada and Indigenous organizations by April 2019.

The revised model will consider the capacity of Indigenous organizations to monitor sub-agreements. Where necessary, additional support measures will be deployed for those Indigenous organizations requiring monitoring assistance, strengthening the capacity of the Indigenous organizations to achieve success.

Applicable findings will be considered for the Skills and Partnership Fund.

6.95 Program officer training. According to the Department, the Strategy and the Fund were complex programs. Department program officers were required to administer high-dollar-value, multi-year contribution agreements that covered various types of services. They also had to work closely with the agreement holders, some of which had capacity challenges and were in remote locations. The Indigenous communities they worked with were also culturally diverse and had varying approaches to education and employment.

6.96 We found gaps in the Department’s training for program officers who delivered the programs. For example, while there was mandatory training on grants and contributions as of September 2016, the Department did not address some of the specific requirements related to the Strategy. The Department made training available to program officers on how to manage relationships with agreement holders and monitor sub-agreements, but this training was limited, and it was not mandatory. Addressing these gaps could help program officers monitor agreement holders more effectively.

Conclusion

6.97 We concluded that Employment and Social Development Canada’s management of the Aboriginal Skills and Employment Training Strategy and the Skills and Partnership Fund was not sufficient to demonstrate that these programs increased the number of Indigenous people getting jobs and staying employed.