2014 Spring Report of the Auditor General of Canada Chapter 8—Meeting Needs for Key Statistical Data—Statistics Canada

2014 Spring Report of the Auditor General of Canada

Introduction

8.1 As Canada’s national statistical agency, Statistics Canada plays a fundamental role in generating information on Canada’s socio-economic conditions. The Agency’s statistical programs generate approximately 350 data products for use by governments, researchers, planners, academics, and others. These data products cover a wide range of topics intended to help Canadians know and understand their socio-economic situation. Statistics Canada reports to Parliament through the Minister of Industry.

8.2 Statistical data is used to support government decision making at all levels by describing changing events, enabling analysis of policies and programs, and monitoring program outcomes. Some data produced by Statistics Canada is also required by legislation.

8.3 Other important users of statistical data include private sector businesses and non-governmental organizations, which use Statistics Canada data for a range of planning and marketing purposes, and academic researchers who use data in their research. Given the wide range of users who rely on its data, and the lack of an alternative source of comparable statistical information, it is essential that Statistics Canada operate efficiently and effectively to meet the needs of its broad client base.

8.4 Under the Statistics Act, Statistics Canada is required to “collect, compile, analyse, abstract and publish statistical information relating to the commercial, industrial, financial, social, economic and general activities and condition of the people.” Statistics Canada has two main objectives:

• to provide statistical information and analysis about Canada’s economic and social structure, and
• to promote sound statistical standards and practices.

8.5 In support of these objectives, Statistics Canada has developed a structured approach to achieving quality. In 2002, the Agency introduced its Quality Assurance Framework, which defined the elements of quality and described the basic mechanisms to achieve it. This framework was supplemented in 2009 by the document Statistics Canada Quality Guidelines, an updated set of detailed quality guidelines to support program managers.

8.6 The Quality Assurance Framework and the quality guidelines define quality as encompassing six related dimensions: accuracy, timeliness, accessibility, coherence, relevance, and interpretability. The Agency notes that in order to manage quality, it must balance these quality elements against the constraints of financial and human resources, the goodwill of respondents in providing source data, and competing demands for greater quantities and detail of information.

Focus of the audit

8.7 The overall objective of the audit was to determine whether Statistics Canada ensures the quality of key socio-economic information it produces and generates this information efficiently and in response to priority user needs. The audit examined whether

• Statistics Canada assessed the quality of selected key information it generates on Canada’s socio-economic conditions and acted on these assessments;
• Statistics Canada assessed whether resources are used efficiently to produce selected key socio-economic data products and acted on these assessments; and
• Statistics Canada established priorities for existing and potential new socio-economic products based on user needs and acted on these priorities.

8.8 Our examination of quality and the efficient use of resources was focused on four data products that provide important information on Canada’s economic and social conditions: the Consumer Price Index, the Labour Force Survey, the National Household Survey, and the Survey of Employment, Payrolls and Hours.

• The Consumer Price Index measures the rate of change of the prices of a fixed set of approximately 600 goods and services such as food, shelter, and transportation. The Agency tracks and publishes these price changes monthly. The Consumer Price Index is the most widely used indicator of changes in the cost of living in Canada. Statistics Canada spent $13.1 million to produce the Consumer Price Index in the 2012–13 fiscal year.
• The Labour Force Survey is a monthly survey that collects information on the labour market activities and demographic characteristics of working-age Canadians. The data is used to produce the widely reported unemployment rate as well as other standard labour market indicators. Data is currently collected from a sample of 56,000 households, involving approximately 110,000 individuals. The cost of producing this survey in the 2012–13 fiscal year was $20.9 million.
• The National Household Survey was introduced in 2011 to replace the long-form questionnaire that was part of previous Censuses of Population. The voluntary National Household Survey was administered alongside the 2011 Census of Population. A key feature of this survey is that it provides information on a range of issues, including education, labour, income, and housing for small geographic areas and small subpopulations, such as ethnocultural groups, in a particular location. In 2011, approximately 3 million dwellings responded to the National Household Survey. The combined cost of the 2011 Census of Population and the 2011 National Household Survey was $652 million over the seven-year Census cycle, including $22 million in added costs associated with changes to the Census and the introduction of the National Household Survey.
• The Survey of Employment, Payrolls and Hours provides information on total numbers of paid employees, payrolls, hours worked, and job vacancies at the industrial and provincial/territorial levels. It combines data collected from a survey of businesses with administrative tax data acquired from the Canada Revenue Agency. It is conducted monthly, with data released two months after collection. The main users are government departments, but it is also intended for use by the private sector. The cost of producing this survey in the 2012–13 fiscal year was $4.7 million.

Our examination of priority setting for existing and potential new data products focused on several other data products that were either eliminated or significantly modified during the audit period.

8.9 The audit covered the period between 1 April 2009 and 15 November 2013. More details about the audit objective, scope, approach, and criteria are in About the Audit at the end of this chapter.

Observations and Recommendations

Application of data quality processes

8.10 Statistics Canada has a Quality Assurance Framework and quality guidelines to ensure that the Agency’s products are of sufficient quality for their intended uses. We examined whether the Agency met these standards for the four selected data products. Our observations are organized according to the six elements of quality defined by the Agency: accuracy, timeliness, accessibility, coherence, relevance, and interpretability.

Statistics Canada applied its quality assurance processes to obtain accurate information for the selected data products

8.11 The accuracy of statistical information is defined by Statistics Canada as the degree to which the information correctly describes the phenomena it was designed to measure. It is usually characterized in terms of error in statistical estimates. Information that is not sufficiently accurate for its uses may mislead users and lead to poor decisions.

8.12 According to the Agency’s 2009 quality guidelines, for a data product to be accurate, appropriate methods and tools must be used to produce the results. These include sampling methods, measures to encourage accurate responses, follow-up on non-response to surveys, dealing with missing data, and measurement of response rates and sampling error.

8.13 We examined, for a sample of individual data releases for each of the four data products, selected steps conducted to ensure accuracy. These included responsiveness to changes in the population sampled, implementation of sampling plans, training and supervision of data collection staff, and creation and use of quantitative indicators of data quality.

8.14 We found that Statistics Canada completed all of these steps to ensure the accuracy of the sample of data releases we examined. For the National Household Survey, which has been conducted only once to date, the Agency took a number of additional steps to ensure the accuracy of the data.

8.15 As noted earlier, the National Household Survey was introduced in 2011 to replace the former long-form questionnaire previously administered as part of the Census of Population. In 2010, the government instructed Statistics Canada to make responding to this survey voluntary as opposed to mandatory, as had been the case for previous censuses. According to Statistics Canada, the switch from a mandatory to a voluntary survey is a major change in method that has the potential to affect response rates, an important determinant of data quality.

8.16 Statistics Canada took a number of steps to mitigate the risks posed by this change, based on its analysis of the potential impacts of the change in method on data quality. For example, the Agency increased the initial sample size from the 3 million households planned for the former long-form Census to 4.5 million households for the National Household Survey. It also took steps to address the anticipated reduction in response rates by targeting its efforts when following up with households that did not initially complete their questionnaires. This targeting focused on geographic areas known to include relatively high proportions of particular population subgroups, for example, low-income earners.

Statistics Canada ensured the timeliness of the selected data products

8.17 According to Statistics Canada, the timeliness of statistical information refers to the delay between the point in time to which the information pertains, and the date on which the information becomes available. In order to be most useful, statistical information should be released as soon as possible after data collection.

8.18 The Agency’s quality guidelines identify actions that need to be taken to ensure the timeliness of the data: specifically, announcing data release dates in advance, and releasing the data on the planned date. We examined these processes for each of the four data products.

8.19 We found that release dates for all four data products were announced in advance. In the cases of the Consumer Price Index and Labour Force Survey, which are released monthly, the lag between data collection and release is approximately one month, and all releases occurred on the planned dates. The Survey of Employment, Payrolls and Hours is also released monthly, but two months after data collection. This reflects the time required by Statistics Canada to receive administrative tax data from the Canada Revenue Agency and process this data. We found that all releases of the Survey of Employment, Payrolls and Hours during the audit period also occurred on the planned dates. Almost all of the users we interviewed were satisfied with the timeliness of the data from these data products.

8.20 The Agency announced three dates for the release of data on specific topics from the National Household Survey, the first in May 2013 and the last in August 2013. These dates are approximately two years from the May to August 2011 data collection period, similar to the timing of data releases from the 2006 long-form Census. The first two releases of National Household Survey data occurred on the planned dates. The Agency announced that the third release would be postponed by four weeks due to the need to address a data processing error. We determined that this error had no impact on the data from the two earlier National Household Survey releases.

Statistics Canada has made data from the selected products accessible to users

8.21 Statistics Canada defines accessibility of statistical information as the ease with which it can be obtained by the user. This includes the ease with which the existence of information can be discovered, as well as the suitability of the form or medium through which the information can be accessed. If users are to make use of Statistics Canada data products, they must be both aware of them and easily able to obtain the information they need.

8.22 According to the Agency’s 2009 quality guidelines, for a product to be accessible, the Agency should describe the availability of its products, make products available to users, and monitor their use.

8.23 For each of the four data products, we examined selected processes called for by the Agency’s quality guidelines intended to ensure the accessibility of the information they yield. In particular, we examined how the information is made available to users, and the identification and reduction of barriers to use. We found that Statistics Canada makes data from the selected products accessible and is planning further improvements.

8.24 We found that Statistics Canada is moving away from publishing its data in reports toward disseminating it on the Internet. Access to data products is provided in several ways, including online bulletins about new releases, online publications, online databases, and social media. In addition, centres supported by the Agency provide researchers at universities across Canada with direct access to detailed data from Statistics Canada’s population and household surveys. In February 2012, the Agency stopped charging users to access standard data products on the Canadian Socio-Economic Information Management System (CANSIM), the main database used by the Agency to store and disseminate socio-economic information.

8.25 However, users told us that the Statistics Canada website is challenging to navigate. According to website evaluations conducted annually by the Agency, user satisfaction with its website averages 67 percent. The Agency is working to increase user satisfaction by developing a new corporate dissemination model to make its data easier to find.

8.26 Users seeking clarification or additional information can contact Statistics Canada staff directly at no charge by phone or email to enquire about specific data products. The Agency tracks the volume of these enquiries. For example, over the audit period, there were more than 10,000 enquiries regarding each of the Consumer Price Index and National Household Survey. Statistics Canada can also provide users with data products and services that are tailored to more particular needs, for a fee. For example, there were 700 custom requests for data for the Consumer Price Index over the audit period.

Statistics Canada assessed the coherence of the selected data products

8.27 According to Statistics Canada, the coherence of statistical information reflects the degree to which it can be successfully brought together with other statistical information within a broad analytic framework and over time. It reflects the comparability of related data for the same point in time, of the same data for different points in time, and of Canadian and international data. Examination of coherence enables the Agency to identify discrepancies in its data that warrant further attention.

8.28 The Agency’s 2009 quality guidelines on coherence call for the use of standard concepts, classifications, and target populations as well as the use of common methodology across surveys. They also promote the review of survey results, and related external sources to identify and address discrepancies. Coherence does not necessarily imply full numerical consistency between surveys and external sources. We examined these processes for the four selected data products.

8.29 We found that Statistics Canada assessed the coherence of the selected data products. We found that Statistics Canada adheres to international standards, definitions, and classifications for industry and occupation for the Consumer Price Index, the Labour Force Survey, the National Household Survey, and the Survey of Employment, Payrolls and Hours. The Agency also routinely compares data from multiple sources to assess the coherence of its data products. Subject matter advisory committees, such as its Methods and Standards Committee, are also used to ensure the coherence of the data.

8.30 According to the Agency, any significant change in survey method or content can affect the comparability of data over time. While the content of the new National Household Survey is similar to the long-form questionnaire used in the 2006 Census, the shift from a mandatory to a voluntary survey constitutes a significant change in method. We found that the Agency’s assessment of the coherence of the data from the National Household Survey determined that this change reduced the comparability of this data to that from the 2006 Census long-form questionnaire. For example, the Agency determined that estimates of the prevalence of low-income households from the 2011 National Household Survey were not directly comparable to those of previous censuses. According to consultations conducted by Statistics Canada and interviews conducted for this audit, the ability to compare National Household Survey data with the previous Census data is important to users. We found that Statistics Canada has informed users that they should use caution when making any such comparisons.

Improvements are needed to ensure the relevance of selected data products

8.31 According to Statistics Canada, the relevance of statistical information reflects the degree to which it meets the needs of clients, and whether the available information sheds light on the issues of most importance to users.

8.32 According to the Agency’s 2009 quality guidelines, Statistics Canada should describe the needs of key users and address gaps between needs and products. Furthermore, the guidelines indicate that this process is facilitated when relationships are established and maintained with the users of information in both the private and public sectors.

8.33 We examined, for each of the four data products, selected processes called for by the Agency’s quality guidelines that are intended to ensure relevance. Specifically, we examined the documented justification for the data products, the extent of the use of the products, the extent of consultation with users, and the response to the needs for data from small areas and subpopulations.

8.34 Use of the selected data products. The data products we examined are required by legislation. In addition, we found evidence of extensive use of the selected data products by diverse users. For example, the data from the Consumer Price Index is used by the Bank of Canada to define and monitor achievement of its inflation rate target, by commercial banks to monitor economic conditions, and by the former Human Resources and Skills Development Canada (now Employment and Social Development Canada) to calculate the yearly pension payments mandated by the Canada Pension Plan and Old Age Security Act.

8.35 We also found widespread use of the Labour Force Survey. For example, federal, provincial and territorial governments, as well as private sector users such as large banks, use the data to assess economic conditions. Regional rates of unemployment derived from the survey are used by Employment and Social Development Canada to determine the eligibility and duration of Employment Insurance benefits for persons living within a particular Employment Insurance region.

8.36 In addition, the Survey of Employment, Payrolls and Hours provides information to support Statistics Canada’s calculation of Canada’s gross domestic product, which contributes to the creation of an up-to-date portrait of the national and provincial economies in the System of National Accounts. It is also used for other purposes, such as by the Canada Revenue Agency to revise retirement savings limits.

8.37 Consultation with users outside the federal, provincial and territorial governments. According to the Agency’s 2009 quality guidelines, relevance is enhanced when relationships are established and maintained with users of information in both the private and public sectors. Statistics Canada has established performance indicators and targets for its products that reflect their use by both government and the private sector. We examined whether Agency consultations on user needs included the full range of users.

8.38 We found that Statistics Canada uses a variety of mechanisms to obtain user input and feedback on its statistical products, including subject matter advisory committees, federal/provincial/territorial consultative committees, and bilateral relationships with federal departments and agencies, including regular meetings with departments such as Employment and Social Development Canada to discuss emerging data needs. These help to keep the Agency’s programs relevant and responsive to the evolving needs of their users.

8.39 Despite the range of feedback mechanisms used regarding the Consumer Price Index, Labour Force Survey, and Survey of Employment, Payrolls and Hours, we found that the Agency primarily consults with the federal, provincial and territorial governments and less with the private sector, municipalities, and non-governmental organizations. On the other hand, we note that consultations by the Agency regarding the Census of Population and the National Household Survey were extensive and represented a broad range of users, both within and outside of government.

8.40 Our interviews with a sample of users outside the federal government revealed that Statistics Canada has not consulted them on their needs. For example, a municipal user of the Labour Force Survey informed us of their need for employment data by workplace location, but there was no ongoing mechanism to provide feedback to Statistics Canada on these needs. Without ongoing consultation with users, the Agency’s data may become less relevant.

8.41 Recommendation. In order to ensure the continued relevance of its data products, Statistics Canada should obtain, document, and analyze ongoing feedback from the full range of its data users.

The Agency’s response. Agreed. Statistics Canada consults on a regular basis with its stakeholders and users through a range of mechanisms to obtain broad-based feedback across a wide array of program areas. As many Agency programs are primarily designed to meet the needs of federal and provincial policy departments, these consultations have focused on these users. Starting in 2014, Statistics Canada will plan its future consultation activities to ensure that the full range of users—including those from the private sector, municipalities, and non-governmental organizations—are consulted where appropriate, and the results of these consultations are documented and analyzed.

8.42 Consultation with experts. According to the Agency, the relevance of its data products is enhanced when the data needs of the full range of users are understood. One important mechanism used by the Agency to understand these needs is its subject matter advisory committees.

8.43 We examined the composition of the Advisory Committee on Labour and Income Statistics, which is a key external advisory committee relevant to a broad range of statistical programs, including the Labour Force Survey and the Survey of Employment, Payrolls and Hours, to see whether it included the full range of users. We found that 10 of the 13 members of the Committee were from academia, with little representation from the private or non-governmental sectors. We also found that the Agency did not have processes for regular reviews of committee membership or for ensuring turnover in that membership. We noted the long tenure of many advisory committee members.

8.44 The limited representation of the private sector, municipalities, and the non-governmental sector on the Agency’s advisory committees limits their opportunities to advise Statistics Canada of their needs. If the Agency’s advisory committees are to assist it to understand the full range of user needs, their membership should reflect that range.

8.45 Recommendation. To ensure that its subject matter advisory committees support the Agency to better understand user needs, a formal and routine process should be implemented to ensure that the full range of users is represented on them. There should also be turnover in their membership to ensure the Agency is aware of, and responsive to, evolving needs.

The Agency’s response. Agreed. In order to maintain the relevance and effectiveness of its programs, Statistics Canada uses a range of mechanisms to consult widely and frequently with data users and stakeholders across a wide array of program areas. The Agency will formalize its membership renewal process for subject matter advisory committees by January 2015 to ensure that the fullest possible range of users are represented on these committees and that the Agency is formally aware of and responsive to evolving user needs.

8.46 Needs for data from small areas and subpopulations. According to the Agency’s 2009 quality guidelines, Statistics Canada should describe the needs of users, and address gaps between needs and products. For the four audited data products, we examined whether Statistics Canada was meeting the needs of users of data from small geographic areas and subpopulations, such as members of ethnocultural groups. We found that needs for data from small geographic areas and small subpopulations in specific locations are not being met.

8.47 For example, data from the Survey of Employment, Payrolls and Hours, a key source of employment data, is only released at the national, provincial, and territorial levels. This means that municipal users requiring this information, for example, are not able to obtain it from this survey. While Statistics Canada is aware of this limitation, and commenced, in 2009, research to identify ways to compile employment data from small geographic areas, the Agency did not complete its research before external funding for the research ended. As of 2013, the Agency was reconsidering this issue but had not decided whether to resume the research. The broader issue of funding cost-recovered research is discussed in paragraphs 69 to 71.

8.48 One specific example of the limited data from the Survey of Employment, Payrolls and Hours concerns data on job vacancies for small geographic areas. In July 2008, the federal, provincial, and territorial Forum of Labour Market Ministers created an Advisory Panel on Labour Market Information to help determine where improvements could be made to the labour market information system in Canada. The Panel’s 2009 report noted the need for local data on job vacancies by both occupation and location. This information could inform job seekers on which occupations are in demand and where. Policy makers could also use this information to develop and manage policies and programs.

8.49 In response, in 2011 Statistics Canada began to produce new data on job vacancies by adding two questions to the business survey component of the Survey of Employment, Payrolls and Hours. This enabled the Agency to release monthly data on job vacancies by types of workplace for Canada and for each province and territory.

8.50 However, the data on job vacancies from this survey has limitations. For vacancies reported at the national, provincial, and territorial level, it is not possible to determine where in a province or territory these job vacancies are located. For example, reported job vacancies in Alberta could be in Fort McMurray, Grande Prairie, Medicine Hat, or any other community in the province. In addition, the survey questionnaire allows for limited classification of types of workplace, but not types of occupations within those workplaces. For example, job vacancies counted under the “Professional, Scientific, and Technical Services” industry classification could include jobs in advertising, legal services, architecture, and biotechnology research. Users informed us that as a result of these shortcomings, available information on job vacancies is of limited value to them.

8.51 We also examined the availability of National Household Survey data from small areas and subpopulations. The National Household Survey is described by the Agency as a key source of national data that can be analyzed for small areas and subpopulations due to its very large database (occupants of approximately 3 million dwellings). As a result, data from this survey is intended to be available to users who need information for geographic areas with relatively small populations. According to Statistics Canada, this information is of importance to many stakeholders in Canada, including all levels of government, and the private and non-governmental sectors. We note that enquiries regarding National Household Survey data for small geographic areas and subpopulations most often came from municipal officials and the non-governmental and private sectors.

8.52 As anticipated by the Agency, the shift from a mandatory to a voluntary household survey lowered the overall response rate from 94 percent in 2006 to 69 percent in 2011. According to Statistics Canada, higher response rates lead to more reliable data. In order to ensure reliable and usable data, Statistics Canada has limited the release of this data to areas for which sufficiently high response rates were achieved.

8.53 Statistics Canada, therefore, withheld National Household Survey data for 1,128 of 4,567 (25 percent) census subdivisions due to quality concerns. (Data for an additional 686 census subdivisions, or 15 percent, was not released for confidentiality and other reasons.) While many of these census subdivisions were sparsely populated, the most populous of these locations had close to 10,000 residents. As a result of data not being released due to quality concerns, potential users of this data for approximately 25 percent of geographic areas do not have reliable National Household Survey data available for their use. Statistics Canada estimates that reliable national data was unavailable from the National Household Survey for 3 percent of the population, compared with less than 1 percent in the 2006 Census.

8.54 Overall, our interviews with a sample of users of the selected data products outside the federal government confirmed that Agency data is of limited use to them when it is not available for their locations.

8.55 Recommendation. Statistics Canada should assess the feasibility of more fully addressing user needs for data from small areas and subpopulations.

The Agency’s response. Agreed. The Agency recognizes that there is a need for further small area data. In November 2013, Statistics Canada initiated a more extensive dialogue with regional and local governments and agencies, the private sector, and non-governmental organizations on their needs and how they can be met within the constraints of available resources. Due to the high cost of producing small area and small population data from surveys, most surveys are necessarily designed to produce data at the national, provincial, and territorial levels. Statistics Canada continues to explore and use alternative sources to provide small area information, in particular, through an increased use of administrative data. The Agency is further exploring innovative methods for the production of small area estimates, using surveys or combining them with other sources of information in order to enrich the information provided through its ongoing programs. As it does at the end of every Census cycle, the Agency is conducting an extensive evaluation of the 2011 National Household Survey in order to further enhance its methodology and to address data requirements for subprovincial geographic areas and subpopulations for the 2016 Census Program.

Guidance to help users interpret information from the selected data products was not comprehensive and up to date

8.56 Statistics Canada defines interpretability as reflecting the availability of the information necessary to interpret and use Statistics Canada data appropriately. If users are not given access to sufficient information to enable them to correctly interpret statistical information, they may not understand and use it properly. The 2009 quality guidelines call for the Agency to provide information to users on what a data product measures, how the measurement is conducted, and the quality of the resulting data.

8.57 For each of the four data products, we examined whether guidance and related documentation were available to help users understand the information from the data products. This included both information for individual data releases, and more general information such as user guides.

8.58 We found that Statistics Canada makes information available to support users to interpret the data from the four products selected for this audit. The Agency maintains an online repository of information (the Integrated Meta Database) for each of its statistical programs, which provides descriptions of data sources and methodology, definitions of concepts and variables measured, and indicators of data quality. We also observed that the Agency issues more detailed user guides for some data products to help users understand its data.

8.59 Statistics Canada has provided extensive guidance to users of the data from the National Household Survey. However, users of data from small geographic areas and subpopulations told us that, while general guidance provided to date for the National Household Survey data has been useful, it is not always clear to them which data for specific locations or subpopulations is reliable and how it may be interpreted. As a result, potential users may choose not to use the data, and actual users may make errors when using the data.

8.60 We also noted that the user guide for the Survey of Employment, Payrolls and Hours was not updated until four years after the 2009 redesign of the survey. Similarly, the user guide for the Consumer Price Index has not been revised since its initial publication in 1996. Both of these data products have undergone significant changes both prior to and during the audit period. For example, questions on job vacancies were added to the business survey component of the Survey of Employment, Payrolls and Hours in 2011. Without current and complete guidance, existing users could make mistakes when trying to interpret the Agency’s data.

8.61 Recommendation. Statistics Canada should ensure that the guidance it provides for users of its data is comprehensive and up to date, and that it supports the needs of the full range of users.

The Agency’s response. Agreed. The Agency recognizes the need to ensure that guidance it provides to users of its data is as comprehensive and up to date as possible. As its core strategy is to support users with the interpretation of the data, the Agency made a strategic direction to invest in and develop the Integrated Meta Database. To complement the Integrated Meta Database, the Agency will continue to provide, as required and as appropriate, more detailed technical user guides for some of its main ongoing statistical programs. To further support users, the Agency will identify, by 2015, which statistical programs would require a more detailed and updated technical user guide and will ensure that these guides are updated on a more current basis as programs are redesigned.

Efficient use of resources

8.62 Statistics Canada, like other federal departments and agencies, has been subject to requirements to reduce its costs since 2009. This timing coincides with an Agency initiative in 2009 to address quality and cost challenges through improved business structures and processes. A key objective of this initiative is to reduce ongoing operating costs with minimal cuts to statistical and analytical programs. According to the 2009–10 to 2011–12 Corporate Business Plan, the Agency is committed to being an efficient organization that produces information at the lowest possible cost. In search of greater efficiency, Statistics Canada committed to examining every aspect of the processes it uses. We examined whether the Agency generated data from the four data products efficiently over the audit period.

Statistics Canada is making more efficient use of resources

8.63 We found that Statistics Canada has identified a number of initiatives intended to make more efficient use of resources in producing the four data products. For example, it now uses common corporate services such as data collection and processing, and it has identified potential alternate data sources to supplement or replace data it directly collects. It has also made enhancements to information systems and technology, including relying more on the Internet to both collect and disseminate data.

8.64 We noted that from the 2010–11 to the 2012–13 fiscal years, 15 of 37 planned initiatives to improve efficiency had been implemented, resulting in savings of $6.7 million according to the Agency. Implementation of these initiatives to the selected data products is at various stages. For example, in 2012, respondents to the business survey component of the Survey of Employment, Payrolls and Hours were given the option of completing questionnaires online instead of by mail or telephone.

8.65 The remaining 22 efficiency initiatives are scheduled for later implementation. For example, the enhancement of information technology systems to support the Labour Force Survey redesign and the Consumer Price Index enhancement initiative are scheduled for completion after the audit period. While Statistics Canada tracks savings from efforts to improve efficiency at the corporate level, tracking these savings at the level of individual statistical programs would, in our view, enable the Agency to better determine the success of these efforts.

Identification and implementation of priorities

8.66 In order to maintain the relevance of its work, it is important for Statistics Canada to identify and respond to emerging needs for socio-economic data. According to its 2012–13 to 2014–15 Corporate Business Plan, the Agency committed to redirecting resources to emerging data needs that are growing in relative priority. We examined eight statistical programs that were eliminated or significantly modified during the audit period to determine how the Agency assesses and addresses identified gaps, and how it ensure that its approved priorities are implemented.

Statistics Canada has a revised process to identify and implement its priorities

8.67 Statistics Canada introduced a revised process in 2011 to improve long-term planning and to maintain the Agency’s standards for quality and efficiency. This involves an annual six-step process that begins with a review of the strategic planning priorities and culminates in resource allocation for approved projects to begin in the new fiscal year. We found that a significant element of this process is the development of a 10-year plan, which includes investments in business continuity and quality maintenance. Given that the Agency’s work is cyclical, this approach is, in our view, useful for managing strategic investments so that expenditures are planned well in advance, including projections of its long-term resource needs and availabilities. In addition, as part of this planning process, we also found that the Agency is monitoring the implementation of planned priorities through, for example, oversight by committees, the use of project management tools, program performance reviews, internal audit, and evaluation.

8.68 We examined whether Statistics Canada implemented this new planning process in order to respond to emerging needs for socio-economic data. We found that the Agency identifies and responds to both emerging needs and needs for improvements to its existing products. For example, the Agency is currently redesigning the Consumer Price Index and Labour Force Survey. It is also conducting feasibility studies to understand needs for First Nations financial data, and is working with partners to maximize the use of existing data to increase the relevance of the Agency’s data products. However, as noted earlier, the Agency received more than 10,000 enquiries regarding each of the National Household Survey and the Consumer Price Index. We found that these enquiries were not analyzed by the Agency to identify potential improvements to these products. In our view, such analysis could assist Statistics Canada to better understand user needs.

Statistics Canada did not systematically consider the need to continue cost-recovered work

8.69 Statistics Canada undertakes cost-recovery work in response to needs for data identified by other federal departments and levels of government. Cost-recovery revenues have generally contributed approximately $100 million annually to the Agency’s total resources. A large portion of these revenues comes from federal departments to fund specific statistical projects. We examined whether the Agency systematically considers the need to continue previously cost-recovered work.

8.70 We found that while the Agency has continued some previously cost-recovered statistical activities through its base funding, it does not systematically assess whether it should continue cost-recovered work when external funding ceases. Without such an assessment, the Agency cannot be sure that priority needs are being met. For example, the Survey of Older Workers, which was sponsored by the former Human Resources and Skills Development Canada (now Employment and Social Development Canada), was a one-time survey designed to assess the labour market intentions and transitions of older Canadians. The data from this survey was released in December 2009. After work ended on the survey, the Agency did not assess whether it should have continued to fund the survey out of its internal resources. We found that data from this survey was used by clients other than Employment and Social Development Canada (the sponsoring client) and note that issues related to an aging workforce have grown in importance as a result of demographic and labour market changes.

8.71 Recommendation. Statistics Canada should systematically assess whether it should continue conducting cost-recovery work after external funding ceases to ensure that priority data needs are met.

The Agency’s response. Agreed. The Agency’s cost-recovery program exists to provide other organizations with a mechanism to have a statistical program delivered when that program cannot be accommodated within Statistics Canada’s base budget. Starting in April 2014, the Agency will direct program managers to review any newly discontinued cost-recovery program. In cases where these changes are considered to be of ongoing strategic importance for the national statistical systems, proposals should be submitted to the annual planning process for possible funding from the Agency’s base budget. These proposals will be evaluated in light of competing priorities and proposals given available funding.

Conclusion

8.72 We concluded that Statistics Canada applied its quality assurance framework to ensure the quality of the statistical programs we examined. However, improvements are needed to better meet user needs.

8.73 Statistics Canada has identified and implemented a number of approaches resulting in more efficient use of resources.

8.74 Statistics Canada has an adequate process to identify and implement its priorities, including emerging needs. The Agency could improve this process by systematically assessing whether it should continue cost-recovery work when external funding ceases.

About the Audit

The Office of the Auditor General’s responsibility was to conduct an independent examination of selected Statistics Canada data products and processes to provide objective information, advice, and assurance to assist Parliament in its scrutiny of the government’s management of resources and programs.

All of the audit work in this chapter was conducted in accordance with the standards for assurance engagements set out by the Chartered Professional Accountants of Canada (CPA) in the CPA Canada Handbook—Assurance. While the Office adopts these standards as the minimum requirement for our audits, we also draw upon the standards and practices of other disciplines.

As part of our regular audit process, we obtained management’s confirmation that the findings reported in this chapter are factually based.

Objective

To determine whether Statistics Canada ensures the quality of key socio-economic information it produces, and generates this information efficiently and in response to priority user needs.

Scope and approach

The audit work was focused on Statistics Canada, which is the only federal entity mandated to collect, compile, analyze, and publish statistical information relating to the country’s socio-economic conditions. During the examination, the audit team also consulted with selected key users of Statistics Canada data, including Employment and Social Development Canada, Department of Finance Canada, and the Bank of Canada, as well as representatives of the private sector, non-governmental organizations, municipalities, and organized labour.

The audit examined whether

• Statistics Canada assessed the quality (accuracy, timeliness, accessibility, coherence, relevance, and interpretability) of selected key information it generates on Canada’s socio-economic conditions, and acted on these assessments;
• Statistics Canada assessed whether resources are used efficiently to produce selected key socio-economic data products, and acted on these assessments; and
• Statistics Canada established priorities for existing and potential new socio-economic products based on user needs (relevance), and acted on these priorities.

To examine the first two issues, we selected the following four Agency data products for detailed examination in the audit: the Consumer Price Index, Labour Force Survey, National Household Survey, and Survey of Employment, Payrolls and Hours. All of these data products provide important information about Canada’s economic and social conditions. They are also reported at varying frequencies, and are carried out by multiple divisions within the Agency.

Our examination of the three data products for which data are released monthly (the Consumer Price Index; Labour Force Survey; and Survey of Employment, Payrolls and Hours) was based, in part, on the review of five releases. These releases were selected to cover the audit period and different points in each year.

For the third issue, whether the Agency has established and acted on priorities for existing and potential products, we examined the process followed by Statistics Canada to eliminate or significantly modify socio-economic surveys during the audit period. To do so, we examined

• the Consumer Price Index and Labour Force Survey, and the addition of two questions on job vacancies to the business survey component of the Survey of Employment, Payrolls and Hours;
• the inactive Composite Leading Indicator, Survey of Older Workers, and Survey of Earned Doctorates;
• the Survey of Labour and Income Dynamics, the longitudinal component of which was eliminated; and
• the redesign of the Survey of Household Spending in 2010.

At the end of our examination phase, the departmental name of Human Resources and Skills Development Canada changed to Employment and Social Development Canada. This change had no impact on our audit work and findings.

Criteria

Management reviewed and accepted the suitability of the criteria used in the audit.

Period covered by the audit

The audit covered the period between 1 April 2009 and 15 November 2013. Audit work for this chapter was completed on 30 November 2013.

Audit team

Assistant Auditor General: Ronnie Campbell
Principal: John Affleck
Director: Colin Meredith

Irene Andayo-Michalowski
Sebastien Bureau
Jeff Graham

For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free).

Appendix—List of recommendations

The following is a list of recommendations found in Chapter 8. The number in front of the recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses indicate the paragraphs where the topic is discussed.

Definition:

Census subdivision—A geographic “area that is a municipality or an area that is deemed to be equivalent to a municipality for statistical reporting purposes (e.g., as an Indian reserve or an unorganized territory).” The population contained in a census subdivision can range from zero (uninhabited) to more than one million.

Source: Statistics Canada, Census Dictionary (Return)