Introduction

Background

5.1 Canada Border Services Agency. The Canada Border Services Agency (the Agency) plays a key role in Canada’s security and prosperity by managing the access of people and goods to and from Canada. The Agency administers more than 90 acts, regulations, and international agreements on behalf of other federal departments and agencies, the provinces, and the territories. It carries out these responsibilities with a workforce of around 13,000 employees, including uniformed officers who provide services at around 1,200 locations across Canada. The Agency has an operating budget of over $1.7 billion. In the 2013–14 fiscal year, the Agency admitted into the country close to 100 million travellers, cleared more than 14 million commercial shipments, made more than 9,000 drug seizures, and removed almost 14,000 failed refugee claimants and other inadmissible persons from Canada. These activities resulted in the collection of $26.9 billion or approximately 10 percent of Government of Canada revenues.

5.2 Information technology investments. Information technology (IT) plays a key part in the Agency’s ability to achieve its strategic objectives and mandate of ensuring border security, providing client service, and achieving operational efficiencies. The Agency’s Information, Science and Technology Branch manages the Agency’s portfolio of IT investments to deliver IT products and services that support the management of Canada’s borders. The Branch’s responsibilities include information management, IT infrastructure and solutions, planning and portfolio management, and science and engineering services.

5.3 The Canada Border Services Agency, like other large federal departments and agencies, relies on IT to fulfill its mandated responsibilities. The demand for its services has increased, with more people and goods entering Canada every year. The Agency has stated that this requires 24/7 services that are reliable, efficient, and cost-effective.

Focus of the audit

5.4 This audit focused on assessing whether the Canada Border Services Agency has the corporate and management practices in place to enable the delivery of information technology (IT) investments that align with and support its strategic corporate objectives. As part of the audit, we consulted the following federal government departments and agencies:

• Canadian Food Inspection Agency,
• Citizenship and Immigration Canada,
• Environment Canada,
• Health Canada,
• Public Health Agency of Canada, and
• Royal Canadian Mounted Police.

5.5 In these consultations, the departments and agencies provided information on their collaboration with the Canada Border Services Agency—specifically, in the areas of governance, monitoring project performance, and achieving expected project benefits.

5.6 More details about the audit objective, scope, approach, and criteria are in About the Audit at the end of this report.

Findings, Recommendations, and Responses

Managing information technology investments

5.7 Overall, we found that the Canada Border Services Agency (the Agency) has had significant challenges in managing its information technology (IT) portfolio in a way that ensured it could deliver IT projects that meet requirements and deliver expected benefits. In December 2013, the Agency put in place a new Project Portfolio Management Framework to strengthen its management of IT investments. We found that the framework was comprehensive. However, in our review of five projects against the new framework, we found that the Agency did not fully put it into practice, which resulted in several issues. One was that senior committees responsible for overseeing the IT portfolio did not have complete and accurate information to ensure that projects were being managed to meet each stage of approval, meet delivery requirements, and align with and support the Agency’s objectives. In addition, projects often lacked clear requirements, had no defined and measurable benefits, or had poorly stated benefits. Projects also experienced a number of issues, including duplication of effort and delays.

5.8 This is important because information technology plays a key role in the Agency’s ability to achieve its strategic objectives and mandate of ensuring border security. Without access to complete, reliable project information with clear business requirements, the Agency is restricted in how efficiently and effectively it can manage the portfolio of projects in a manner that will help the Agency achieve its desired outcomes and benefits of security, service, and savings.

5.9 The Canada Border Services Agency has an IT portfolio totalling more than $1 billion, mostly consisting of two major programs. The Border Modernization initiative aims to modernize business processes, particularly IT, which includes eight projects, budgeted at $733 million. The Beyond the Border Action Plan was established in 2011 by the governments of Canada and the United States. The goal of the Beyond the Border Action Plan is to enhance border security while expediting legitimate cross-border trade and travel. The Canada Border Services Agency is one of many Canadian departments and agencies working to implement the Plan. The Plan includes 32 government-wide initiatives to be completed between 2011 and 2015. Of those initiatives, the Agency has a leading role in 10, with a budget of $245 million.

5.10 In December 2013, the Agency implemented the Project Portfolio Management Framework to better manage the large portfolio of IT investments. This framework was developed as a result of an external review. The Agency also centralized the management of all projects, including IT, under one branch: the Information, Science and Technology Branch. The Project Portfolio Management Framework outlines the Agency’s processes and controls for managing projects within four portfolios: Traveller, Commercial, Corporate, and Common. The framework focuses on undertaking the right projects at the right time and overseeing them to maximize service delivery and benefits at the portfolio level. The framework lists the following important inputs for prioritizing projects: Agency priorities, investment plan, risk profile, and Government of Canada priorities.

5.11 In addition, the Agency has a Project Management Framework, which it also updated in December 2013, to strengthen the management of individual projects. All projects must adhere to the updated framework.

The Agency designed a strong Project Portfolio Management Framework but has not fully put it into practice

5.12 We found that the Agency has designed a strong Project Portfolio Management Framework but has not fully put it into practice. The framework sets out a governance structure that clearly outlines the responsibilities of senior committees in managing the Agency’s information technology investments, including the need to consider key strategic inputs, monthly project reports, and project benefits and outcomes. However, we found that, until the summer of 2014, senior committees responsible for overseeing the IT portfolio were making decisions about the portfolio without all the necessary information. These committees needed to fully demonstrate that project requirements and conditions were met at each stage of approval, and that the projects aligned with and supported the Agency’s strategic objectives.

5.13 Our analysis supporting this finding presents what we examined and discusses

• governance for IT investments,
• the Agency Investment Plan and Annual IT Plan,
• enterprise architecture, and
• the IT portfolio risk profile.

5.14 This finding matters because information technology plays a key role in the Agency’s ability to achieve its strategic objectives and mandate of ensuring border security, providing client service, and achieving operational efficiencies. The Treasury Board Policy on the Management of Projects also states that appropriate systems, processes, and controls for managing projects be in place at an Agency level. This is to support the overall achievement of project and program outcomes with an acceptable level of risk and at an acceptable cost.

5.15 Our recommendation in this area of examination appears at paragraph 5.30.

5.16 What we examined. We examined whether the Canada Border Services Agency has the management practices in place to align its portfolio of IT investments with its strategic corporate objectives. In particular, we looked at portfolio governance, the Agency Investment Plan and Annual IT Plan, the enterprise architecture, and the management of IT portfolio risk.

5.17 Governance for IT investments. A governance structure is important in a portfolio management framework because it is the mechanism by which the Agency assures itself that project portfolios are aligned with Agency objectives, interdependencies are managed, and stakeholder requirements are integrated and coordinated. The governance bodies provide direction for projects within the IT portfolio by setting priorities, making decisions, monitoring performance, and managing financial and human resources.

5.18 At the Agency, the Transformation, Innovation and Project Portfolio Committee is responsible for project portfolio oversight and alignment with Agency priorities and the assessment of the project portfolios against the Agency’s strategic objectives and resource capacity.

5.19 We found that the Transformation, Innovation and Project Portfolio Committee had not been given the detail it needed to fully exercise those responsibilities. For example, the Committee is responsible for ensuring that sub-portfolios align with the Agency’s strategic objectives. However, the information the Committee reviewed consists mainly of project dashboards, a business management tool that lists the status of the individual projects and financial information by sub-portfolio, and individual project briefings. There is no ongoing assessment of how the existing sub-portfolios are strategically aligned and are within resource capacity. The Committee could not demonstrate that it has identified benefits for its sub-portfolios or that the projects achieved expected outcomes.

5.20 The Canada Border Services Agency has had a Project Management Framework since February 2012. This framework required that projects meet certain prerequisites before moving to the next phase of development. We found that the five projects we reviewed (Exhibit 5.1) were approved to move to subsequent phases of development without the prerequisite conditions. In December 2013, the Agency revised the Project Management Framework, created a Service Lifecycle Management Framework, and revised governance responsibilities. The Transformation, Innovation and Project Portfolio Committee was given responsibility for ensuring that project prerequisites are completed before moving to the next project phase (Exhibit 5.2). For the five projects we reviewed, we noted that the Committee is challenging the projects; however, the prerequisite conditions have not been met. These include business cases, risk assessments, detailed project plans, and project benefit plans. The absence of these prerequisite conditions has led to issues such as project delays, duplication of effort, and a lack of measurable project benefits. (See paragraphs 5.37 to 5.41.)

5.21 Agency Investment Plan and Annual IT Plan. The Canada Border Services Agency has an Agency Investment Plan, which establishes the investments in projects, assets, and acquired services to fulfill the Agency’s mandate. The Agency Investment Plan incorporates components from the Agency’s Annual IT Plan. The Annual IT Plan is important because it identifies the right people, technology, and financial investment required to support the Agency’s mandate and deliver the intended value. By having this Plan, the Agency can identify gaps in resources and any risks to achieving objectives. The Annual IT Plan is also required by the Treasury Board Directive on Management of Information Technology.

5.22 The last Agency Investment Plan was prepared in 2011. It incorporated an overview of the IT investments required and listed 15 technology projects totalling $742.8 million. At that time, the Agency stated that the Investment Plan would be updated annually and presented to the Treasury Board when a significant change occurred. The Treasury Board Policy on Investment Planning—Assets and Acquired Services requires that the Agency update the Investment Plan every three years, or when a significant change occurs. Since 2011, the Agency has taken on several Beyond the Border Action Plan initiatives, which has significantly increased its IT portfolio size to 30 projects at an approved budget of over $1 billion. The Agency has not updated its Investment Plan to account for this significant change.

5.23 However, the Agency has conducted some ad hoc IT investment planning activities over the past 12 months. The following are examples.

• In April 2014, the Agency developed a Five-Year Capital Plan. The Plan breaks down the funding between Government of Canada and Agency commitments, asset renewal, and discretionary investments. This Plan was developed in part to manage capital funding and address the fact that capital surpluses had grown significantly, while IT infrastructure projects were delayed due to lack of funding. The Agency also recently introduced a prioritization methodology for projects using a number of factors, including strategic fit, alignment to Government of Canada and Agency priorities, and benefits. This methodology has been applied to the discretionary projects.
• In June 2014, the Agency completed an Annual IT Plan to demonstrate how IT aligns with Agency strategic objectives. This Plan outlines the IT priorities, key initiatives, operational finances, IT risks, and human resource capacity. It also incorporates the results of other analysis and plans, such as an analysis that identified 42 of the 288 IT applications used in the Agency’s operations as needing to be replaced due to aging and operational budget projections. However, this Plan does not include the Agency’s IT projects.
• We also noted that the Agency prepared an Integrated IT Project Plan at the request of the Treasury Board of Canada Secretariat in August 2014. The Plan states that the Agency provides a comprehensive portfolio view of IT-enabled projects, including commitments, dependencies, milestones, capacity, and risk. The Plan includes 14 of the 30 projects currently under way.

5.24 Without an up-to-date Agency Investment Plan, the Agency limits its integrated view on the appropriate investment mix to ensure IT investments are meeting established program outcomes. The Agency told us that it intends to incorporate all identified needs for updates into the Agency Investment Plan for March 2015.

5.25 Enterprise architecture. Enterprise architecture is important because it maximizes the value of the Agency’s IT investments by promoting reuse, agility, and innovation. We found that although the Agency does have architecture documents on a project-by-project basis, it does not have a target enterprise architecture. The current enterprise architecture does not provide direction on how the portfolio of projects should be prioritized and scoped to avoid redundancies and promote innovation. The lack of an overall portfolio enterprise architecture results in duplicate information technology systems and workarounds.

5.26 As an example of duplication, the Canada Border Services Agency’s 2012–2016 IT Strategic Technology Plan identified the need for a reusable enterprise master data management (MDM) system to provide consistency in creating and exchanging key data for the Agency. However, instead of building one MDM system, we found that two MDM systems were being built individually, with future plans to merge the two. The Traveller Portfolio identified the cost of its MDM system to be $14.3 million, while the Commercial Portfolio identified the cost of its system to be $11.7 million. The Agency was not able to provide us with an estimate of what the savings would have been if one MDM system had been built. It was also not able to provide us with an estimate of how much it would cost to merge the two systems. (Details on the duplication of MDM systems can be found in paragraph 5.37.)

5.27 In July 2012, the Agency identified a business-to-business service that would allow the Canada Border Services Agency’s IT systems to communicate with the systems of other organizations, while minimizing costly changes for both parties. The Agency proposed that this service be used by 11 Agency projects to enable secure and efficient exchange of information. This Business-to-Business Project was budgeted at $5.4 million for three phases over five years ending in 2017. In January 2014, the Agency ended the project after completing phase 1. We found that an important component that is needed to connect the 11 Agency projects was not built. Of the 5 projects we examined, the Single Window Initiative, Field Operations Support System replacement project, and Entry/Exit Initiative identified the need to invest in additional solutions to be able to use the business-to-business service.

5.28 IT portfolio risk profile. An IT portfolio risk profile takes into account portfolio risks from external and internal sources. The objective of portfolio risk management is to accept the right amount of risk in proportion to the expected benefit to deliver the optimum outcome for the Agency in the short, medium, and long terms. Risk management is critical where high-priority portfolio components depend on each other, where the cost of portfolio component failure is significant, or when risks from one portfolio component raise the risks to another portfolio component.

5.29 We found that although the Canada Border Services Agency has conducted some risk assessments, it does not have an overall risk profile for its IT investment portfolio, nor for sub-portfolio investments, as required by its Project Portfolio Management Framework. For example, in July 2012, the Agency carried out a risk assessment exercise to identify the overall risks that may affect the implementation of Agency-led Beyond the Border Action Plan projects. However, the assessment was applied only to the Beyond the Border projects, which account for 61 percent of the Agency IT portfolio. Also, the Agency developed an Annual IT Plan in June 2014 that identified IT risks and mitigation strategies. Lastly, the Integrated IT Project Plan, developed in August 2014, has identified horizontal risks across the projects. These separate risk assessments need to be considered to understand the impact at the portfolio and sub-portfolio levels. Without an IT project portfolio risk profile, the Agency cannot determine the level of risk in proportion to the expected benefits to deliver the optimum outcome for the portfolio of IT projects.

5.30 Recommendation. The Canada Border Services Agency should ensure that all elements of the Project Portfolio Management Framework are implemented and strengthen its governance of IT investments by

• updating its Agency Investment Plan and Annual IT Plan and incorporating all significant capital projects, the Agency’s capacity to undertake the projects, and the financial investment required;
• defining and completing its target enterprise architecture and using it to provide direction on how projects should be prioritized and scoped to avoid redundancies; and
• completing an IT project portfolio risk profile, by building on the Beyond the Border IT portfolio risk profile to understand how the risks affect the IT portfolios.

The Agency’s response. Agreed. The Canada Border Services Agency has developed and will continue to strengthen project portfolio management and update its Agency Investment Plan for spring 2015 and its Annual IT Plan for June 2015.

The Agency is updating its Agency Investment Plan and its Annual IT Plan as part of its regular cycle and expects to present them to the Treasury Board for approval in spring 2015. The Agency Investment Plan includes all significant capital projects to be undertaken over the next five years, including required financial investments and a capacity assessment confirming the Agency’s ability to undertake its project portfolio in full alignment with its governance structure and project management practices.

The Agency will continue to expand its end-state enterprise architecture in both breadth and depth; a functional directive covering every domain of the enterprise architecture will be finalized by September 2015. Prior to final directives being published, the Agency has already begun to steer individual projects toward architecture standards that fully align with Shared Services Canada directions. The Service Lifecycle Management Framework will ensure that enterprise architecture directions are adhered to by all projects through formal gate reviews and approvals.

Finally, the Agency will continue to maintain the Beyond the Border project-level risk profile and will provide a full portfolio risk update roll-up at the quarterly meeting of the Beyond the Border Senior Project Advisory Committee.

The Agency has been experiencing challenges in ensuring that projects meet business requirements and that measurable benefits have been identified

5.31 For the projects we reviewed, we found that the Agency has been experiencing challenges in ensuring that projects meet requirements and deliver expected benefits. The Agency’s IT projects had high-level business requirements but did not have clear corresponding IT system requirements before they started the execution phase, and the Transformation, Innovation and Project Portfolio Committee did not always ensure that all requirements were met prior to approving projects for the next phase of development. In addition, none of the projects we reviewed had clearly defined measurable benefits, and some projects had poorly stated benefits.

5.32 Our analysis supporting this finding presents what we examined and discusses

• business requirements, and
• project outcomes and benefits.

5.33 According to the Agency’s February 2012 Project Management Framework, IT projects must have IT systems requirements by the end of the planning phase before moving to the execution phase. This is still a requirement in the revised December 2013 Project Management Framework and in the new Service Lifecycle Management Framework. IT systems requirements should be clearly defined so that business owners can demonstrate that the project aligns with its strategic direction, meets business needs, and has appropriate funding and resources. Clear definition of a project’s IT systems requirements at the planning phase also allows a business owner to identify the proposed system’s measurable benefits and demonstrate that the system has achieved desired outcomes after it has been built.

5.34 Our recommendation in this area of examination appears at paragraph 5.42.

5.35 What we examined. We examined a sample of multi-stakeholder IT projects and assessed whether the Agency implemented IT project management practices and whether it demonstrated results. In particular, we looked at business requirements and outcomes and benefits.

5.36 Business requirements. While the projects we examined had high-level business requirements, we found that they did not have clear corresponding IT systems requirements before they started the execution phase. Furthermore, the Transformation, Innovation and Project Portfolio Committee, which is responsible for approving projects for the next phase of development, has not always ensured that requirements of the Project Management Framework and the Service Lifecycle Management Framework have been met before granting approval.

5.37 We also found several challenges that the Agency had to address during the execution phase of these projects because of the lack of clarity on IT systems requirements.

• Duplication of effort: We found the development of duplicate master data management–like systems for the Entry/Exit Initiative, Field Operations Support System replacement project, and eManifest project. eManifest, which we came across during our audit, is a project that will enable traders to electronically transmit data on shipments. Recently, the Agency recognized the duplication of effort and mandated a task force to consolidate all master data management systems across the projects. (The financial impacts are described in paragraph 5.26.)
• Cost breakdown: Although the projects had a general understanding of what work had been deferred, some of the projects we examined do not have costs broken down by detailed deliverable, and thus the Agency cannot tell us what work is being deferred and at what cost. For example, for the Entry/Exit Initiative, Single Window Initiative, and Interactive Advance Passenger Information initiative, we found that the Agency had requested more funds than it was able to use in a given fiscal year and that it planned to move the funds forward to be spent in the next fiscal year.
• Project delays: For the Field Operations Support System replacement project, we found issues that delayed the project. In September 2013, the project was initiated to replace an aging system with an original budget of $25.4 million and a planned completion date of December 2014. This coincided with the date that vendor support would also end. The Agency knew about the need to replace the aging system since 2008, when Citizenship and Immigration Canada told the Agency about it. In January 2014, the Agency had a difference of opinion with Citizenship and Immigration Canada over how a critical function of the replacement system would be built. The eventual resolution resulted in a delay of three months, which forced the Agency to extend a vendor contract at a cost of $2.3 million. As well, the functionality that was finally agreed upon cost an additional $1.7 million. The Agency also clarified other requirements, which added $4.9 million to the budget. As a result, issues identified with lookouts by the Agency’s internal audit function and the Office of the Auditor General’s fall 2013 chapter on preventing illegal entry into Canada will not be addressed until the project is completed. Until the issues with lookouts are addressed, there is still a risk of incomplete immigration lookout information and unknown status of lookouts.
• Project deliverables: Some projects are at risk of not delivering what was expected. In the case of the Interactive Advance Passenger Information initiative, despite starting in 2012, the Agency was still finalizing the deliverables and their milestones with Citizenship and Immigration Canada’s Electronic Travel Authorization project at the time of the audit. The Electronic Travel Authorization project, which would require nationals of certain countries to obtain an electronic authorization before entering Canada, depends on the successful delivery of the Interactive Advance Passenger Information initiative to enforce electronic travel authorizations. In the case of the Entry/Exit Initiative, the Agency began work with Citizenship and Immigration Canada in October 2013 to draft project deliverables and milestones. As of September 2014, these were still being finalized. The Agency also changed a key component in favour of a new solution (master data management) that has caused Citizenship and Immigration Canada to revisit the components it is building.

5.38 Had the projects in the above examples followed the requirements of the Project Management Framework, and followed the process for approving projects to proceed to the next phase, the above challenges could have been avoided.

5.39 Project outcomes and benefits. The Treasury Board Policy on the Management of Projects requires that the Canada Border Services Agency have in place processes to ensure that the desired business outcomes have been clearly defined, and are realized through a structured approach and with assigned accountability and ownership. The Canada Border Services Agency’s Project Management Framework and the Project Benefits Management and Realization Guideline state that a project’s outcomes and associated benefits should be identified and agreed to at the concept phase, which is at the beginning of project approval, and be measurable.

5.40 Of the five projects we examined, we found that they all had defined project outcomes at the planning phase. However, four of them did not define measurable benefits and are now in an advanced stage of development and are still establishing clear benefits. Without the identification of benefits, the Agency cannot demonstrate alignment between project delivery and business needs or outcomes. The following are examples.

• The Interactive Advance Passenger Information initiative has defined outcomes in various documents. It also has a benefits realization plan. However, the plan is incomplete. For example, the plan currently states that financial savings and cost avoidance are unknown. The Agency informed us that a final version is expected in April 2015 that will estimate benefits such as cost savings. Also, a baseline study is expected prior to the implementation date in October 2015. This project is currently in the execution phase.
• The Entry/Exit Initiative initially identified security-related outcomes. As the project progressed, draft financial benefits were identified. In addition, a draft performance measurement strategy and framework were developed in July 2014 that detailed key outcomes and performance indicators. However, the project has not finalized a strategy to evaluate the effectiveness of project outcomes, nor was a benefits realization plan in place. This project is currently in the execution phase.
• The Temporary Resident Biometrics Project is a Citizenship and Immigration Canada–led initiative; the Canada Border Services Agency is a key partner. Citizenship and Immigration Canada did identify measurable outcomes and assigned the Agency responsibility for measuring seven indicators within one year of full implementation. However, the Agency has not yet measured outcomes or set target values. A project benefits realization plan was also not completed for this project. This project was completed in June 2014.
• In November 2012, the Single Window Initiative developed a performance management strategy and framework that identified a baseline and target values for outcomes. As well, there are project benefits reports stating that financial savings would be identified through the business and efficiencies business case. This business case was due by March 2015 and is a condition for Treasury Board to release additional funds. This project is in the execution phase.

5.41 Furthermore, an initial assessment of benefits readiness conducted by the Agency in August 2014 identified that over 50 percent of the portfolio of projects have “low readiness,” meaning there was minimal information on established benefits in their initial project documents. The assessment also noted that another 27 percent are unknown, even though many of these identified projects are past the project planning phase.

5.42 Recommendation. The Canada Border Services Agency should ensure that project requirements are met and measures are defined to assess if the projects deliver expected benefits.

The Agency’s response. Agreed. In alignment with the Canada Border Service Agency’s Benefits Management Framework and the corporate project portfolio governance model, the Corporate Affairs Branch ensures that all benefits are fully managed until fully realized (since January 2015). The Branch provides a coordination and oversight function across all project stakeholders and, beginning in September 2015, will report quarterly to the Agency’s Executive Committee on the status of benefits realization. Coordination and oversight activities include the monitoring of project benefits, risks to benefits, benefits realization plans, and benefits health assessments.

In June 2015, the Agency will further strengthen its benefits management through a second benefits review and challenge of benefits documentation. The challenge and oversight function will continue through all Project Portfolio Management Framework gate reviews to ensure effective decision making throughout the project life cycle.

The benefits management process and its integration within the Project Portfolio Management Framework and executive governance model will continue to be fine-tuned until March 2016. A pilot report will be presented by June 2015 and will include an initial baseline set of performance benefits indicators.

Project information reported to senior management is inconsistent and incomplete

5.43 We found that the project information presented on the monthly project dashboards to senior management was not consistent with source project documentation and was incomplete with respect to cost, schedule, and scope.

5.44 Our analysis supporting this finding presents what we examined and discusses

• project dashboards.

5.45 This finding matters because project dashboards are presented to senior management to make decisions on whether more active monitoring is required and whether corrective action is needed to keep the projects on track for cost, schedule, and scope.

5.46 The project dashboard is a business management tool that shows the status of a project or a portfolio of projects. It communicates key project information consistently, accurately, and succinctly to Agency senior executives who play a governance and oversight role. Because it draws attention to project areas that may require correction, the quality and accuracy of the information are critical.

5.47 Our recommendation in this area of examination appears at paragraph 5.61.

5.48 What we examined. We examined a sample of multi-stakeholder IT projects and assessed whether the Agency implemented IT project portfolio management practices, with a focus on whether the information was reported in accordance with Agency frameworks and guidelines.

5.49 Project dashboards. To report IT project information to decision makers, the Agency uses two types of monthly project dashboards. The project dashboards, prepared by the project managers, show a project’s health, which is determined by whether the project is meeting schedule milestones, undergoing changes in scope, and staying within budget. The enterprise dashboard is a consolidation of the project dashboards and provides the Executive Committee with key project information, including actuals, commitments and planned spending, schedule, scope, risks, and other issues. This information is critical in allowing the Committee to understand the current project status and to address project issues early. The enterprise dashboard is prepared by the Agency’s Enterprise Project Management Office. The Office issued a Guide to Executive Project Dashboards to help project managers complete the monthly project dashboards.

5.50 For the projects we examined, we found that the information provided on the project dashboards was inconsistent and incomplete. For example, the project costs on the project dashboards are different from what is reported in the financial systems. We found that the project dashboards contained information from various sources in addition to the financial system reports, including informal project management spreadsheets used to report on projects’ forecast spending.

5.51 In addition, we found that there were inconsistencies in the financial information reported in these project dashboards when compared with other Agency reports for the same period. For example, the financial information reported in the individual project dashboards was not consistent with what was in the enterprise dashboard or other key IT documents, such as the Integrated IT Project Plan.

5.52 We found that project managers had discretion in what they report on the dashboards, such as schedule milestones and changes to scope, thereby influencing the overall project health status. Without reliable project information for portfolio monitoring, senior management does not have a true and complete picture of project status in order to make informed decisions. Specific examples of these status reporting deficiencies from project dashboards for a selected period are in paragraphs 5.53 to 5.56.

5.53 For the four projects in our sample that were still in development, we reviewed the project financial information reported from June to October 2014 on both the enterprise and project dashboards and compared them with the financial system reports. We found significant variances between the project dashboards and the financial system reports for all four projects we reviewed. Although the Agency’s Comptrollership Branch does challenge the information on the project dashboards, there is no monthly analysis conducted to explain the variances. These variances range between 6 and 157 percent, depending on the period and project reviewed. This is important since the project dashboards for three of the projects are sent to the Treasury Board of Canada Secretariat as part of the ongoing monitoring of projects. Comparing the enterprise dashboards with the financial system reports for the same period, we found fewer and smaller variances. However, the Agency had not conducted an analysis on these variances.

5.54 We found that there were variances in the reporting of project status between the project dashboards and other documentation. The following are examples:

• The Single Window Initiative includes discrepancies in key deliverables that cannot be reconciled between different key project documents. In a review of the 12 most recent dashboards up to 2014, we noted that key deliverables in the project schedule and plan were not consistently reported.
• The Entry/Exit Initiative key milestones and deliverables do not agree with key project documents. Approved completion dates within the project dashboard do not align with dates in approved documents. For example, the July 2014 dashboard reported a phase 3 delivery date of November 2014 to implement the exchange of information of all land travellers. However, the project charter had a delivery date in June 2014.

5.55 We found some discrepancies in project scope between the project dashboards and key project documents.

• The Interactive Advance Passenger Information initiative changes are not included in monthly dashboards. For example, project dashboards for 2014 did not mention any change requests. However, other project documentation indicates that change requests were communicated to senior management.
• The Entry/Exit Initiative changes affecting key components were not appropriately reflected in the dashboard. Several change requests that have impacts on the project’s scope have not been reflected in the project dashboards. In addition, significant deliverables approved in the original project documents were not on the dashboard; they are discussed in other documents. For example, the deployment of radio frequency identification technology and licence plate readers at predetermined ports of entry, costing about $20 million, is not reported on the monthly project dashboards.

5.56 We found that project health was not accurately captured for the Field Operations Support System replacement project and was incompletely captured for the Business-to-Business Project, which we came across in the course of our audit.

• In July 2014, the Field Operations Support System replacement project dashboard showed some areas at “yellow” status, indicating some problems; according to the Guide to Executive Project Dashboards, however, we determined the project status to be “red,” indicating serious problems, meaning the project was not under control.
• The Business-to-Business Project was closed in April 2014 after completing the first of three phases. However, the Agency is still working on this project without reporting on project health. This project had an approved budget of $5.4 million from 2012 to 2017: $3.4 million for phase 1 and $2.0 million for phases 2 and 3. The final close-out report stated that the project had spent $3.6 million for phase 1 and that all deliverables had been completed, even though an important component had not been built.

5.57 Since May 2014, the Information, Science and Technology Branch introduced a new process to monitor project performance using a technique known as earned value management reporting to mitigate discrepancies in the monthly project dashboards. The intention is that this approach would function as a warning system to ensure project problems are addressed early.

5.58 Currently, 18 of the 30 IT projects are reporting using this new earned value management method. This method improves project performance monitoring; however, we noted some inconsistencies between the earned value management report and the project dashboard for the Entry/Exit Initiative and the Field Operations Support System replacement project.

5.59 The Enterprise Project Management Office is responsible for coordinating and providing a challenge function for all projects, including IT; however, the authority resides with the project manager to make a final decision about which project status information is presented. In support of the Enterprise Project Management Office challenge function, the Agency’s Comptrollership Branch also reviews the financial information on monthly project dashboards. However, due to financial system limitations and a complex project costing methodology, detailed manual reconciliations are needed to ensure that these dashboards provide an accurate picture on project financial health.

5.60 In our opinion, the absence of consistent controls over project reporting raises several risks that reporting on project status is ineffective because the reported information is subject to adjustments that reduce its reliability.

5.61 Recommendation. The Canada Border Services Agency should establish clear procedures and practices on how the information for the project dashboards is collected, reported, and enforced to ensure consistent and complete project status reporting for its portfolio of IT projects.

The Agency’s response. Agreed. The Canada Border Services Agency will continue to clarify its procedures and practices on how the dashboard information is collected, validated, and reported, to ensure consistent and complete project status reporting for its portfolio of IT projects. A formal review of the process will be completed by June 2015.

Conclusion

5.62 We concluded that the Canada Border Services Agency (the Agency) has developed the necessary corporate and management practices to deliver on IT investments through its Project Portfolio Management Framework. However, the Agency has not put into practice all the elements of the framework that would enable it to ensure that the delivery of information technology investments align with and support its strategic corporate objective. Also, the committees that oversee the portfolios have not ensured that they have all the information they need to fully exercise their duties and responsibilities for ensuring that project requirements and conditions are met at each stage of approval.

5.63 Based on the projects we examined, the Agency has experienced challenges in ensuring that projects meet its business requirements and deliver expected benefits. At the time of the audit, the Agency was still defining its requirements for the projects we examined. We also noted that although clearly defined and measurable benefits are required by Treasury Board and Agency frameworks and guidelines, the Agency had not defined measurable benefits that demonstrate alignment between project delivery and business outcomes.

5.64 Finally, we noted that the Agency’s mechanism to provide senior management with an understanding of the current project health and to address project issues was deficient. The information provided on monthly dashboards was inconsistent and incomplete, as various discrepancies were identified with respect to cost, schedule, and scope.