Internal Audit ReportReview of Governance—Audit Working Paper Software (AWPS) Project

Reflections from the Chief Audit Executive

First and foremost, I would like to thank all those who were consulted during the 2 distinct phases of this review for their cooperation and collaboration. In particular, I want to extend my gratitude to the dedicated members of the Audit Working Paper Software (AWPS) project team who willingly shared their valuable experience and insights with us to help produce more positive outcomes for future Office of the Auditor General of CanadaOAG projects.

The intended purpose of this report is to provide information to foster accountability and to support the organization in improving its ability to manage projects in the future. The purpose is not to assign blame. Distinguished author Brené Brown writes on this subject: “Blame is simply the discharging of discomfort and pain. It has an inverse relationship with accountability. People who blame a lot seldom have the tenacity and grit to actually hold people accountable.” As the OAG’s governance and project management mature, it will be critical to ensure that appropriate systems are in place around its projects. We hope that this report will be used as a tool for learning and growth for the organization’s many important future projects and initiatives.

Furthermore, this report does not opine on why this project was cancelled or whether it was the appropriate decision. According to the Standish Group, only 31% of software projects are successful, and only 46% of those return high value for the organization^{Footnote 1}. It’s a perspective that should be front of mind as the organization reflects on this project’s failures and successes. It is also important to remember that failures, small and less small, are often par for the course. The measure of an organization lies in whether it learns from them and improves the next time around. In keeping our sights on the future, it is imperative that full and complete information be communicated and that we trust that leaders will make the best decisions they can with the information they have at a particular point in time.

In addition to keeping our focus on our objectives, keeping the ever-changing context constantly in our line of sight can be a daunting task. This is where risk management can be a powerful tool. As the organization continues to mature its risk management, those making decisions and those contributing information for decision making will need to understand and appreciate the organization’s risk appetite and tolerance, the inherent risks of any digital project, and the potential opportunities that come from smart risk taking. Staying the course, adjusting, pivoting, or even reversing decisions are all elements that come from properly managing risks and opportunities, and the ability to do these is a sign of an organization’s growing maturity.

Finally, I want to highlight the important role that Priority 1—One Office, One Team, One Vision has on supporting the way forward for project management, links which are made throughout this report. To borrow Assistant Auditor General Paule‑Anny Pierre’s words from her recent Fireside Chat, this priority “is us acknowledging that we cannot be effective and be successful in delivering our mandate if we are not all in it. Each and every one of us has a role to play.” Knowing what each of those roles are, and how they affect others’ roles, will support better coordination, cooperation, and alignment of our projects. This will also promote the appropriate level of individual and collective accountability.

At the core of author Cy Wakeman’s reality-based leadership philosophy is “stop judging, start helping.” She puts forward that asking the simple yet sincere question—“How can I help?”—will lead to instant teamwork. Proactively offering and asking for help is important in any team environment; however, it is especially impactful in a context that has many of us stepping outside of our comfort zones and being tasked with things that we have limited experience with. Furthermore, every member of a team brings a different perspective. Respecting and leveraging the diversity of those perspectives brings tremendous value to the management of any project, especially if one is ready to approach new or even contradictory information with curiosity—it is the recipe for learning and growth.

Management has developed a detailed action plan to address the recommendations outlined in this report. Since the AWPS project was cancelled, the organization has already taken important strides in maturing its governance and project management capacity. It will be important for management to monitor how these changes affect the achievement of intended outcomes and whether they strengthen both small and big “g” governance around projects and to adjust accordingly. As project-related activities may be new for many, and there will be competing priorities for resources, gaps in resources and skills are likely to continue to exist. These gaps may require upskilling, prioritization, coaching, additional capacity, and more.

The Internal Audit and Evaluation Office will monitor the implementation of management’s actions during the lifecycle of other projects and through a combination of products and services aimed at providing the best value for the organization.

— Julie Bastarache, Chief Audit Executive

Objective and scope of the review

The objective of this governance review was to understand what elements contributed to how the project unfolded in order to improve the outcomes for the next audit working paper software project and other similar projects for the OAG.

The review was scoped to focus on the following elements:

• strategic assessment and the project delivery approach
• solution requirements and contract performance monitoring
• project management planning and execution, including
  • project governance and stakeholder management
  • risk management
  • project progress tracking, including schedule and quality considerations
  • human resources management

The internal audit team initially performed review work on the AWPS project, with a different scope and objective, from February to August 2022. Although a final report was not issued at that time, information obtained during that phase informed the work and the report on this current review engagement.

See Appendix A for who and what we consulted for this review engagement.

See Appendix B for resourcing information about this review engagement.

Project and governance context

This project was initiated in 2018 to replace the OAG’s audit working paper software. The project spanned several years during which significant changes affected the office, including the introduction of a formal project management governance framework. Since this project was “in flight” when this framework was being introduced by the newly centralized project management office, the project team had to adopt the evolving project management methodology as it was being released. This included integrating certain elements of agile project management, which, although used previously for other information technologyIT projects, was a departure from the traditional waterfall approach that was more familiar to many in the organization.

The OAG purchased an off-the-shelf software solution and entered into a contract with the vendor to configure and customize the product to the OAG’s specifications. The initial expected expiry of maintenance support for the existing working paper software in 2023 affected the planned project timelines. The vendor used a third-party supplier for the implementation and the project execution involved the vendor and its subcontractor’s personnel and OAG personnel who were jointly responsible for testing the supplier’s output.

The project team experienced ongoing challenges to configure the software with the OAG’s audit methodology and to deal with quality concerns. Despite these obstacles, the project team made slow but steady progress and expected to deliver a minimum viable product by the revised go-live dates. However, beginning in spring 2022, questions started arising about whether the intended outcomes of the project were attainable. An analysis was prepared in May 2022, and the decision to continue the project was made. Questions arose again in late August following a series of setbacks, and the project was eventually suspended and then cancelled in October 2022 when it was determined that its expected outcomes did not align with the desired future state of the organization, which had embarked on a digital transformation journey.

Corporate governance components also evolved during the project, starting with the naming of a new executive committee team during the request for proposal (RFP) process, an attempt to set up a cross-functional collaborative planning team in 2021, and the implementation of a new corporate governance structure involving the creation of cross-functional Tier 2 committees in 2022 and plans for Tier 3 committees to start in 2023. These committees were not in place to benefit the AWPS project.

Important events are presented in the project timeline.

Project timeline

Results

Themes

• Oversight, communication, and collaboration
• Maturing project management capacity
• Strategic assessment

While our work was executed according to the 3 areas of focus noted in our plan, the root causes of our findings aligned with these 3 themes.

Oversight, communication, and collaboration

The audit working paper software project was a significant project for the OAG based on project classification metrics, such as duration, cost, technology, work effort, groups affected, and others. The software tool itself was to be used to document audit work and to support the delivery of quality audit products, which is the core mandate of the organization. This IT‑enabled project involved the participation of stakeholders from across the office, including both audit practices, audit services, procurement and contracting, IT security, IT infrastructure, and others. Contracted services with the software vendor also significantly contributed to the project outcome and added complexity to the project.

Oversight and challenge function

We found that the quality of the oversight of a project of this magnitude was not sufficient and that those assigned these roles may not have been asking the right questions or applying an adequate level of professional skepticism expected of a mature project management function. The following are some examples:

• Our review of correspondence indicated that the executive committee sponsor, who was accountable for the project, performed an oversight and challenge function based on the information that was provided by team management, which focused primarily on project execution. However, we noted both a lack of questioning and assessment of the aggregate effects of the individual risks, issues, and project metrics being reported and the advancement of the project toward the benefits and outcomes that had been envisioned.
• There was a lack of active and visible participation by the executive committee sponsor and business owner in working-level team meetings, partially because of time constraints as a result of operational workloads not being alleviated when assigning project responsibilities to individuals. This ongoing participation would have provided an opportunity to expand oversight, to obtain a broader perspective on the project, and to enhance communication and collaboration.

• The executive committee was the only cross-functional committee overseeing the project, as Tier 2 and 3 committees were not established at the time. Our review of executive committee meeting minutes and presentation material indicate that up until spring 2022, AWPS project updates and discussions were based on information included in a monthly priority project dashboard prepared by the Portfolio and Project Management team. The committee focused on individual project health indicators (such as budget, risk, issues, and time). The nature of the reporting to the executive committee prior to spring 2022 likely affected the committee’s ability to exercise its oversight function. We noted the committee performing a greater challenge function beginning in June 2022.
• During our interviews with executive committee members, most expressed that there might have been a limited understanding of what appropriate project oversight is and of the information needed to exercise that role. More education on how to prepare and receive briefing materials to enable effective project oversight may be needed.
• The executive committee sponsor asked the internal audit function to perform an independent review of this project as early as 2021. However, internal audit did not begin that work until February 2022, and substantive work was not completed until July 2022. The review objective focused on assessing whether the project was appropriately planned and monitored to achieve a minimum viable product. This goal was not aligned with either project or strategic objectives, and the Internal Audit team should have approached the project differently. Although elements of the preliminary findings were discussed with the executive team during the examination phase, a final report was not issued because of concerns about the level of evidence to support the initial proposed findings. In addition, discussions about assessing the viability of the project were occurring at the time of reporting^{Footnote 2}.
• The Portfolio and Project Management (PPM) team’s stated role on individual projects is to act as a facilitator and as the OAG’s centre of expertise on project management. As such, the PPM team did not play any oversight role on this project. However, given their expertise, and the maturing nature of project management and governance within the organization, the project team would have benefited from more active involvement from the PPM team.

Communication and collaboration

• The core project team adopted the agile project management mindset and was generally successful in implementing agile principles related to communication and collaboration at the working level. An agile project management coach was hired to support the learning of team members who wanted to adopt a new way of doing things.
• However, the hierarchical nature of the project’s governance may have impeded open and timely exchanges of information. The governance structure was not aligned with the wide and flat structure that is helpful in managing projects in an agile way to support rapid exchanges of information and decision making.
• We also noted an opportunity to improve collaboration at the executive committee level to support the executive committee sponsor and to seek better alignment when addressing concerns about the project.
• The project status was regularly reported to the executive committee sponsor and the Chief Information Officer via detailed monthly reports and meetings. We noted, however, that the summary report provided did not always align with the underlying information. For example, the project indicators in the status updates tab of the project reporting workbook were reported as green, and this status did not always align with the information in the individual risks and issues tabs of the same workbook. We also noted that there were no indicators for contract health. In projects where the contract significantly contributes to the project outcomes, we would suggest that its health status be included as a metric for project monitoring and reporting.
• We found that the nature of the reporting on the project status to the executive committee, which was done primarily via a dashboard prepared by the Portfolio and Project Management team using the summarized data from the project reporting workbook noted previously, did not result in accurate and detailed updates during much of the project. It was only in June 2022 that the project team started presenting more detailed updates on the project. Had this additional reporting been occurring earlier, it would have likely contributed to important risks or issues being escalated earlier and to a more collaborative approach in managing them.
• Client review meetings between the executive committee sponsor, business owner, and project team, planned for every 3 weeks, did not occur. In addition, meetings with stakeholders planned for every 1.5 to 2 months occurred much less frequently. More frequent and timely meetings would have facilitated the appropriate sharing of information with project stakeholders and would have allowed for better information flow (top-down, bottom-up, and across), minimizing the likelihood of surprises and ensuring that key players were better informed.

• Coordinating the many contributors to this project was challenging, and certain contributors were not always engaged at the appropriate times. In particular, the Procurement and Contracting team was not advised of issues with project progress in a timely manner. In a complex procurement such as this one, continued engagement with the procurement team may have been needed to obtain guidance and ensure that contracting obligations were being respected. Staffing constraints, including those resulting from the labour dispute, affected this active engagement.

Recommendations for oversight, communication, and collaboration

Recommendations and alignment to related OAG strategic priorities, outcomes and actions

Maturing project management capacity

Project management resources and guidance were evolving throughout the project lifecycle

• The Portfolio and Project Management team has and continues to develop comprehensive project management guidance, such as the Project Management Handbook and its supporting methodologies. These resources were evolving throughout the project and at times were perceived to be hard to access. For instance, the Project Reporting Workbook was updated several times during the project’s lifecycle.
• The changing nature of the templates and tools may have unintentionally created confusion for the project team as to what templates and tools had to be revised or completed, who would be reviewing them, what level of detail was required for them to be considered complete, and perhaps most importantly, what their purpose and significance to the project was.

Challenges related to the completion of the project management plan

• We examined 7 of the key sections of the Project Management Plan and determined that only 2 contained a sufficient level of detail to be assessed as completed. The remaining 5 sections were either not sufficiently detailed, lacked clarity, or were missing critical information to allow for that component to be appropriately implemented.
• A summary of the sections reviewed is available to the right; details on the 5 incomplete sections is provided next.

Risk management plan

• We observed that the risk management plan did not provide information or instructions on the criteria for identifying and assessing the likelihood and impact of identified risks. We noted that some guidance was made available (risk assessment matrix) in the AWPS project reporting workbook. However, it was different than that being provided directly by the Portfolio and Project Management team. For example, the criteria for assessing risk likelihood and effects were different in each document. Also, the project reporting workbook template asked the project team to use a 5 X 5 matrix while the guidance shared by the Portfolio and Project Management team referred to a 4 X 4 matrix. These differences could have affected the interpretation of risk information.

Human resources plan

• The human resources plan was written at a very high level and provided a general description regarding human resources planning. The accompanying resource plan detailed the overall resource needs by practice, group, and team, the various tasks that these resources would be responsible for, and the level of experience required for these tasks.
• Neither of these plans detailed the necessary skills and knowledge required for a particular role, including the project’s governance roles, which may have contributed to competency gaps on the project. Competencies related to engagement, collaboration and influencing others, and writing technical requirements were examples of such gaps. The organization should recognize that the skills needed to manage large projects are different from those needed to manage operations and that additional support and guidance will be required as the OAG continues to build the organization’s capacity.
• Information characteristic of a comprehensive human resources plan, such as a needs assessment; recruiting strategy for identified resourcing gaps (that is, in-house versus outsourced complement and rationale); team development (including conflict resolution); and performance metrics, was also absent from these plans. This is an important consideration as the project team faced resourcing and performance challenges. Had these elements been considered when planning the project, it could have facilitated more efficient and effective responses when these issues, common to both small and large projects, arose.

Benefits realization plan 

• In our view, the plan did not contain sufficient information related to how the project benefits were to be tracked and reported. The plan references certain benefits with a general statement that they will be periodically reviewed. No measures were defined that could be monitored during the life of the project to assess if it was on track to deliver expected outcomes successfully according to predetermined targets.

Change management plan

• Some elements of the change management plan were fairly well defined, such as thresholds that could trigger formal change requests affecting the budget and timeline, although these were not defined for scope changes. 
• The organizational change management section was not fully completed. Early in the project, the team decided to postpone engaging a change manager until more information was known about the solution and its impacts on the organization. A change manager would have helped with managing this uncertainty. A formal change management plan is needed for large projects such as this one that affect a significant portion of employees and the office’s main operations and should be initiated early in the project lifecycle and adapted as needed.

Roles and responsibilities

• While roles and responsibilities were defined in the plan, we noted some challenges in the performance of those roles. We discussed our findings related to project oversight, responsibilities for monitoring and reporting project progress, and issue escalation in the previous section. We address issues related to strategic oversight in the section strategic assessment.

Insufficient project planning contributed to the following challenges for appropriate project execution

• Certain roles lacked clarity, such as the overlap in responsibilities between the project sponsor and project manager, and the governance chart had 2 distinct reporting lines, which likely contributed to the fact that the escalation of concerns did not always follow the established governance structure for the project.
• Issues with capacity planning likely contributed to the challenges related to the availability of resources and “good fit.” We noted, however, that despite difficulties to secure resources to perform testing when project delays occurred, the team worked diligently and collaboratively to successfully fill these needs. The high turnover in the project manager and business analyst roles added additional challenges for the project.
• The lack of clarity over what constituted a change, a risk, or an issue contributed to some inconsistencies in determining the appropriate steps to be taken to manage or escalate them. The individual issues that continued to arise were being dealt with and resolved by the project team, but their compounding effect on future risks did not seem to be adequately considered.
• Reporting on progress toward benefit realization and achievement of expected outcomes did not occur, likely because the mechanism and measures for this assessment were not established in the plan.

Recommendations for maturing project management capacity

Recommendations and alignment to related OAG strategic priorities, outcomes and actions

Strategic assessment

Strategic assessment

• The project objectives and expected outcomes documented in the business case aligned with the organization’s strategic priorities when it was developed in October 2018. However, we found that continued alignment was not adequately considered or used to inform decision making until later in the project lifecycle. This included assessing the effect of significant changes with respect to corporate priorities and direction (that is, several months of discussion leading up to an official kickoff of the transformation initiative in March 2022) and project developments such as the de-scoping of significant solution components starting in February 2022 which affected expected functionality.
• Because of the timing of this project, the requirements and resulting procurement instrument were developed before the project management approach was established (see the context section). The contract was written in a traditional, or waterfall, form with milestone payments tied to specific deliverables. This did not align with the agile project management approach that was subsequently adopted for the project, which made it challenging to manage contract performance and confirm the degree of contract requirement completion.
• The business problem articulated at the initiation of the project focused on the inefficiencies caused by the lack of automation of the existing audit working paper software, and the solution was approached singularly as a software replacement project. Strategic considerations include immediate, intermediate, and long-term goals. When viewed through the OAG’s current transformation lens, we see that there may have been a lost opportunity at that time and subsequently to concurrently consider other inefficiencies in the audit process or ways that it could be modernized as part of such a project.

Integrated planning

• We also noted that the OAG’s Integrated Risk Management Framework did not provide guidance as to how projects should approach their assessment of project risk appetite and tolerance. Having common definitions and understanding of these important elements, and defining them up front, would have allowed the project team and the oversight body (the executive committee) to ensure that they were aligned in terms of project decision making and when escalation of risks and issues would be expected and required.
• Integrated planning requires a clear understanding of the organization’s readiness and capacity to take on a portfolio of projects. The AWPS experience exemplifies how the organization struggles with gathering the information that will be necessary to perform a robust portfolio readiness and capacity assessment. First, project teams find it challenging to develop realistic expectations and estimates of the financial and human resource requirements for its projects, including material costs and the skills, experience, timing, and duration of human resource needs. Second, the OAG does not have a clear view of the capacity and availability of personnel to work on projects^{Footnote 3}. Addressing these concerns will facilitate the Portfolio and Project Management team’s ability to better use its intake and prioritization tool to support the organization’s decision making regarding project acceptance and appropriate project scheduling.

Recommendations related to strategic assessment

Recommendations and alignment to related OAG strategic priorities, outcomes and actions

Appendix A: Information sources for the engagement

Project documents

Key project documentation reviewed (non-exhaustive list):

• Request for information
• Project proposal form
• Request for proposal
• Business case
• Contract
• Project charter
• Project plan
• Project reporting workbooks (updated monthly)
• Status update emails to the executive committee sponsor and Chief Information Officer (provided monthly)
• Priority project dashboards (provided to the Executive)
• DevOps documentation (project management software program)
• Agile coach report titled OAG Agile Mentoring for Audit Working Paper Software Project

Interviews

Key stakeholders interviewed (non-exhaustive list):

Internal

• Members of audit working paper software project team
• Members of Portfolio and Project Management team
• Members of Contracting, Legal, IT, and Professional Development teams
• Members of the Executive team

External

• Agile project management coach
• OAG Business Continuity representative
• Vendor representative

The above includes interviews conducted during the first phase of PRIA’s work on the audit working paper software project.

Authoritative sources

• Treasury Board of Canada SecretariatTBS Policy on the Planning and Management of Investments
• TBS Directive on the Management of Projects and Programmes
• TBS Guide to Project Gating
• TBS Guide to Independent Reviews
• TBS Independent Reviewer’s Handbook
• Project Management Body of Knowledge (PMBOK) Guide

Internal project management resource

• Project Management Handbook

Appendix B: Resourcing of the review engagement

Text version

For the planning phase, 175 hours were budgeted and 200 hours were used.

For the execution phase, 275 hours were budgeted and 625 hours were used.

For the reporting phase, 100 hours were budgeted and 200 hours were used.

Execution hours exceeded the budget, as more work was required to complement the work done by PRIA in Phase 1, to revalidate, to onboard new team members, and to sufficiently assess selected areas of inquiry. Time invested in execution is considered reasonable by the Chief Audit Executive given the scope and potential impact of this review on ongoing and future projects.

Text version

In total, 550 hours were budgeted, and 1,025 hours were used.

Appendix C: Management Action Plan