¶¶ÒùÊÓƵ

Audit of the Peace and Stabilization Operations Program

October 2020

Table of contents

• Report summary
• 1. Background
• 2. Observations and recommendations
  • 2.1 Program governance and oversight
  • 2.2 Risk management
  • 2.3 Program delivery
  • 2.4 Monitoring and reporting
• 3. Conclusion
• Appendix A: About the audit
• Appendix B: Management action plan
• Appendix C: Projects selected as part of the sample (20)

Report summary

In accordance with ¶¶ÒùÊÓƵ’s approved 2019-2020 Risk-based Audit Plan, the Office of the Chief Audit Executive conducted an audit of the Peace and Stabilization Operations Program (PSOPs).

Why it is important

PSOPs is the Government of Canada’s principal platform for conflict prevention, stabilization and peacebuilding in fragile and conflict-affected states. The Program is an important vector of Canada’s values and commitments at the international level. ¶¶ÒùÊÓƵ receives $150 million annually for PSOPs, including $117.8 million to disburse through grants and contributions. PSOPs is the largest grants and contributions program within ¶¶ÒùÊÓƵ not directly related to international development.

What was examined

The objective of this audit was to determine whether an effective management control framework had been implemented for the Peace and Stabilization Operations Program to support the achievement of the Program’s objectives. This audit examined PSOPs’ management framework, including program planning and funding, project delivery, and monitoring and reporting activities. The audit focused on the last three fiscal years (2017-18 to 2019-20) but also covered projects that started in 2016-17 and were still active during that period. The audit focused on the activities of the Programing Division (IRG) and the Strategic Planning and Coordination Division (IRC), more specifically programming activities and the support provided to project officers. The audit did not look at the activities of the other two divisions. Interviews were conducted and documentation was reviewed up to and including September 2020. More details about the audit objective, scope and criteria are presented in Appendix A.

What was found

An effective management control framework has been established for the Peace and Stabilization Operations Program. A few areas for improvement were identified to further improve the effectiveness of the framework such as:

• Consistent application of the PSOPs Risk Management Guide and business process tools;
• Sufficient and consistent documentation of monitoring and reporting activities; and
• Developing a system to measure the timeliness of project implementation to ensure efficiency and effectiveness of project delivery. 

Recommendations

Recommendation 1: The Assistant Deputy Minister of International Security (IFM) should require PSOPs to systematically document and track decisions of its governance bodies in order to demonstrate effective oversight exercised over the implementation of its mandate, objectives, and strategies.

Recommendation 2: The Assistant Deputy Minister of International Security (IFM) should:

• Establish measures to improve and better document program compliance with the PSOPs Risk Management Guide; and
• Implement the recommendations formulated in the spot check exercises and monitor their effectiveness.

Recommendation 3: The Assistant Deputy Minister of International Security (IFM) should:

• Establish measures to improve and better document program compliance with the PSOPs business process; and
• Develop the capacity to measure and monitor the timeliness of project funding implementation to ensure efficiency and effectiveness of project delivery.

Recommendation 4: The Assistant Deputy Minister of International Security (IFM) should put measures in place to:

• Document project files, including monitoring activities, in a complete, consistent, and effective manner; and
• Properly review and report performance information as per departmental requirements.

Statement of Conformance

The audit was conducted in conformance with the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing and with the Treasury Board Policy and Directive on Internal Audit, as supported by the results of the quality assurance and improvement program.

1. Background

In accordance with ¶¶ÒùÊÓƵ’s approved 2019-2020 Risk-based Audit Plan, the Office of the Chief Audit Executive conducted an audit of the Peace and Stabilization Operations Program (PSOPs).

In August 2016, the Government of Canada launched PSOPs as the new platform for conflict prevention, stabilization and peacebuilding. PSOPs replaced two programs: the Stabilization and Reconstruction Task Force and the Global Peace and Security Fund. It enables Canada to take rapid and coordinated action to prevent and respond to emerging and ongoing instances of violent conflict and state fragility. The Program leverages resources of, and partnerships with, multilateral institutions, non-government organizations, implementing partners, and like-minded nations. PSOPs’ core responsibilities are to:

• Provide policy leadership on peace and stabilization, including strategic coordination of Government of Canada responses to complex political crises on an as-needed basis; and
• Deliver gender-responsive conflict prevention, stabilization, and peacebuilding initiatives through both programming and deployments.

At ¶¶ÒùÊÓƵ, the Peace and Stabilization Operations Bureau (IRD), under International Security (IFM), is responsible for the management of PSOPs through the following four divisions:

• Programming Division (IRG) delivers grants and contributions in geographic and thematic priority areas in accordance with PSOPs’ Terms and Conditions.
• Strategic Planning and Coordination Division (IRC) acts as the corporate secretariat for PSOPs and leads strategic planning, reporting, communications, performance measurement, and training. The division also maintains the PSOPs business process and provides guidance for program delivery, including the integration of gender equality, innovation, and risk management.
• Conflict Prevention, Stabilization, and Peacebuilding Division (IRZ) provides the Department and the Government of Canada with a centre of expertise in conflict prevention, stabilization and peacebuilding.
• Peace Operations Policy and Deployments Division (IRP) consolidates leadership on all policies and contributions to international peace operations, including police and civilian deployments, and is responsible for coordinating efforts with the Department of National Defence, the Canadian Armed Forces, and the Royal Canadian Mounted Police.

The PSOPs financial assistance envelope is $150 million annually, including $117.8 million for grant and contribution agreements. Potential recipients include: Canadian international and local non government organizations, multilateral organizations (e.g., United Nations agencies), academic institutions and foreign governments - affected States and donor countries (including their institutions, organizations, and agencies), Canadian provincial, territorial, and municipal governments, and Crown corporations^{Footnote 1}.

Grant agreements/arrangements are used with well-established international partners, primarily multilateral organizations of which Canada is a member. Contribution agreements/arrangements are most often used with local non government organizations, professional organizations, and some private sector organizations, but may occasionally be used with multilateral institutions. These transfer payments are subject to performance conditions specified in the implementing instrument. Contribution allocations are based on the project activities, budget, and the capacity of the organization to deliver the project.

The PSOPs Strategy is aligned with Canada’s priorities and departmental commitments. It includes, amongst others, commitments for gender equality, peace and security, conflict prevention and mediation, the Women, Peace and Security agenda, and engagement in the Middle-East. The PSOPs Strategy also aligns with the Feminist International Assistance Policy, as peace and security is one of the six action areas. The Program’s Strategy identifies priority countries in which to focus efforts.

To provide timely interventions in conflict prevention and stabilization in fragile and conflict affected states, PSOPs developed its own unique business process and Terms and Conditions, which allows for agility and flexibility. It also allows the Program to change focus when conditions evolve and to address emerging priorities. Most of the grants and contributions are delivered through unsolicited mechanism whereby potential recipients submit a concept paper requesting funding for a project under PSOPs.

PSOPs works with numerous domestic partners to implement its mandate, including: the Department of National Defence, the Canadian Armed Forces, Public Safety Canada, the Royal Canadian Mounted Police, the Department of Justice Canada, Correctional Service Canada, and other Government of Canada experts.

2. Observations and recommendations

This section sets out key findings and observations, divided into four general themes: Program Governance and Oversight, Risk Management, Program Delivery, and Monitoring and Reporting. Recommendations are provided, where appropriate. Details related to the audit approach are included in Appendix A.

2.1 Program governance and oversight

Governance committees play an important role in overseeing program delivery and ensuring effective decision-making. Accountabilities, roles and responsibilities also need to be clearly defined, communicated and exercised to avoid duplication and inefficiencies in program delivery and oversight.

The PSOPs governance structure includes two boards:

• The PSOPs Policy Board which provides a program-level venue for leadership to address issues related to strategy, policy development, planning and execution of PSOPs’ mandate. Permanent members include the Director General and the Director of each division.
• The PSOPs Advisory Board which provides a formal Directors General-level forum for consultation on policy, priorities and good practices in preventing and responding in a whole-of-government manner to emerging and on-going situations of violent conflict and political crisis overseas. Its membership includes senior management from ¶¶ÒùÊÓƵ and other government departments.

The mandates, accountabilities, and roles and responsibilities are clearly defined and communicated. Program staff also confirmed that their accountabilities, roles and responsibilities were clearly defined and communicated to them. In addition, PSOPs Strategy and its business process define each division’s accountabilities as well as individual roles and responsibilities.

A review of meeting minutes and agendas of both governance bodies demonstrated that discussions were held on PSOPs’ strategy, program policy issues, reorganization of the organizational structure, and determination of priority countries. However, meeting occurrences and record of decisions were not consistently maintained. As such, the extent of oversight exercised by these two oversight bodies could not be adequately examined to ensure full implementation of PSOPs’ objectives and strategies.

Recommendation 1

The Assistant Deputy Minister of International Security (IFM) should require PSOPs to systematically document and track decisions of its governance bodies in order to demonstrate effective oversight exercised over the implementation of its mandate, objectives, and strategies.

2.2 Risk management

PSOPs operates in volatile and challenging environments where violence, corruption, and high crime rates are prevalent. Since results are more difficult to achieve and measure in this context, the Program must accept a certain degree of risk and have an agreed level of risk tolerance.

The PSOPs Risk Management Framework was established to identify, assess and mitigate, and monitor program risks. It provides details on the governance and accountabilities related to risk management within the Program. It also describes how to assess risks and establish requirements regarding the payment terms, the level of approval required, the type of arrangement, and monitoring and reporting for each risk category and risk level.

The Risk Management Framework is incorporated into the PSOPs Performance Indicator Profile, which is based on the key program risks outlined in the PSOPs 2018 Treasury Board Submission. In addition, the PSOPs Strategy includes a section on risks and risks responses at a program-level aiming to help programs address key operational risks (Political, Security, Recipient Capacity, and Reputational risks) for the PSOPs’ Performance Information Profile.

A PSOPs Risk Management Guide is also available, as a complement to the Risk Management Framework, to provide additional guidance at project-level for project officers. The Guide describes the risk-related steps required to manage risk through each project phase.

In 2016, after Financial Resource Planning (SWP) raised concerns about PSOPs’ risk management practices, the Program committed to monitor project-level risk management practices. The Strategic Planning and Coordination Division (IRC) then started to conduct an annual compliance “spot check” exercise. This spot check exercise consists of reviewing a sample of project files from the previous fiscal year to assess compliance against the requirements of the Risk Management Guide. The resulting report describes the findings and makes recommendations to reinforce areas of improvement and feeds into the process of updating PSOPs’ Risk Management Guide and other risk management documents, as appropriate. The Strategic Planning and Coordination Division (IRC) conducted annual spot check exercises in 2017, 2018, and 2020.

However, an examination of 20 sampled project files identified inconsistencies with the requirements set in the program risk documentation:

• While project officers conducted a risk assessment at the initiation and assessment phases for all 20 projects, there is no evidence that they continued to assess risks throughout the lifecycle of the project. However, it should be noted that the functionalities of PSOPs’ tool do not currently provide version control that would indicate if or when risks are being assessed during project implementation and mitigation strategies adjusted, when relevant as required by the Guide. If project officers do not assess risks throughout the lifecycle of a project or properly document their subsequent assessments after the initial one, it may lead to some risk updates and emerging risks not being tracked or effectively captured.
• Amongst the six (6) sampled projects that were required to complete a Recipient Financial Viability, three (3) of them did not complete it as prescribed. It means that the financial viability of the recipient may not have been properly assessed before signing the funding agreement, which would put the Department more at risk of investing funds without achieving program objectives.
• None of the 20 sampled projects included a formal PSOPs’ monitoring plan, as prescribed by the Guide. Preparing a monitoring plan reflecting the project risk level would allow for consistent monitoring of risks through the project duration.
• Reporting requirements defined in the financial instrument and subsequent amendments reviewed did not always reflect the overall risk rating of the project. This lack of consistency may weaken the program’s risk management strategy, especially when higher risk projects have less restrictive requirements than low-risk ones, and create unnecessary burden to the recipient.

The observations provided in this section are similar to the results of the annual spot check. For instance, the 2020 spot check exercise recommended that Deputy Directors should carefully review all project risks identified to ensure in-depth analysis of risk responses and review all risks in the fiduciary risk category to ensure they are completed correctly, with fulsome descriptions and documentation of the required recipient. In addition, the 2018 spot check recommended that project officers use the PSOPs Risk Management Guide more extensively. Finally, the last spot check exercise reiterated many findings from the first assessment conducted in 2017.

Recommendation 2

The Assistant Deputy Minister of International Security (IFM) should:

• Establish measures to improve and better document program compliance with the PSOPs Risk Management Guide; and
• Implement the recommendations formulated in the spot check exercises and monitor their effectiveness.

2.3 Program delivery

Key elements, such as resources, procedures, guidelines, and tools are important to support the delivery of PSOPs in a timely, effective, and consistent manner. Without these key elements, the Program may not be able to achieve its objectives.

Human Resources Capacity

In order to deliver projects in an effective and efficient manner, PSOPs management needs to maintain the appropriate human resources capacity, which means the right number of employees with the required qualifications. In recent years, PSOPs experienced challenges to fill all its positions with qualified people. As of January 3, 2020, PSOPs had 27 vacant positions out of 86. However, the PSOPs Strategy has a section on talent management acknowledging the current situation and the Program undertook a mobility planning exercise to plan for future needs. Not having the appropriate level of full-time equivalents may have negative impacts on staff and its ability to deliver quality service/program in a consistent manner.

Program Business Process

The Program developed and implemented a business process to guide project officers throughout the life cycle of a project. Project officers are expected to adhere to this guidance so that all projects are processed in a standardized manner and align with PSOPs’ objectives.

Project officers are expected to complete specific steps such as:

• Environmental Initiative Assessment form signed by a Deputy Director;
• Anti-Terrorism Due Diligence and/or Grant Due Diligence (if applicable);
• Gender Equality/Women Peace and Security section in the Security and Stability Project Management Tool;
• Team summary report detailing the overall application’s assessment;
• Project budget review by a Financial Management Advisor; and
• Due Diligence Confirmation form, including the risk assessment and key information on the project objectives.  

Based on the review of the 17 sampled projects out of 20 that the Program funded through the unsolicited mechanism, project officers did not fulfill all the listed requirements. For instance:

• In three (3) projects, there was no evidence that a Financial Management Advisor reviewed the budget;
• In four (4) projects, the team summary report was not on file;
• In four (4) projects, the Environmental Initiative Assessment form was incomplete. When it was completed (in seven (7) projects), four (4) of them did not include the appropriate authority’s wet signature to validate the project officer’s analysis.

If project officers do not comply with the business process requirements and exercise due diligence, some projects may not meet PSOPs’ objectives.

Timeliness of project delivery

PSOPs defines itself as a rapid-response program that purposely carries out a distinct business process that takes into account the need to ensure the agility, flexibility, and higher risk tolerance required for timely and catalytic interventions in conflict prevention and stabilization in fragile and conflict-affected states. It is therefore important for the Program to measure its response time to ensure efficiency and effectiveness of project delivery. The Program did not measure timeliness of project delivery.

The audit examined 239 PSOPs projects covering the 2016-2017 to 2019-2020 fiscal years to determine if it was possible to measure timeliness for these projects. The audit identified key process steps and analyzed the overall average length of time between these key steps.

The key process step which appeared to have taken the longest time to complete was between the Project Accountability Team signatures in the Management Summary of Intent and the Assistant Deputy Minister’s signature in the Approval Memorandum. On average, it took around 3.8 months between those two points. A more detailed analysis would be required to explain the length of time between key process steps and reasons for potential delays.

The audit also examined potential impacts certain variables had on the timeliness of project delivery, including financial instruments (contribution versus grant), recipient types, nature of the project, and fiscal year. There was no significant differences when comparing average time between process steps based on these variables. Again, further analysis would be required to identify other possible factors that affect timeliness.

Measuring the timeliness of a response would provide PSOPs with the ability to understand and better manage project implementation through identification of process stranglehold to resolve them and improve strategic planning to determine appropriate timing to launch projects.  Furthermore, if the recipient obtains funding in a timely manner, it would improve the achievement of the program’s objectives.

Recommendation 3

The Assistant Deputy Minister of International Security (IFM) should:

• Establish measures to improve and better document program compliance with the PSOPs business process; and
• Develop the capacity to measure and monitor the timeliness of project funding implementation to ensure efficiency and effectiveness of project delivery.

2.4 Monitoring and reporting

It is important that project officers monitor projects closely to ensure that they align with PSOPs’ mandate and Government of Canada priorities throughout their life cycle, the results achieved meet program objectives, and the recipient spent the allocated funds appropriately in compliance with the terms and conditions of the funding agreement. PSOPs senior management also needs to receive the most complete, accurate, and up-to-date information in order to make informed decisions to ensure that PSOPs fulfills its mandate and meets its objectives.

Monitoring

Almost all of the sampled projects were aligned with PSOPs’ mandate and all of them included monitoring mechanisms in the funding agreement. However, there is insufficient evidence demonstrating that project officers monitored project activities. In addition, some key financial documents were also missing from the electronic files, which could prevent project officers from exercising proper financial due diligence.

Nineteen (19) of the 20 sampled projects aligned with the Program’s mandate. The only exception was politically driven, which is beyond the Program’s control. As for the funding agreements, all of them included monitoring mechanisms to enable project officers to monitor project activities. Nineteen (19) of the 20 funding agreements reviewed indicated the frequency and content of the reports that the recipient needed to provide during the project life cycle as well as the conditions of payment with the supporting documentation to be provided. The remaining funding agreement did not require the provision of financial reports.

Based on the review of the 19 sampled projects that required reports, all the required reports were in the electronic files but there were insufficient evidence to determine if and how project officers reviewed these reports. For 12 projects, the project officer had documented its review of financial and narrative reports. However, the only evidence of review on file was clarification emails between program officers and recipients. For the remaining seven (7) projects, the project officer did not document its review at all. It should be noted that neither the PSOPs’ Terms and Conditions nor the business process include formal requirements to document the review of reports. In order to strengthen monitoring practices, the project officer’s review of financial and narrative reports could be more systematically documented. If project officers do not document their review of narrative and financial reports, they cannot clearly demonstrate that the recipients achieved PSOPs’ objectives and spent the allocated funds as required by the funding agreement.

In addition to the aforementioned reports, the recipient also has to provide a completed payment request, including a cash flow statement outlining how funds are going to be used, to receive payments. For four (4) of the 20 sampled projects, the project file did not include a cash flow statement. If project officers did not systematically save cash flow statements in their electronic file, it is difficult to assess whether payments were made as planned and for the intended purposes.

Reporting

The Performance Information Profile is the tool supporting senior management in coordinating program performance information requirements. The Performance Information Profile outlines the outputs, outcomes, and indicators used to measure, manage against, and report on the Program’s performance. The Program’s Performance Information Profile provides a performance measurement framework to measure and assess performance against expected results and articulates how PSOPs programming implements departmental priorities and action areas identified in the Feminist International Assistance Policy.

As per departmental international assistance annual reporting requirements, the Program must complete Management Summary Reports. The Management Summary Reports are used to provide outcomes information to meet several reporting requirements, such as the Report on Canada’s International Assistance, and support senior management’s decision-making. As such, project officers report on project results using the standard report template and data from reports provided by recipients.

At the time of the audit, the most recent Management Summary Reports were for 2018-2019. Fourteen (14) of the 20 sampled projects required a Management Summary Report but five (5) projects did not have one. For the remaining nine (9) projects, three (3) had full and complete performance information while six (6) only had minimal information. If project officers do not fully complete the required reports, the quality of the information for decision-making will be incomplete and senior management will not have full information and be able to exercise appropriate oversight over how funds are used to achieve program and departmental objectives.

Recommendation 4

The Assistant Deputy Minister of International Security (IFM) should put measures in place to:

• Document project files, including monitoring activities, in a complete, consistent, and effective manner; and
• Properly review and report performance information as per departmental requirements.

3. Conclusion

An effective management control framework has been established for the Peace and Stabilization Operations Program. A few areas for improvement were identified to further improve the effectiveness of the framework such as:

• Consistent application of the PSOPs Risk Management Guide and business process tools;
• Sufficient and consistent documentation of monitoring and reporting activities; and
• Developing a system to measure the timeliness of project implementation to respect the program rapid-response mandate.

Appendix A: About the audit

Objective

The objective of this audit was to determine whether an effective management control framework had been implemented for the Peace and Stabilization Operations Program to support the achievement of the Program’s objectives.

Scope

This audit examined PSOPs’ management control framework, including program planning and funding, project delivery and monitoring, and performance measurement and reporting activities. The audit focused on the last three fiscal years (2017-2018 to 2019-2020). However, to gain a more complete understanding of the subject matter of the audit, relevant material outside this period was also analyzed, most notably projects that started in 2016-2017 and were still active during that period. The audit focused on the activities of the Programing Division (IRG) and the Strategic Planning and Coordination Division (IRC), more specifically programming activities and the support provided to project officers. Interviews were conducted and documentation was reviewed up to and including September 2020.

Criteria

The criteria were developed following the completion of the detailed risk assessment and considered the audit criteria related to the Management Accountability Framework developed by the Office of Comptroller General of the Treasury Board Secretariat. The audit criteria were discussed and agreed upon with the auditees. The detailed criteria are presented as follows.

Audit Criteria are reasonable and attainable expectations against which compliance, the adequacy of controls and overall performance are assessed. These audit criteria are based on acts and regulations, policy, guidelines, generally recognized industry norms, results of previous audits or other criteria developed in consultation with Program management. The following criteria were assessed during this audit and form the basis for developing audit observations and recommendations.

Met – the audit evidence sufficiently supports the audit criteria’s expectations as being met (for example, appropriate controls found (and are commensurate with the risk) and they are operating effectively).

Partially met – the audit evidence mostly supports the audit criteria’s expectations but improvements are required – (for example, some controls in place and some are operating effectively – recommendations made)

Not met – the audit evidence is insufficient to support the audit criteria’s expectations – (for example, controls not in place or not operating effectively – risk exposure is high - recommendations made – other action taken as required)

Approach and Methodology

The audit was conducted in conformity with the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing and with the Treasury Board Policy and Directive on Internal Audit. These standards require that the audit be planned and performed in such a way as to obtain reasonable assurance that the audit objective was achieved. 

To conclude on the above criteria, the following methods were used to gather evidence:

• Identify and review relevant policies, directives, guidelines and operational standards;
• Review and analyze financial and non-financial information;
• Conduct walkthroughs of key processes to identify and assess the effectiveness of key controls;
• Test selected projects through file reviews as required;
• Conduct interviews with departmental officials, stakeholders, and recipients;
• Perform data and trend analysis; and,
• Conduct other tests and document review as deemed necessary.

The audit selected a sample of 20 projects (including five (5) grants and 15 contributions) that were active within the last three fiscal years (2017-2018, 2018-2019, and 2019-2020) to assess whether the Program met the audit criteria. However, it should be noted that due to the COVID-19 pandemic, the audit did not have access to departmental facilities where the Program stores its hard copy files. Therefore, the audit cannot comment on the structure, content, and relevance of these hard copy files.

See Appendix C for the list of sampled projects.

Appendix B: Management action plan

Appendix C: Projects selected as part of the sample (20)