¶¶ÒùÊÓƵ

Evaluation of International Assistance Programming in Colombia, 2011-12 to 2017-18

Final Report

International Assistance Evaluation Division (PRA)
¶¶ÒùÊÓƵ
November 1st, 2018

Table of Contents

• Acronyms and Abbreviations
• Executive Summary
• Program Background
  • Colombian Context
  • Donor Context
  • ¶¶ÒùÊÓƵ Programming
• Evaluation Scope and Methodology
  • Evaluation Scope
  • Purpose of evaluation
  • Evaluation Questions
  • Methodology
• Findings
  • Responsiveness and Agility – Internal context
  • Responsiveness and Agility: External Context
  • Results Achieved: Sustainable Economic Growth
  • Results Achieved: Children and Youth
  • Results Achieved: Peacebuilding
  • Conflict-Sensitive Programming
  • Results Achieved: Gender Equality and the Empowerment of Women and Girls
  • Good Practices in Integrating Gender Results
  • Sustainability
  • Program and Policy Coherence
• Conclusions
• Recommendations and Management Responses
• Considerations for Horizontal Learning
• Appendices
  • Appendix 1: Colombia Program Logic Model
  • Appendix 2: Methodology – Data Limitations
  • Appendix 3: Evaluating Coherence

Donor Context

Bilateral Relations

Colombia is a key hemispheric partner for Canada with a close, growing and multi-faceted government-to-government relationship. Priorities for bilateral engagement include dialogue on human rights, mobility between the two countries, people-to-people relationships and cooperation on multilateral issues.

Trade and investment has expanded in recent years, facilitated by the implementation of the 2011 Canada-Colombia Free Trade Agreement. In 2016, bilateral trade reached $1.6 billion and two-way foreign direct investment stood at $3.7 billion. Colombia is a founding member of the Pacific Alliance trade bloc, of which Canada is an observer.

Official Development Assistance to Colombia

In 2016, net Official Development Assistance (ODA) to Colombia totaled US$1.1 billion, representing 0.4% of gross national income. Despite its low dependency on ODA, Colombia was the top ODA recipient in South America from 2014 to 2016.

In 2016, Canada was the fifth largest bilateral donor in Colombia, following France, the United States, Germany, and the United Kingdom. Canadian ODA to Colombia represented 3.2% of total gross ODA disbursed.

Donor Coordination

The country’s main bilateral and multilateral partners harmonize their development efforts and investments through a coordination group, known as Grupo de Cooperantes (GrUC). The group aims to align donor aid to priority areas identified by the government of Colombia: peacebuilding, economic development, transitional justice, human rights, gender equality, and the environment. Canada chaired the GrUC in 2010 and 2017.

Canada participated in a number of other donor sub groups and coordinating mechanisms including: the Gender Roundtable, Human Rights Sub-committee, National Roundtable on Rural Education and Post-Conflict and the Group of Friends of UNSC Resolution 1612 on Children and Armed Conflict.

¶¶ÒùÊÓƵ Programming

Program Disbursements

¶¶ÒùÊÓƵ delivers international assistance through a range of development, humanitarian, and peace and security initiatives. Between 2011-12 and 2017-18, international assistance disbursements to Colombia totalled $298 million, with an average yearly disbursement of nearly $43 million.

Americas Branch (NGM)

Almost two-thirds of international assistance spending ($188 million) was disbursed through NGM. Disbursements were largely allocated across development projects in areas of sustainable economic growth ($83 million) and children and youth ($82 million), with remaining disbursements allocated to peacebuilding initiatives ($23 million). Disbursements peaked in 2016-17 with an increase in support for the implementation of the peace accord.

International Security and Political Affairs Branch (IFM)

Peace and stabilization programming^{Footnote 1} accounted for over three-quarters of IFM disbursements ($42 million). Initiatives included short to mid-term projects on transitional justice, security, reconciliation, peace monitoring, demining and transformation of Colombia’s army. Through its Planning and Development Division, the Peace and Stabilization Operation Program (PSOPs) also supports police and civilian deployments.

Global Issue and Development Branch (MFM)

MFM disbursements ($32 million) largely focused on humanitarian assistance. This included projects to provide access to safe water, adequate sanitation and hygiene systems and basic health care services in conflict-affected areas.

Partnerships for Development Branch (KFM)

KFM disbursements ($22 million) focused on multi-country initiatives with a strong presence of youth internship and volunteer programs. Partners included Oxfam-Quebec and Cuso International, among others.

Evaluation Scope and Methodology

Evaluation Scope

• covered the period from FY 2011-12 to FY 2017-18.
• considered both project and non-project activities (e.g., policy dialogue and advocacy, donor coordination).
• included all bilateral development projects supporting the bilateral development program logic model (Annex I); administrative support funds and the Canada Fund for Local Initiatives were not included. 
• included a purposive sample of projects supported through partnership programming (KFM), humanitarian programming (MHI) and peace and stabilization programming (PSOPs and START). Security projects from the Anti-Crime and Capacity Building Program were not included due to low disbursements over the evaluation period.

Purpos of evaluation

• to contribute to informed decision-making, support policy and program improvements and advance departmental horizontal learning.
• to provide an evidence-based neutral assessment of international assistance in Colombia to Canadians, Parliamentarians, Ministers, Central Agencies, ¶¶ÒùÊÓƵ’s management, partners and beneficiaries.

Evaluation Questions

Responsiveness and Agility

1. To what extent has international assistance programming responded to evolving needs and opportunities in Colombia?

Results, Gender Equality and Sustainability

2. To what extent has international assistance programming contributed to expected development outcomes in areas of sustainable economic growth, children and youth and peacebuilding?
3. How has programming contributed to gender equality results and the empowerment of women and girls?
4. Is there evidence that results have been sustained in areas where support has ended? What is the likelihood that the results achieved will continue?  

Program and Policy Coherence

5.  To what extent has coherence across international assistance programming (development, humanitarian, peace and stabilization) improved? Across other areas of departmental engagement (international assistance, trade, diplomacy)?

Methodology

The evaluation was conducted by the International Assistance Evaluation Division (PRA), with the support of an external consultant with regional and gender expertise. Two data collection missions to Colombia were conducted in early 2018.

Details on data limitations can be found in Annex III.

• Key Stakeholder Interviews (n= 96)
  • Semi-structured individual and small group interviews, comprised of:
    • ¶¶ÒùÊÓƵ management and current and former staff (n = 28)
    • implementing agencies (n = 21)
    • local partners, including private sector (n = 23)
    • Government of Colombia (n = 4)
    • Other donors and international agencies (n = 6)
    • CSOs and/or academics experts (n = 8)
• Project Review (n = 39  projects)
  • Assessment of project implementation and results (22 NGM, 6 KFM, 7 PSOPs and 4 humanitarian projects. The review of the humanitarian projects a total of 14 one-year projects implemented by 4 different humanitarian agencies over multiple years.
  • Systematic review of all available project documents including:
    • approval and implementation documents
    • monitoring, performance  and management summary reports
    • evaluation reports.
• Project Financial and Timeline Analysis
  • The review included the universe of international assistance programming in Colombia. The analysis included project disbursements, gender integration, implementing partners, OECD-DAC sectors and project duration.
• Site visits and Focus Groups with Project Beneficiaries (n=23)
  • Project site visits included 23 focus groups with beneficiaries of seven different projects (4 development, 2 humanitarian, 1 PSOPs). Site selection was geographically planned to focus on three different conflict-affected regions
  • Focus groups included a total 227 participants (130F/ 97M).
  • Feminist evaluation methods were piloted in collaboration with Colombian evaluators in two project site visits.
• Document Review
  • Review of internal departmental policy/strategy documents as well as external documents pertaining to the political, economic and social context of Colombia along with other donor evaluations.

Findings

Responsiveness and Agility – Internal context

An early and sustained focus on programming in areas of importance to the peace process helped to build the capacity to respond to new needs and opportunities.

As Colombia’s context evolved, ¶¶ÒùÊÓƵ responded by maintaining continuous programming in some sectors and adapting programming to new developments in others.

In the decade prior to the 2016 peace accord, ¶¶ÒùÊÓƵ began establishing programming in areas of strategic importance to the peace process. Niche sectors, such as rural education, were selected where few donors were active. Early and long-term support for de-mining initiatives and building transitional justice mechanisms and institutions was provided. Programming was geographically focused, targeting some of the most conflict-affected regions in Colombia.

As peace negotiations advanced and Colombia’s economy strengthened, new opportunities emerged. Canadian expertise in areas of rural economic development, including agricultural cooperatives and rural finance, was leveraged. Implementing partners shifted from multilateral organizations to Canadian NGOs. Some Canadian partners developed knowledge of the Colombian context and established credibility through small partnership projects (KFM) before implementing large bilateral projects.

Stakeholders, including government and local partners, noted that ¶¶ÒùÊÓƵ staff, particularly locally-engaged staff, brought deep knowledge and understanding of the Colombian context to decision making. There was a strong consensus that Canada was in the “right” areas with programming tailored to Colombia’s needs and priorities.

In 2016, Canada provided timely and significant support for the implementation of the peace accord^{Footnote 2}. A rapid response was possible in part because implementing partners had established the credibility and capability to address priority areas of the peace agreement. An early and substantive investment in the United Nations Multi-Partner Trust Fund (UN MPTF) signaled Canada’s support for the peace process and helped to leverage funds from other donors. This elevated Canada’s profile and increased opportunities to engage in strategic discussions with government partners, multilateral agencies and other donors.

Following the signing of the peace accord, many donors began to decrease humanitarian support despite an increase in needs. Canada’s continued humanitarian support was recognized by coordinating agencies and implementing partners as responding to immediate and unmet needs and helping to bring attention to humanitarian issues.

Approaches to international assistance were aligned with good practices for donor engagement in middle-income countries and fragile states.

Responsiveness and Agility: External Context

The Colombian context provided an enabling environment to test new and innovative models for international assistance.

A number of contextual factors provided the preconditions for testing new modalities and approaches in Colombia including:

• a relatively strong state;
• a sound regulatory and legislative environment with good legal and justice institutions;
• a well-resourced private sector with financial requirements for corporate social responsibility;
• a comprehensive and inclusive peace accord;
• a highly educated and competent human resource base; and
• a robust civil society.  

The fragile context of peace, alongside challenges in implementing the peace agreement, limited the responsiveness and agility of programming.

An increase in conflict and instability in the months following the signing of the 2016 peace accord limited how and where projects worked. For instance, humanitarian partners found they could no longer access vulnerable populations in some regions as it was not possible to negotiate with new and emerging groups of armed actors. Further, interviewees representing local partners and project beneficiaries noted the fragility of the peace process as their highest concern.

While strong formal institutions were put in place to address priority areas of the peace agreement, the ability of the state to implement strategies and consolidate peace was perceived to be weak. Little progress was made in building state legitimacy and trust in peripheral regions. While interviewees noted that a genuine transition to peace will take years, contributing factors to the slow implementation of the peace accord included:

• revolving leadership of key institutions;
• a continued lack of state capacity and presence in conflict-affected rural areas;
• a limited capacity of territorial and local governments; and
• a weak articulation of national and territorial governments. 

While the suite of international assistance programming provided mechanisms to address different types of needs, some institutional barriers to responsiveness and agility were identified.

The bilateral development program responded to long-term structural needs, with the bulk of projects being multi-year and of high materiality. While stable support was valued, there was a perception among internal and external stakeholders that responsiveness was limited by risk aversion and difficulties in adapting projects to changing contexts. The need for flexible funding mechanisms to identify and fund local priorities was noted.

In general, programming in fragile and conflict-affected states requires a tolerance for risk, with literature pointing to the need for donors to use iterative and adaptive approaches. This includes building systems for feedback and continual learning. Examples of intra-project flexibility were noted as some projects adapted to changing contexts by shifting timelines and including new partners, activities and geographic regions. However, there were mixed perceptions amongst both ¶¶ÒùÊÓƵ staff and implementing agencies on the extent to which contracting mechanisms and results-based management frameworks allowed for flexibility. Further, as conflict and tensions increased in the post-accord period, the need for guidelines and support for both the identification of acceptable levels of project risk and the development of strong mitigation strategies was identified.  

Peace and stabilization programming provided a responsive and agile mechanism to address peacebuilding needs. However, slow project approvals throughout the evaluation period limited the agility of these programs. Average project approval times peaked at twelve months in 2015 under the START Program. This left some partners with little time to implement planned activities and left significant portions of planned budgets unspent. Additionally, short project time frames created gaps in programming in some areas (e.g., transitional justice) while partners waited on approvals for subsequent phases. One partner was able to ensure the continuity of some activities through longer-term and complementary partnership (KFM) funding.

Humanitarian programming responded to immediate needs resulting from conflict and displacement. Agencies noted that providing multi-year funding to humanitarian agencies operating in contexts of protracted crisis, such as Colombia, may have the potential for increasing the effectiveness of their work. 

Results Achieved: Sustainable Economic Growth^{Footnote 3}

Some progress was made in strengthening the participation of vulnerable groups in formal economic activities

Sustainable economic growth (SEG) projects contributed to the economic advancement of vulnerable groups by improving productive and employment opportunities, creating linkages with markets and the private sector and facilitating access to credit.

Some projects specifically targeted rural youth, as a means of both strengthening local economies and incentivizing youth to remain in rural communities. Support for youth leadership and entrepreneurship led to the creation of new businesses and spin-off effects. On average, each new business led to an additional 2.5 jobs with 84% of participants reporting significant increases in their income and improvements in their family’s living conditions. Challenges in reaching the most vulnerable youth were identified, as educational requirements created barriers for participation in training activities.

Other projects focused on strengthening cooperatives and producer associations. The adoption of new production practices and post-harvest techniques improved the quality and value of agricultural products. This, alongside an improved ability to collectively negotiate more favorable prices, led to increased incomes. While significant progress was made, producers faced challenges to fully engage in associations and improve market connectivity. Changing attitudes and practices to support cooperative models was slow, particularly given legislative barriers for the use of collective loans and insurance. A recent project evaluation recommended increasing the length of value chain projects from 5 to 8 years to allow sufficient time for producers to fully commercialize and connect with markets.

Projects supported the initial steps towards women’s economic empowerment.

Some projects contributed to improved understanding of gender relations and increased participation of women in decision-making. Associations adopted gender plans and quotas for their boards leading to greater representation of women. In some cases, the use of manager apprentice programs helped equip women with the leadership and business skills needed to assume managerial roles. Less progress was made in improving women’s access to and control of value-added productive activities. Significant barriers for gender equality remained, with project participants noting that a long-term view was required to change deep-seated cultural and socioeconomic barriers.

The bilateral development program defined the most vulnerable groups as women and girls, afro-Colombians, indigenous peoples, internationally displaced persons and people living in rural areas.

Engaging public and private sector actors increased the effectiveness and reach of projects.

The ability to bring together private and public sector actors, to collaborate in new and innovative ways, was identified as a key factor for project success. Some projects piloted approaches to demonstrate models that could be both socially and financially viable. Interest from private sector actors was high, as opportunities to invest in social development and leverage government policies supporting corporate social responsibility and tax incentives increased.

In some cases, blended finance approaches leveraged significant private sector resources. For example, one SEG project supporting agricultural cooperatives secured an additional $10 million in private sector contributions and $3 million in public sector contributions. While increasing the diversity and sources of financing was leading to expanded project reach and improved stakeholder buy-in, challenges were identified. Implementing agencies noted difficulties in addressing the requirements of multiple donors with different institutional cultures, interests and reporting requirements.

Projects contributed to strengthening local-level governance through increased participation of marginalized groups.

Governance-related programming aimed to improve processes for planning, managing and allocating resources at territorial and municipal levels and to increase the participation of traditionally marginalized groups in these efforts.

Pilot approaches for integrated rural economic development included training and capacity-building on decentralized and participatory planning, financial transparency and accountability. The creation of multi-stakeholder governance structures bringing together a wide range of public and private institutions, civil society organizations, and local, territorial and indigenous governments helped to increase the participation of women, afro-Colombian and indigenous groups in decision-making processes. This improved the ability of municipalities and territories to plan and execute rural development plans. An emphasis on transferring governance “toolkits”, documenting processes and training materials, helped to foster sustainability.

Canada made significant contributions to policy dialogue and advocacy relating to rural women’s development and empowerment as co-chair of the donor coordination group on gender equality. ¶¶ÒùÊÓƵ’s contribution to the development of a national policy on rural women, the establishment of the Rural Women Directorate in the Ministry of Agriculture and support for rural women’s organizations were noted. Opportunities to strengthen work with women’s organizations and better link projects with municipal and territorial gender plans and committees were identified.

Support for extractives sector governance promoted the equitable distribution of benefits.

Extractives sector programming aimed to build the decision-making capabilities of local government and communities to engage with private sector actors. Capacity building efforts focused on transactional negotiations based upon sound development techniques and transparency. There was broad agreement among stakeholders that this offered the best path forward for sustainable development, providing an alternative to the traditional, one-off “patronage” relationships.

Community-level projects demonstrated positive linkages between private sector extractive companies, local government and communities. A trend toward local ownership both with the local government and participatory community processes was emerging, but sustainability is dependent on practices becoming entrenched. While there were some indicators of success, projects were still at the early stages of realizing outcomes.

Results Achieved: Children and Youth^{Footnote 4}

Children and youth programming contributed to improved access and quality of education in rural areas.

¶¶ÒùÊÓƵ’s contribution and long-term commitment to rural education was widely recognized. Projects achieved significant results despite implementation challenges caused by external events, such as deteriorating security situations and teacher strikes. Activities to improve the quality of education led to some improvements in test scores in core subjects. Efforts to increase safe access to school for youth at risk helped to provide alternatives for girls and boys that were targeted for recruitment in armed conflict or illegal activities related to narco-trafficking. Additionally, support for financial planning and local education plans contributed to building institutional capacity.

Good practices for increasing access to educational opportunities included the development of an afro-Colombian flexible education model. The curriculum, built through the leadership of afro-Colombian teachers and community organizations, incorporated stories, tales and ancestral knowledge that reflected local culture. A flexible delivery approach enabled vulnerable youth and adults, particularly women, to receive accelerated schooling while staying in their communities. The transformative nature of the model was recognized by teachers, with some participants attributing their empowerment as community leaders to the learnings they gained.

Capacity-building activities included the introduction of new teaching methodologies. One technique, known as RESPIRA, promoted the use of breathing and emotional awareness techniques in the classroom. Use of this methodology resulted in teachers reacting less aggressively, students reporting less sad and anxious feelings, improved communication, reduced conflict in the classroom, and increased levels of attention. Some interviewees noted that the benefits of using RESPIRA extended beyond the classroom, helping to decrease conflict and family problems outside of the school.

Projects promoted educational activities to increase awareness of sexual and reproductive rights.

Activities promoting gender-sensitive and inclusive schools included the development of strategies to increase knowledge and awareness of gender equality, girls’ rights and empowerment, and sexual and reproductive health and rights. Students identified issues through anonymous submissions and various methods, including theatre, were used to actively engage teachers, students and parents. Focus group participants indicated that some positive changes in attitudes and behaviours in schools and in households were being observed during the first year of activities. Recognition of the approach led to plans for the Ministry of Education to integrate the strategy into curriculum more broadly.

Support for increasing awareness of the rights of children and youth contributed to adoption of new plans, policies and resolutions.

Projects worked with state institutions and civil society organizations to build awareness of the rights of children and youth. In some cases this helped put the needs of children and adolescents on the public agenda. For example, one project increased awareness of the needs of youth using a rights framework. Capacity-building activities targeted public agencies across different sectors, including health, protection, planning and budgeting. These efforts contributed to the adoption of legislation in five departments (regions) and six municipalities. Additionally, activities targeting schools led to the implementation of institutional improvements with a rights focus in 168 public schools. In some cases, increased awareness of rights contributed to a loss of fear and the denunciation of abuses.

Other projects focused on building the capacity of civil society organizations, including women’s groups and youth groups, to advocate for improved services for conflict-affected and vulnerable children and youth. Advocacy efforts led to the successful adoption of three policies in the areas of women, youth, and child protection in the municipality of Tumaco and their incorporation in the municipal development plan. Further, the process helped to build trust in public authorities and empower civil society organizations. The project also increased awareness of child sexual tourism. This led to changes in perceptions and reporting practices in Cartagena, in part through collaboration with local media. As a result, the number of sexual commercialization cases reported increased from an average of 4 cases to 243 cases per month (3/4 girl victims, 1/4 boy victims).

¶¶ÒùÊÓƵ actively participated in policy dialogue and donor coordination activities relating to children and youth. Canada was one of the original members of a  “Friends of 1612” group, convened to support UNSC Resolution 1612 on children and armed conflict. Colombia’s adoption of the resolution was attributed in part to the advocacy work of this group. Canada also chaired the National Roundtable on Rural Education and Post-Conflict, making significant contributions to the improvement of rural education policies and plans. While the role of ¶¶ÒùÊÓƵ in convening stakeholders and enabling implementing partners to actively participate in policy dialogue tables was recognized, some stakeholders noted that the impact of activities was hard to define.

Results Achieved: Peacebuilding^{Footnote 5}

Early results were emerging from initiatives announced in support of the implementation of the peace accord.

Key areas for the implementation of the peace accord, including humanitarian de-mining and transitional justice, were supported through both PSOPs and development programming.

Demining was initially supported by PSOPs projects that helped build the government's technical capacity, develop national standards and oversee operations by international agencies. Building on that groundwork, a development project supporting HALO Trust to clear landmines in 16 rapid response communities generated visible and fast impacts. In addition to reducing mine-related causalities, the project employed people from mine affected communities, facilitated land restitution and enabled the return of internally displaced people. Significant gender results were achieved as women were hired in non-traditional roles (e.g., working with explosives), helping to break stigmas in traditionally male-dominated fields. In areas of transitional justice, support for integrating gender and ethnic components into the transitional justice system was provided through peace and stabilization programming. These areas were reinforced through efforts to design and implement local justice systems, as part of the UN Multi-Partner Trust Fund. However, external stakeholders noted that restrictions on funding demobilized FARC ex-combatants limited the flexibility of Canadian support to the implementation of the peace accord and complicated multi-donor initiatives.

The peacebuilding context enabled new ways of working across international assistance initiatives.

Efforts to connect development, security and humanitarian initiatives increased following the signing of the peace accord. New spaces to share information and coordinate projects across international assistance streams were created. However, as conflict and insecurity persisted, stakeholders noted that the distinction between humanitarian and development interventions was blurred. The same populations often had complex needs, encompassing development, humanitarian and security needs at the same time.

Synergies across international assistance projects operating in the same geographic regions were identified. In some cases, the same beneficiaries first received support through humanitarian projects before participating in development projects. Participants noted that receiving mental health services through humanitarian projects was the first step in their empowerment. This enabled them to be mentally and physically prepared to participate in development projects focusing on education and entrepreneurship. In other cases, humanitarian de-mining initiatives supported through development and peace and stabilization programming enabled the safe entry of development projects supporting productive activities and agricultural cooperatives.

Conflict-Sensitive Programming

Projects across international assistance streams addressed drivers of conflict. The need for all projects to be conflict-sensitive was recognized.

A comparative analysis of project activities and results against known conflict drivers^{Footnote 6} was conducted to understand in what ways programming may be reducing the likelihood of conflict. The analysis found that projects contributed to addressing twelve conflict drivers. Activities such as strengthening the transitional justice system, supporting the recovery of victims of sexual and gender-based violence (SGBV), building trust in the police and growing the economy in conflict-affected areas had the potential to contribute to defusing conflict.

Internal and external interviewees emphasized the need for all projects to consider the context of conflict. This includes designing projects with an awareness of how activities could be impacted by conflict. For example, education projects considered safety at school, safety walking to school and landmine awareness as part of the overall project design. The importance of including psycho-social support components, supported by earmarked budgets, was identified in several projects. For instance, a SEG project aiming to promote inclusive employment provided candidates with access to psychologists at training facilities and worksites.

Information-sharing and collaboration with humanitarian agencies helped development partners to be better informed on the local dynamics of conflict.

Humanitarian agencies developed a deep knowledge of the local dynamics of conflict through years of working in a protracted crisis. This information, shared through coordinating mechanisms and informal networks, enabled ¶¶ÒùÊÓƵ staff and development partners to be better informed of changing contexts. Humanitarian principles, such as neutrality, as well as the “do no harm” principle were integrated into development project strategies to gain community acceptance and buy-in. One education project consortium leveraged comparative expertise by bringing together a humanitarian partner, focusing on access to education, with a development partner, focusing on quality of education. This helped to better identify and meet the needs of targeted groups and scale-up assistance.

Results Achieved: Gender Equality and the Empowerment of Women and Girls

Support for gender integration increased over the evaluation period, leading to improved gender analysis and reporting.

At the beginning of evaluation period, there was limited institutional support for effective gender integration. The bilateral development program did not have an operational gender strategy, adequate technical resources in the field or sufficient support from headquarters specialists.

¶¶ÒùÊÓƵ staff and partners faced challenges operationalizing the Feminist International Assistance Policy.

The introduction of the Feminist International Assistance Policy required a re-think of programming priorities, approaches and activities. Initial training on the policy was provided to Global Affairs staff and project partners. While the training raised awareness about women’s rights and empowerment issues, many partners wanted more guidance from ¶¶ÒùÊÓƵ on how to implement the policy.

Partners were unclear on the degree to which they were expected to adjust current programming. A better understanding of the difference between the gender-sensitive work they were doing versus a feminist approach was needed. Challenges in adding new components to projects and adapting existing activities in the absence of additional funding were noted. Specialized gender expertise was needed to develop training and capacity-building activities, to engage in advocacy initiatives and to strengthen gender-specific indicators. Some partners were concerned that a change in the focus and scope of their programming would put the sustainability of current results and established value-added niches at risk. Others noted long-term challenges to be faced in addressing cultural norms amidst growing resistance from conservative groups in Colombia.

Notable results to address sexual and gender-based violence were achieved by projects in different international assistance programming streams.

Sexual and gender-based violence (SGBV) was a significant problem in Colombia during the conflict and after the Peace Accord. A 2017 Oxfam report^{Footnote 7} estimated that there were over 875,000 female victims of gender-based violence from 2010-2015. Only 20% of those estimated victims reported the violence.

Eliminating SGBV is a priority in both the Feminist International Assistance Policy and Canada’s National Action Plan on Women, Peace and Security. ¶¶ÒùÊÓƵ disbursed over $15 million for SGBV-related projects through its development, humanitarian and peace and stabilization programming.

Canada’s contribution to addressing SGBV in Colombia was recognized by many stakeholders. However, a lack of common performance indicators and reporting across projects in different programming streams made it difficult to assess overall program and policy impact in this area.

Good Practices in Integrating Gender Results

Some ¶¶ÒùÊÓƵ projects developed good practices and innovative models to promote gender equality and women’s rights. These practices were shared and replicated by other partners in Colombia.

Sustainability

Strategies to foster sustainability included the design and transfer of new models of development.

Many projects focused on developing and piloting new models for delivering services and facilitating linkages between public and private sector actors. The uptake and replicability of these models by state institutions, financial organizations, and private sector companies was a key strategy for achieving sustainability and greater impact.

There were some promising examples of the early uptake of models by financial sector organizations and private sector companies. SEG projects that were successful in demonstrating both financial and social viability, such as the development of inclusive employment policies and practices, achieved high levels of uptake by Colombian companies. Further, the scaling out of approaches was being achieved through ownership and replication of training activities and methodologies by new partners, such as Chambers of Commerce.

Projects transferring models to state institutions faced some challenges. For example, the development of the afro-Colombian flexible model for education was recognized as a high-quality and effective model by the Ministry of Education. There was significant buy-in and the  project successfully transferred the model to the Ministry through a two-year process. The model was further adapted to deliver on specific education commitments for demobilized groups. However, a lack of political will to invest the necessary resources to implement the model more broadly in rural areas may limit its sustainability and potential for greater impact. Project partners and beneficiaries advocated for Canada to strengthen policy dialogue efforts aimed at increasing state attention to and funding for rural education.

Implementing partners noted that more attention could be placed on replicability and transferability strategies, particularly when targeting public sector actors. This could be facilitated by adjusting performance measurement targets to place less emphasis on meeting service-related targets (e.g., number of students reached by project) and more on efforts to develop, test and transfer financially-affordable models for service delivery.

Increased conflict and insecurity posed risks for the long-term sustainability of initiatives.

The fragility of the peace process was identified as a key factor impacting the long-term sustainability of initiatives. This was particularly evident in sectors such as rural education. For example, teachers interviewed in one conflict-affected region noted declining enrollment as families no longer felt safe allowing their children to travel to school. Partners and beneficiaries across all sectors called for an increased role for the international community to monitor the progress of the peace agreement and to strengthen policy dialogue efforts in support of the peace process.

Program and Policy Coherence

The evaluation considered coherence at two levels.  More details on the approach to assessing coherence can be found in Annex III.

• Coherence Within International Assistance: development, humanitarian, peace and stabilization
• Coherence Across International Engagement: international assistance, trade, diplomacy

Five core elements, identified as essential to supporting coherence, were assessed for each level:

1. Policy: There are complementary objectives and priorities guided by a shared policy suite.
2. Institutional: Engagement in Colombia is guided by a shared vision and strategy for how different parts of the department will individually and/or collectively achieve it.
3. Administrative: Formal planning, coordination and reporting tools are supportive of complementary and collaborative programming.  
4. Knowledge: Formal systems and tools are in place to enable knowledge sharing in support of coherent programming.
5. Informal: Informal networks of communication and collaboration foster collaboration and contribute to coherent programming.

Coherence Within International Assistance

There were strong examples of well-coordinated and complementary international assistance initiatives.

There was often willingness to collaborate among staff across different international assistance programming streams. The importance of cooperation and coordination was recognized, particularly given the complex needs arising from the peacebuilding process.

Within development programming, initiatives were well-coordinated within sectors of focus. For example, ¶¶ÒùÊÓƵ played a convening role in bringing together different implementing agencies in the SEG sector. This led to a greater complementarity of activities across different partners’ work with cooperatives.

Collaboration between development and humanitarian streams was noted as an area of strength, by both internal and external stakeholders. Regular communication between development staff in the field and humanitarian staff at headquarters, clear roles and responsibilities, and informal networks among partners contributed to these efforts.

While gaps in information-sharing were identified, communication efforts between START/PSOPs and both political officers and development staff improved over the evaluation period. This was facilitated by coordination of support for the peace package and extended field visits of PSOPs HQ staff.

This was facilitated by coordination of support for the peace package and extended field visits of PSOPs HQ staff.

While efforts to collaborate increased over the evaluation period, formal systems to promote knowledge-sharing and incentivize collaboration remained weak. Many staff noted that they were unaware of other branches’ activities and the universe of projects supported by the department.

Differing timelines for project approvals, multiple formats for reporting results and a lack of an accessible corporate database posed challenges for aligning activities and leveraging greater impact. This created barriers for the identification of policy dialogue opportunities, strategic partners and projects.

While there were some examples of public communication of individual projects, there were no integrated communications products to reflect the full portfolio of international assistance projects or provide an overview of how the different streams were collectively addressing ¶¶ÒùÊÓƵ’s priorities in Colombia. This made it difficult for government partners, other donors and the public to obtain useful information

Coherence Across International Engagement

Collaboration across streams was largely driven by external events and supported by informal systems for communication and information exchange.

There were some examples of collaboration across areas of mutual interest for development, diplomacy and trade sectors.

In some cases, collaboration was driven by external events. For example, increased scrutiny of Canadian extractives firms, accompanied by negative media coverage of Canada’s support to extractives industries, sparked the need for a more coordinated response. Efforts to align initiatives in areas of extractives sector governance, corporate social responsibility and human rights, and to develop joint communication strategies increased.

More often, collaboration was driven by personal initiative. For example, linkages between development programming in sustainable economic growth (SEG) and trade sector interests were forged through positive working relationships and supported by informal systems for communication and information exchange. The decentralization of the development program, enabling development staff to work in close proximity to trade and political staff, facilitated these types of interactions.

There were few formal mechanisms to systematize knowledge-sharing and incentivize collaboration.

¶¶ÒùÊÓƵ’s mix of corporate governance structures, distinct work cultures and staff mobility were frequently identified as barriers for coherence. Within this institutional context, a reliance on individual initiative and willingness to collaborate created risks for maintaining and strengthening coherence. Many staff noted the role of a Head of Mission (HoM) as a key driving force for incentivizing collaboration and fostering coherence.

Some efforts to shift towards more formal mechanisms of knowledge-sharing were taken over the evaluation period. Roles and responsibilities for development staff reflected participation in working groups focusing on trade and corporate social responsibility. New spaces for information sharing on activities, meetings and events were created. While these efforts were leading to improved internal communication, formal mechanisms to collaborate on emerging priorities and issues remained weak. This made it difficult to identify and address coherence gaps. For example, it was noted that synergies across policy dialogue efforts for human rights (responsibility of political sector) and gender (responsibility of development sector) could be strengthened. And collaboration and knowledge sharing around emerging issues of concern, such as the safety of human rights defenders and potential risks for rural women leaders and farmers participating in development projects, could be improved.

Corporate planning and reporting tools did not effectively support coherent and cohesive international engagement.

Twelve distinct tools for integrated strategic planning and reporting were implemented over the evaluation period. These tools were largely put in place after the department’s amalgamation in 2013. Staff across branches noted a heavy administrative burden as tools were rolled out. Deadlines were often too tight to allow for internal or external consultation. Rather than promoting integrated planning, tools tended to become inventories of departmental activities. Overall, the department’s package of tools was not perceived to be effective in supporting cohesive international engagement at a country level.

New tools to support coherence, including a whole of department Integrated Country Framework (ICF), were introduced. The intent of the ICF was to “support enhanced integrated planning by missions and HQ to maximize the impact and coherence of Canadian engagement”. Other donors have observed that the use of an integrated strategy has helped them to identify nexus points, increase synergies across streams and better coordinate policy dialogue efforts.

Challenges in the process of implementing the ICF, rather than the tool itself, may have limited its effectiveness. Few staff were consulted in the preparation of the ICF and some were unaware of its implementation. Tools were not sequenced in an order that would have allowed them to be mutually reinforcing. For example, the ICF was rolled out before tools that could have supported its development, such as the integrated conflict analysis, and after processes that it could have fed into, such as visioning statements.

There were challenges in communicating a clear “Canada message” to external stakeholders.

Efforts to develop and communicate a shared vision for Canada’s engagement in Colombia were largely internally focused. Integrated briefing packages, outlining Canada’s priorities and approach for international engagement, targeted senior officials. Strategic frameworks, such as the ICF, aimed to provide senior management with an overview of integrated departmental objectives.

The department had few processes or tools to support outward-facing communication at an integrated whole-of-mission level. This made it difficult to elevate Canada’s profile and communicate a coherent and consistent “Canada message”. External stakeholders, including government partners, other donors and civil society organizations noted difficulties in obtaining a clear picture of Canada’s overall engagement in Colombia. Challenges included confusion identifying initiatives supported by Canada, not knowing how to access information and inaccurate perceptions of what type of activities and initiatives were being supported by the department.

Coherence

Each element of coherence was rated on a four-point scale, with the lowest rating (1) signifying that the element hindered coherence and the highest rating (4) signifying that the element fully supported coherence.

Coherence Within International Assistance

1. Policy: [3]
   

   Departmental policies helped to promote coherence by identifying broad priorities across international assistance, but there was some lack of clarity as new policies were rolled out.

2. Institutional: [2]
   

   Projects were typically complementarity and well-coordinated, but were not guided by a clearly-defined strategic vision for all international assistance programming in Colombia.

3. Administrative: [1]
   

   Planning and coordination mechanisms were burdensome and did not incentivize integrated strategic planning and reporting.

4. Knowledge: [3]
   

   Some effective mechanisms to share knowledge within similar thematic areas and across some streams of international assistance.
   Weak information management systems to consolidate and share information across all streams of international assistance.

5. Informal:[3]
   

   A strong willingness to collaborate, supported by informal systems of communication and information-sharing, helped to leverage synergies across some streams of international assistance.

Coherence Across International Engagement

1. Policy: [3]
   

   Departmental policies helped to promote coherence by identifying broad and complementary priorities across streams.

2. Institutional: [1]
   

   There were no effective mechanisms to support the development and communication of a shared strategic vision and strategy across areas of international engagement.

3. Administrative: [1]
   

   Planning and coordination mechanisms were burdensome and did not incentivize collaborative strategic planning and reporting.

4. Knowledge: [2]
   

   Some efforts to communicate information and knowledge on activities through meetings and events. Weak information management systems to consolidate and share information across all streams and between the field and HQ.

5. Informal:[3]
   

   Collaboration supported by personal initiative and informal sharing of information helped to identify synergies across some areas of mutual interest.

Conclusions

There was a strong consensus that Canada was programming in the “right” areas in Colombia. Early and sustained programming in areas of importance to the peace process helped to build the capacity to respond to new needs and opportunities as Colombia’s context evolved. The mix of humanitarian, development, and peace and stabilization programming provided by ¶¶ÒùÊÓƵ addressed a range of needs including immediate responses to conflict and displacement and sustained efforts to address long-term structural issues and promote peace.

Significant results were achieved at a project-level across children and youth, sustainable economic growth and peacebuilding initiatives. Projects contributed to strengthening civil society organizations, such as agricultural cooperatives and women’s rights groups, particularly at the local level. The ability to bring together private and public sector to collaborate in new and innovative ways was a key factor for project success. Projects that were successful in demonstrating both financial and social viability achieved high levels of uptake by private sector actors, increasing the prospects for sustainability and impacts. Some challenges were faced in transferring new models of service delivery to state institutions. Overall, approaches to delivering international assistance aligned with good practices for donor engagement in middle-income countries and fragile and conflict-affected states.

A significant contribution to policy dialogue was made in areas of child protection, rural education, rural women’s economic development and gender equality. The department played a convening role in bringing stakeholders together and enabling implementing partners to participate in policy dialogue efforts.

Some projects developed good practices and innovative models to promote gender equality and women’s rights. Notable results in the area of sexual and gender-based violence were achieved across all international assistance streams. However, significant barriers for gender equality remained, with the recognition that a long-term view was needed to change deep-seated cultural and socio-economic and institutional barriers.

Examples of well-coordinated and complementary international assistance initiatives and collaboration across areas of mutual interest for development, diplomacy and trade sectors were identified. Where coherence was identified, cases were largely driven by external events or personal initiative and supported by informal systems of communication and information exchange. There were few formal mechanisms to systematize knowledge-sharing and incentivize collaboration. Corporate planning and reporting tools, largely put in place after the amalgamation of the department in 2013, were not found to have effectively supported cohesive international engagement. A lack of integrated outward-facing communication products at a country level made it difficult for external stakeholders, including government partners, other donors and civil society organizations to obtain a clear picture of Canada’s overall engagement in Colombia.

Recommendations and Management Responses

Bilateral Development Program

Recommendation 1

Continue to provide guidance and regular training for staff, implementing agencies and partners on how to align projects with the Feminist International Assistance Policy. Foster exchange of good practices and innovative approaches to women’s empowerment and rights among ¶¶ÒùÊÓƵ projects in Colombia.

Agreed: The Colombia Development Program will continue to regularly convene staff, implementing agencies and partners in Colombia to exchange information, updates and good practices around the implementation of the Feminist International Assistance Policy (FIAP), with an emphasis on development innovation. In addition, the Colombia Program will continue to seek opportunities for training related to FIAP implementation (particularly in a conflict-affected / middle-income environment) building on previously-offered training on Results-Based Financing; GBA+; and FIAP-related policy dialogue. Identified corporate training gaps will be brought to the attention of CFSI and the Colombia Program will work with CFSI, where appropriate, to pilot new FIAP-related courses. CFSI will work with the Colombia program to address identified corporate training gaps and pilot new FIAP-related courses and other learning tools where appropriate.

Recommendation 2

Develop an advocacy and policy dialogue strategy with clear objectives, associated tools and training. The strategy should be developed with partners to promote greater coherence and impact across advocacy and policy dialogue efforts.

Agreed:  The Colombia Program will enhance its efforts to explicitly communicate and prioritize its advocacy and policy dialogue objectives through a Program-level strategy, in consultation with implementing partners in Colombia and other international engagement streams (notably trade and diplomacy).  While advocacy and policy dialogue objectives will need to be tailored to the context and intended audiences, broader messaging around FIAP and ¶¶ÒùÊÓƵ priorities will be developed in consultation with PVP, NLD and others responsible for various Action Areas. In coordination with MFM and IFM branches, the International Assistance Policy Coordination division (PVP) is finalizing policy guidance on each of the Action Areas of the Feminist International Assistance Policy. Related guidance, tools and training are also under development and will be shared with programming branches as they are finalized. The policies and other tools will all support missions in their policy dialogue and advocacy efforts

Colombia mission

Recommendation 3

At a whole-of-Mission level, continue to strengthen formal systems for knowledge and information sharing across international assistance streams (development, humanitarian, peace and security) and international engagement streams (international assistance, trade, diplomacy). Identify mechanisms to incentivize collaboration.

Agreed: BGOTA will further strengthen its knowledge and information sharing across international assistance streams, and international engagement streams. This will include formalization of policy/programming coherence mechanisms as well as additional collaboration as identified through the integrated country framework (ICF) planning process.   Through its daily interaction with BGOTA, NLA will promote greater information sharing and exchanges across all three streams at mission and headquarters, and with various HQ divisions working on foreign policy, trade and development issues. Spaces for information sharing on activities, meetings and events will also be maintained and encouraged e.g. regular conference calls at the Deputy-Director / Director level with all three streams at mission and HQ. These should be systematized and go beyond personal initiative with a view to having each stream aware of important issues other streams are working on.

Through formal mechanisms such as the Annual Report on Human Rights and Free Trade, greater synergies, collaboration and knowledge sharing around cross-cutting issues will be further strengthened e.g. FPDS policy dialogue efforts on human rights; development projects on poverty reduction, inequality and improvement of human rights and security; trade and corporate social responsibility; impact of the Free Trade Agreement (FTA).

Recommendation 4

Develop an integrated whole-of-Mission communication strategy to ensure a coherent and consistent “Canada message”. Strategy should include communication products that provide an overview of (i) how the different international assistance streams (development, humanitarian, peace and security) are collectively addressing ¶¶ÒùÊÓƵ’s priorities in Colombia; and (ii) ¶¶ÒùÊÓƵ’s overall engagement (international assistance, trade, diplomacy) in Colombia.

Agreed:  BGOTA will more explicitly communicate its Whole-of-Mission activities and priorities in Colombia through an integrated communications strategy. LCD commits to working with the Americas Branch and BGOTA on an integrated, Whole of Canada message. Strategy should focus on what Canada is doing to support/work with Colombia in a comprehensive way, to tell the full picture of Canada’s engagement, not the individual pieces. The communications effort should be supported with targeted communications and partner outreach in Canada and Colombia. Development of plan could be supported by a communications temporary duty assignment (TD) in BGOTA to work with mission.

Corporate

Recommendation 5

Review process for roll-out of integrated country framework (ICF) to ensure timelines allow for internal and external consultation and proper sequencing with complementary tools. Assess how the ICF can be better used to foster the coherence of Canadian engagement.

Agreed:  The Integrated Country Framework (ICF) planning process is in a pilot stage for Canada’s plans and programs in select countries.  The pilot was launched in Fiscal Year 2017-18 to test the effectiveness of an integrated planning approach.  While two countries were selected for whole of government planning, Colombia was selected to pilot an ICF at the level of ¶¶ÒùÊÓƵ.   The effectiveness of the ICF is in the process of being reviewed with a “lesson learned” approach led by the geographic bureaus involved.  Questionnaires were disseminated to geographic DGs in late September and are currently being collected.  Results will be assessed from October-November with a view to providing consolidated recommendations aimed at improving the ICF approach for the FY19-20 planning cycle.  Once the assessments are completed and a determination is made as to the value of continuing this process, POL will turn over the lead on ICFs to the Geographic Coordination and Mission Support Bureau (NMD). BGOTA will continue to engage with POL and others involved in the ICF pilot process to share lessons, opportunities and challenges of the ICF process roll-out on the ground.

Considerations for Horizontal Learning

Programming in middle-income countries

1. Middle-income countries can provide opportunities to test innovative ways of delivering and financing development assistance, such as results-based financing and partnerships with private sector actors. When piloting new initiatives, it is important to capture and disseminate early learnings.
2. Programming in middle-income countries requires shifting more time and resources to policy dialogue activities. Staff members require support and training to build the necessary skills and competencies.

Programming in fragile and conflict-affected states

3. Programming in fragile and conflict-affected states requires iterative and adaptive approaches for project implementation. Systems for feedback and continual learning are important. The flexibility allowed through contracting mechanisms, results-based management frameworks, and local funds should be maximized. 
4. There is an implicit need for projects to address issues of trauma when working in fragile and conflict-affected regions. Good practices include the integration of psycho-social support components, with ear-marked budgets, in projects across all sectors. 
5. In areas of protracted crisis, humanitarian agencies can have a deep knowledge of the local dynamics of conflict and the trust of affected communities. Mechanisms to share information and collaborate on projects can help development partners better identify and meet the needs of targeted groups.  

Feminist International Assistance Policy

6. Strengthening the use of Canada’s National Action Plan on Women, Peace and Security in development programming to complement the Feminist International Assistance Policy, in Colombia and other conflict-affected states, could help to promote synergies and coherence across international assistance programming in areas such as sexual and gender-based violence (SGBV).
7. In culturally conservative contexts, it can sometimes be more effective to approach feminism indirectly by demonstrating the social and financial viability of initiatives. Actively engaging boys and men can help to promote gender equality.

Coherence

8. Coherence often depends upon individuals’ initiative and willingness to collaborate. There is a need to balance informal practices with formal mechanisms and incentives for strengthening coherence. Tools such as the integrated country framework, the integrated conflict analysis and gender-based plus analyses have the potential to foster coherence if all business lines are meaningfully engaged in their development.
9. A mix of corporate governance structures, distinct work cultures, and frequent staff mobility creates challenges for departmental coherence at a country level. The role of the Head of Mission (HoM) is key to fostering and incentivizing coherence across all lines of business.

Appendices

Appendix 1: Colombia Program Logic Model^{Footnote 8}

Appendix 2: Methodology – Data Limitations

Limitations included security restrictions in Colombia and challenges with accessing and aggregating program data across international assistance.

Limitations

Challenges locating and accessing program/project information/data due to:

• incompatible information systems
• inaccessible or missing core documents
• many documents not being saved in a shared document management system

Not possible to aggregate results and beneficiary data across international assistance given that:

• reporting provides yearly, not cumulative results
• there was a lack of clarity on how reach numbers were determined
• a multitude of performance indicators were not aligned to a common strategy

The timing of site visits and unexpected security concerns resulted in:

• limited access to beneficiary groups in one conflict-affected region
• barriers to women’s participation
• limited access to government officials, due to the election period

Mitigation Strategies

• Evaluators dedicated considerable time and effort to locate and gather documents, with the support of program staff
• Evaluators took advantage of directories of key documents (where available)
• Nvivo 11 software was used to analyze project data and identify common trends and results
• Where aggregation was not possible, broad categorizations were developed for reporting purposes
• Site visit itinerary was adjusted and alternative safe and neutral locations for meetings were selected to allow for interaction with beneficiaries
• Attempts to reach government officials were conducted as early as possible before election

Appendix 3: Evaluating Coherence

Overall Approach

Because coherence is a core principle of Canada’s international engagement, the evaluation’s approach to assessing coherence focused on what we mean by coherence, what we could measure to know whether or not the Colombia Program was coherent, and where its strengths and weaknesses lie. The approach focused on defining the underlying elements of coherence, and assessing whether or not they were contributing to coherence and leading to desired results.

Unpacking Coherence

The OECD (2018) defines coherence as “an approach and policy tool to systematically integrate the economic, social and environmental dimensions of development at all stages of domestic and international policy analysis.” This concept is based on the need to align the multiple interests and actors that support international cooperation, and better leverage their collective impact towards sustainable development outcomes. The understanding of coherence for this evaluation was informed by the guidance note for the department’s Integrated Country Framework, a strategic planning tool. The note discusses the goal of improving coherence as ensuring that different lines of effort are “mutually reinforcing and do not work at cross purposes.” Coherence does not imply that all actors need to be working together on the same initiatives. Rather it suggests that there is good understanding across the effort (e.g., Canada’s partnership with Colombia) of what the different parts are doing, exploiting opportunities for joint efforts, ensuring that one part of the program is not in contradiction with another, and building initiatives off each other where possible.

Identifying the Level of Coherence

The focus of this evaluation was limited to assessing internal coherence within international assistance programming (development, humanitarian, peace and stabilization) and intra-organizational coherence across the department’s international engagement activities (international assistance, trade, diplomacy). A third level of coherence, inter-organizational coherence, was defined. Assessing inter-organizational coherence would consider all Canadian government departments (OGDs) active in Colombia, coherence with the policies of Colombia, and coherence between Canada and other countries and agencies working in and supporting development in Colombia. Assessing this level of coherence was beyond the scope of this evaluation.

Assessing Coherence

After identifying the levels of coherence that would be considered, the evaluation set about defining the core elements of coherence. These included institutional, administrative, knowledge management, and informal systems (see slide 21). Understanding the extent to which these elements are supporting a coherent program can help uncover what has worked, what has not and what opportunities there are for improving coherence. Next, the evaluation gathered information through document review and interviews on the extent to which the elements of coherence were in place. This included identifying examples of coherence, as well as supporting and constraining factors. As these issues are largely qualitative in nature, a rubric-based approach was used to define levels of achievement. Rubrics set out clear criteria to represent different levels of performance, and, if used over time, can help track progress and changes in performance. The excerpt below provides an example of the rubric for assessing administrative coherence: