Summative Evaluation of Five Maternal, Newborn and Child Health (MNCH) Projects in Haiti

Executive Summary

Rationale, purpose, and specific objectives of the evaluation

The Government of Canada made a commitment to provide $3.5 billion to improve maternal, newborn and child health (MNCH) in developing countries between 2015 and 2020. One of the components of this MNCH commitment included $375 million over 5 years for proposals to the Partnership for Strengthening Maternal, Newborn and Child Health Initiative (PSMNCH). This initiative supported Canadian non-governmental organizations (NGOs) in carrying out proven, large-scale interventions to increase maternal, newborn and child survival.

The initiative focused on 4 thematic priorities:

This summative evaluation, carried out by a team of independent experts (the Consultant), is intended to: (i) inform stakeholders about lessons that can be drawn from the 5 projects included in the evaluation; (ii) clarify the implementation of the next health initiative of the Partnerships for Development Innovation (KFM) or other ¶¶ÒùÊÓƵ (GAC) sectors; and (iii) provide stakeholders with evidence to inform the design of future similar projects.

The evaluation has 2 specific objectives: (i) evaluate the various approaches used in implementing the projects under study; and (ii) identify findings, conclusions, recommendations, and lessons to achieve the purpose of this evaluation.

Scope and coverage of the evaluation

The evaluation covers 5 MNCH projects implemented in Haiti between 2016 and 2021. It aims to examine 8 key areas, under 2 categories:

Development context

Six major characteristics of the Haitian context have played an important role during the implementation of the projects: (i) the relatively low coverage of essential health services; (ii) underfunding of the health system; (iii) inadequate governance of the health system; (iv) gender inequalities; (v) political instability and insecurity; and (vi) the COVID-19 pandemic.

Development intervention and intervention logic

The 5 projects included in the evaluation are:

These projects are distributed across 7 of Haiti’s 10 departments. With budgets ranging from $5.6 million to $11.0 million, they were launched between 2016 and 2017 for an initial duration of 3 to 4 years, all ending in 2020. Four projects had no-cost extensions. Subsequently, 3 projects received a total of $4.28 million in additional funding for COVID-19 crisis response.

The ultimate outcome is the same for all the projects: the reduction of maternal and child mortality rates in the targeted regions.

This outcome is to be achieved through 4 intermediate results:

  1. Improved delivery of essential health services to mothers, pregnant women, newborns, and children under 5.
  2. Improved utilization rate of essential health services by mothers, pregnant women, newborns and children under 5.
  3. Increased rate of consumption of nutritious foods and food supplements by mothers, pregnant women, newborns and children under 5.
  4. Increased rate of dissemination and use of demographic data.

Stakeholders

Stakeholders include: (i) Canadian implementing organizations, including international and national organizations established in Haiti (“executing agencies”); (ii) the central and decentralized directorates of Haiti’s Ministry of Public Health and Population (MSPP); (iii) community actors and health institutions (HIs) directly targeted; (iv) final beneficiaries within communities (women, men, youth, and children); and (v) KFM sector of GAC.

Evaluation approach and methodology

For security reasons (national context and COVID-19), the evaluation was conducted virtually, i.e. without a field mission. It addressed 5 main questions:

  1. 1. What main elements characterize the strategic approaches adopted by the projects related to the PSMNCH thematic priorities in Haiti?
  2. What are the main challenges that the projects have faced, particularly but not exclusively in relation to the key areas of the evaluation?
  3. To what extent have the strategies used by the projects fostered conditions for the sustainability of outcomes?
  4. What practices, particularly but not exclusively in the key areas, meet the characteristics of the best practice model?
  5. What are the main lessons identified by project actors based on evidence?

An Evaluation Evidence Matrix (EEM) was constructed to guide data collection and analysis. For each main evaluation question, the EEM allowed the identification of critical elements to be examined and data sources, methods and tools to be used. It ensured a first level of data triangulation.

Data collection was carried out through: (i) descriptive and analytical grids applied to the documentation reviewed; and (ii) individual interviews conducted remotely with cooperation partners and international health experts. A total of 38 people were interviewed (14W, 24M).

The data sources used consisted of: (i) a document database of more than 160 items (orientation and programming documents, national policies, studies, evaluation reports, planning documents and reports from the 5 projects); and (ii) the opinions and views of the various key actors interviewed.

The strategic approaches and challenges were extracted from an analysis across the interventions carried out by the 5 projects. Best practices were identified using a 7-criteria assessment model, and lessons were consolidated based on the frequency with which they appeared in the document database and interviews.

The main limitations of the evaluation are: (i) the absence of direct observation of the interactions among the actors and of project outcomes; (ii) the fact that it was not possible to interview representatives of the direct beneficiaries of the projects; and (iii) the limited resources (in terms of budget and timeline) made available to the Consultant in view of the complexity and real scope of the mandate.

Key findings

Programming context. Until the early 2010s, Canada exercised significant leadership in the health sector in Haiti through its role in policy dialogue and through a health policy framework developed over a 10-year period that guided its programming interventions and took advantage of structural benefits from project results, promoting complementarity between different health projects. From 2015 when the PSMNCH began until the time of this evaluation in 2021, Canada has implemented 22 health sector projects partially or entirely in Haiti, through its various sectors, (being KFM, Global Issues and Development Branch and the Haiti Bilateral Program). Despite tentative coordination, no framework allowed the creation of synergies between these projects or the leveraging of the significant Canadian presence in the sector.

Strategic approaches adopted by the projects. In terms of management, all the projects established committees in which they have involved the bodies within the Ministry of Public Health and Population by assigning them mainly advisory and monitoring roles. The involvement of the Ministry bodies in the design of the projects took the form of consultations of varying degrees of depth and focused on improving MNCH services.

In terms of the intermediate result, improving the delivery of essential MNCH services, project approaches to capacity building in the health institutions (HI) addressed different categories of staff and have been characterized by workplace-based training in action. Capacity building in the decentralized Ministry of Public Health and Population’s bodies was mostly limited to coordination, supervision, the health information system (HIS), and the referral system. Finally, the provision of community-based MNCH services has been strengthened, especially through community health centres, multi-skilled community health workers (MCHW) and mobile clinics. Family planning issues were less addressed in project implementation, and youth of reproductive age were not specifically targeted due to barriers for reaching this group of people..

In terms of improving the rate of use of MNCH services, a common strategy for all projects combined community mobilization and information education communication (IEC). All interventions included objectives to promote gender equality (GE) and women’s rights, as well as awareness of and education on GE-related health issues, and MCHW played a central role in this. In the area of the intermediate results, improved nutrition for mothers/pregnant women, newborns and children under 5, although only 2 projects specifically targeted this intermediate result, all projects have worked on it to varying degrees.

In terms of increasing the rate of dissemination and use of local demographic data by civil registry offices, local partners, and leaders, all of the projects supported the strengthening of the health information system with relatively similar approaches, but to varying extents. However, only one project included a specific component on birth registration during the 2 months following birth in the civil registry offices and trained the MCHW to accompany families in this process.

In terms of cross-cutting themes, namely gender equality, environmental sustainability and governance, all projects have systematically integrated them into their interventions. For gender equality, the projects aimed to strengthen the capacities and knowledge of the supported health structures, communities and specific groups. Environmental sustainability was integrated through prior analysis and mainstreaming of the environment within the health institutions as well as environmental IEC in the communities. In terms of governance, the projects have helped to strengthen the capacities of state partners, the leadership of the Ministry of Public Health and Population and its territorial entities, and management capacities of certain health institutions. In addition, with the exception of human rights and, more particularly, women’s rights, the theme of governance was addressed only to a limited extent at the community level.

Key issues and challenges addressed by the projects. The main issues targeted in the strategies for the first intermediate result, improving MNCH service delivery, were: (i) quality of reception in health institutions; (ii) HR skills; (iii) availability of basic medical materials, equipment and inputs; (iv) infrastructure improvement; and (v) management of MNCH services. In terms of the second intermediate result, improving the utilization rate of MNCH services, the main issues targeted were: (i) the various categories of MNCH health services offered and the importance of using them; (ii) the availability and proximity of preventive health services for mothers, newborns and children (e.g. prenatal visits); (iii) women’s capacity to decide to use these services; (iv) affordable access; and (v) referral systems. The main issues targeted for the third intermediate result, improvement in nutrition, were: (i) timely detection of malnutrition cases; (ii) increased and facilitated access to adequate institutional management of services and programs; and (iii) the weakness of the information system to plan for and adapt the supply of nutritional supplements to the real needs of the target populations. In the area of gender equality, the main issues and themes addressed were: (i) the recognition of women’s and girls’ sexual and reproductive rights; (ii) the importance of increased participation of women and girls in the various decision-making processes and entities, including but not limited to MNCH; (iii) more equitable access to and control of health system resources for women and girls; (iv) the involvement of men and fathers in MNCH and family health; and (v) gender-based violence and its impact on health, social and justice issues.

Sustainability of results. Project achievements are numerous and relate to strengthening human resources, community leaders and organizations as well as the technical and material capacities of the departmental health authorities, the health institutions, the communities and the community groups. These achievements remain vulnerable due to the context in Haiti. Project sustainability strategies were based primarily on: (i) the alignment of the projects with the policies, strategies and guidance of the Ministry of Public Health and Population; (ii) information sharing and dialogue with the relevant Ministry bodies; (iii) the ongoing potential of human resources trained both in the health institutions and in organizations that were already active and mobilized in their domains, and of bringing health infrastructure up to standard. These strategies reflected little or no analysis of the health system’s capacity to continue to deliver the same level of basic health care services or to absorb recurring costs associated with project achievements.

Practices that meet the characteristics of “best practice.” Strictly applied, no project intervention meets all of the following 7 criteria of the “best practice” model, especially given the context of a fragile country like Haiti: effective, sustainable, gender-sensitive, realistic, participatory, replicable, and risk reducing. However, the model distinguishes different stages in the evolution of a practice, and those selected in this evaluation may be considered “promising practices”.

Conclusions

C-1: Leveraging project funding could enhance Canada’s contribution to MNCH in Haiti. The 5 projects carried out in Haiti in the context of the PSMNCH initiative represent approximately half of all Canadian aid provided to the health sector in Haiti. This funding, in addition to that allocated through other GAC aid channels, could be used as leverage for policy dialogue to increase both the effectiveness and the influence of Canadian aid in the sector.

C-2: The need for the PSMNCH to involve national authorities in strengthening MNCH and integrating it into the national health system. Integrating a multi-country program into a national health system requires adjustments that must be the subject of policy dialogues to avoid further penalizing already fragile health systems and to increase ownership by national authorities. According to the available data (document database and interviews), the 2 key central directorates of the Ministry of Public Health and Population for maternal health (Family Health, and Planning Studies) were not consulted during the design of the PSMNCH or during the approval process for projects carried out in the country. This is a limitation in terms of strengthening the health system and its governance.Footnote 1

C-3: The added value of a theory of change to guide action. In response to a program structured around the 4 thematic priorities of varying complexity, the 5 projects have developed strategic approaches based on the implicit assumption that actions on the delivery and management of MNCH services in an intervention area, combined with community mobilization activities with target beneficiaries, would reduce maternal and infant mortality. This hypothesis does not refer to any explicit theory of change covering: (i) the multisectoral dimension of population health interventions; (ii) the interdependence and prioritization of the factors at play in the reduction of maternal and infant mortality; and (iii) the articulation of causal links between the various levels of desired outcomes (intermediate outcomes and ultimate outcome).

C-4: Project management models limit shared responsibility for achieving outcomes. The main responsibilities of the health authorities involved in the project management structures were to ensure that project activities were consistent with national guidelines and policies, to agree on the annual work plans and activity reports and to remove obstacles to project implementation. However, according to KFM’s provisions for implementing PSMNCH, accountability for achieving project outcomes rested exclusively with the NGOs that signed contribution agreements with GAC, which had sole decision-making authority.

C-5: Important achievements at the end of the projects. The executing agencies have generally mobilized the resources made available to them effectively. At the end of their interventions, they leave behind achievements in the targeted departments related to the improved skills of service providers, service managers, and community actors and groups. They also leave behind tools and mechanisms to contribute to the reduction of maternal and infant mortality. Although not all the interventions were completed, overall, the projects demonstrate a business model that can be used to mobilize and empower the various stakeholders to strengthen a health care system that can inspire confidence and bring communities closer to health institutions.

C-6: Executing agencies demonstrated ability to address project implementation challenges, but varying ability to produce intermediate outcomes by thematic priority using their strategic approaches. Executing agencies demonstrated flexibility in addressing key challenges to their activities. The strategic approaches of the projects were distinguished by varying importance given to issues such as: (i) strengthening system governance; (ii) partners’ participation in project planning, implementation and management; and (iii) the extent of institutional and community support. These differences are rooted in the executing agencies’ missions and the expertise and experience they bring to the table. The interventions targeting improvements to MNCH service delivery and those related to identifying and managing malnutrition cases have demonstrated strong performance on the whole. This success is largely attributable to the careful tailoring of support to the needs identified at the beginning of each project. The performance of interventions aimed at improving the utilization of health services was more variable in achieving intermediate outcomes following immediate outcomes. This is partly explained by the greater role of behavioural change in this thematic priority. Economic barriers are also a major issue hindering the poor and marginalized populations that make up the vast majority of project beneficiaries from accessing MNCH services.

C-7: Several critical issues for MNCH services targeted; others addressed to a significantly lesser extent. The bulk of project interventions at the service level in health institution focused on strengthening the capacity of providers and infrastructure to manage safe deliveries and deliver pre- and post-natal care. At the community level, and in addition to IEC activities, preventive health interventions have focused on screening for high-risk pregnancies and malnutrition and referrals to institutions, as well as on immunization and provision of essential nutrients and vitamins to young children. However, one important issue has been largely overlooked: women’s right to access modern and effective contraceptive methods. In addition, project interventions, with one exception, have not targeted young people as a group, even though the risk of maternal mortality is significantly higher in adolescent pregnancies.

C-8: Systematic integration of gender equality into projects was a notable contribution, but performance is difficult to measure. All the projects have deployed notable efforts to support the systematic integration of gender equality and women’s rights in all their interventions and with all targeted stakeholders. To this end, they developed a variety of strategies and approaches that helped: (i) increase knowledge of MNCH-related gender equality issues; (ii) increase women’s participation and leadership in relation to the quality and relevance of MNCH services in their communities; and (iii) involve men more widely in maternal and child health within their families. In addition, significant support was given to considering and addressing gender-based violence, leading to concrete actions in this regard. Canada’s leadership is recognized in this area, especially by the Ministry of Public Health and Populationhealth authorities. However, the effective contribution of MNCH projects in terms of gender equality remains difficult to assess and measure due to the lack of specific gender equality outcomes and corresponding indicators in the projects’ logic models and Project Measurement Frameworks.

C-9: Strategies to strengthen governance are relevant, but generally limited to certain governance functions. All the projects defined, in varying degrees of detail, their strategy for strengthening governance in MNCH. In practice, the scope and scale of interventions in this area varied greatly between projects. Interventions focused on certain health sector governance functions that are primarily the responsibility of the departmental health directorates, such as supervision of the HI, coordination of internal and external actors, and operation of the health institutions. Among the difficulties encountered, the evaluation found mainly: (i) a lack of expertise and experience on this issue on the part of some of the executing agencies; and (ii) a lack of interest or commitment shown by some of their partners with respect to their own mission and role in the health system.

C-10: Sustainability: an insurmountable challenge without clear long-term commitments. The question of sustainability of the results goes far beyond the capacity of the projects, whose limited duration prevents them from completing an initial experimentation with their various approaches, testing them with partners, and adjusting and relaunching them. Sustainability of project outcomes is a joint challenge for the Haitian state and the Canadian government that goes beyond the strict framework of projects as envisaged in the current form of the PSMNCH. The challenge has 3 key elements in a fragile country like Haiti: (i) a clear plan around which the state can mobilize its external partners and within which Canada can elaborate its aid program; (ii) predictability of aid, with explicit, confirmed long-term commitment; and (iii) realistic cost-sharing among partners.

Best Practices

Best practices can result from a single project or from several projects. However, their adoption by the greatest number of executing agencies in future designs of similar projects would be highly desirable.

Best practice 1: Adapt interventions to the context of insecurity and political instability. Strategies that successfully supported completion of project action plans without jeopardizing the security of staff, partners and beneficiaries involved a combination of: (i) developing communication and security plans; (ii) preparing work teams to implement these plans; (iii) introducing remote work; (iv) adapting work plans to periods of calm in order to carry out training activities; (v) delivering inputs and equipment to the health institutions; and (vi) holding meetings with community organizations.

Best practice 2: Support cervical cancer screening services. The integration of cervical cancer screening services into major awareness-raising days on the importance of MNCH, combined with free screening tests and referrals to specialized centres for advanced follow-up, was a direct response to women’s needs and fills an important gap at the national level.

Best practice 3: Support community-based malnutrition screening, outpatient therapeutic programs (OPT), and nutritional stabilization units (NSU). The integration of nutrition interventions into the Ministry of Public Health and Population’s community-based nutrition care model has resulted in: (i) training of MCHW at the community level to organize screening sessions and referrals to health institutions; (ii) ongoing support to OPT and NSU to ensure identification and proper management of cases of acute malnutrition; and (iii) financial support for families accompanying acutely malnourished children during their outpatient follow-up or hospitalization.

Best practice 4: Empower women to make their own health decisions. Reducing maternal mortality requires action on 3 main aspects in which women face “delays”: (i) recognition of danger signs; (ii) getting to health service locations; and (iii) ensuring they receive prompt attention in health institutions. The projects contributed directly to strengthening women’s ability to recognize a pathological situation on their own and to make informed decisions about seeking pre- or post-natal counseling and delivery in health institutions. These projects applied approaches recognized as “promising” in the scientific literature for this first delay, i.e. recognizing danger signals by the following means: (i) implementation of interventions to raise women’s awareness of childbirth complications and the importance of giving birth in the presence of a skilled health professional; (ii) identification of women leaders who could pass on prevention messages at women’s meetings; (iii) training MCHW or volunteers in maternal health and organizing education sessions for men and women; and (iv) training and support for matrons or traditional birth attendants in identifying warning signs, providing first aid, and making referrals to an appropriate level of care.

Best practice 5: Promote positive masculinity approach. Project participants shared expertise with each other on positive masculinity, which aims to change men’s behaviours and engage them in maternal and child health care. Promoting this approach has resulted in, among other things, increased support by men for women in their decision to use MNCH services, increased recognition of women’s rights in sharing tasks, and a decrease in gender-based violence.

Best practice 6: Anchor interventions in local strengths. The use of health services depends in large part on the confidence of communities in their institutions and services. This trust is developed through community involvement at three levels being the information level,the awareness of and support for action. The most successful projects (i) developed deeper knowledge of their target communities to identify the existing community structures most likely to influence attitudes and behaviours; (ii) developed their messages with key players within the various groups; (iii) strengthened the social communication and awareness-raising skills of these players; and (iv) empowered the community members to make them active partners in supporting changes in their own behaviour.

Best practice 7: Improve human resources training. Success factors in HR training interventions have included: (i) targeting actors at all levels of the system and in all components of the system, i.e. managers, providers and beneficiaries in health and community facilities; (ii) the development of an integrated and adapted adult training approach using awareness-raising, IEC, on-the-job training, peer training, and coaching techniques; (iii) the use of an approach based on diagnosis of needs with partners, a skills-building plan and monitoring and supervision of learning; and (iv) capitalizing on local resources to complement the technical assistance provided by the projects through the use of the health system’s own expertise and that of other ministries, as well as specialized national, regional or local organizations and trainers of trainers and community-based “models”. This approach directly addresses the third “delay” in the fight against maternal mortality: ensuring women receive prompt care by qualified health personnel.

Best practice 8: Address gender-based violence. Addressing gender-based violence is of particular interest to the Ministry of Public Health and Population’s Family Health Department, which has developed a national guide for service provision that guarantees respectful care of victims of this type of violence. Project interventions have helped fill a significant information and training gap within the Ministry’s structures. Depending on the area of intervention, they have strengthened the identification and management of gender-based violence in the health institutions or, through collaboration with the Ministry on the Status and Rights of Women in Haiti, set up a shelter for female victims of gender-based violence.

Best practice 9: Conduct operational research. Under the heading of operational research, some projects supported interventions enabling different categories of actors to address unresolved challenges or issues based on their project experience. Among other things, this led to better understanding of stigmatizing factors for pregnant girls and of factors related to the attitudes of institutions that have the effect of reducing the access to and use of health services for poor or marginalized beneficiaries.

Best practice 10: Support across the continuum of care. In the area of maternal and child mortality, support across the continuum of care covers approaches to ensure effective management of the “3 delays” mentioned previously relating to optimal access to care. The projects that were most successful in providing this type of support implemented interventions across the continuum of care. Regarding the first delay, actions were aimed at empowering community actors (including leaders and organized groups) and outreach providers by supporting women and household members to make informed decisions about seeking care and by providing preventive health services directly. Regarding the second delay, actions were aimed at empowering community actors and first-level health care providers and health institutions to intervene on patient referrals and reduce the delay in access to care. In response to the third delay, the actions aimed to make first- and second-level health institutions responsible for the timely availability of care through ensuring support and quality of care. The actions also aimed to support the management of the supply of care through the monitoring and supervision of the stakeholders in the health system, from the departmental level to the local level. Lastly, with respect to the accountability of the actors, these actions were characterized by support rather than substitution of local stakeholders with respect to their roles in the proper functioning of the health system.

Lessons learned

Lesson 1: Closely involving partners is essential to achieve results. This lesson received the greatest recognition and emphasis by the executing agencies. It is in keeping with the principle of respect for national leadership and the Ministry of Public Health and Population’s policies and standards and contributes to the relevance of interventions, ownership of outcomes and, as a corollary, the sustainability of gains.

Lesson 2: Be flexible to adapt to changing contexts. The feasibility of projects in the particular context of Haiti requires flexibility to adapt interventions, when necessary, to changes in the socio-political and natural environment and to respond, in this context, to the demands of partners.

Lesson 3: Allocate adequate time to projects for optimal outcomes. The duration of projects illustrates the connection between the challenges and constraints of project implementation with the nature of the changes sought. Time is a critical variable when changes involve perceptions, attitudes and behaviours, especially when these are strongly rooted in cultural or religious values.

Lesson 4: Obtain a formal commitment to HR staffing from the Ministry of Public Health and Population at the project design stage. Despite the recruitment of staff needed to deliver services by some projects in the target areas, the Ministry lacks formal commitment at the design stage of interventions to take over health institutions to ensure their adequate staffing, one of the biggest threats to sustainability.

Lesson 5: Improve MNCH approaches and practices through collaboration and exchange with other projects. According to project stakeholders, opportunities to meet and share experiences with other projects have allowed them to better focus their actions and thus improve their performance.

Lesson 6: Provide projects with human resources specifically assigned to gender equality and governance.

Executing agencies benefit from providing their local teams with resources and expertise specifically assigned to gender equality and governance so that they can better adjust their interventions as these evolve.

Lesson 7: Regularly adapt IEC content and approaches to the target audiences and the intervention context. Interventions aimed at mobilizing communities are carried out over time during which the acquisition of knowledge evolves within the beneficiaries. Thus, regular updating of content and approaches promotes greater effectiveness of these interventions.

Recommendations

R-1: 1. That KFM, in collaboration with the other GAC divisions concerned and with the health authorities of partner countries, develop a programming framework specific to the health sector.

R-2: 2. That KFM, on the recommendation of the host country’s health authorities, identify priority intervention locations for projects in their country based on the nature of the program’s support, the other actors present and Canadian technical expertise.

R-3: That KFM, through project management structures, make Canadian and national partners accountable in terms of identifying, implementing and achieving project results. This means implementing formal agreements between stakeholders to make them more accountable for outcomes, while giving them the flexibility they require to adapt their interventions to evolving contexts and emerging needs.

R-4: That KFM require projects to incorporate an explicit theory of change that serves as a basis for their action and determines the factors at play, as well as their dynamics in achieving major outcomes.

R-5: That KFM ensure that projects have an explicit and complete gender equality strategy, that this strategy is rooted in a gender‑based analysis, and that it contains specific gender equality results in the logic models and corresponding indicators in the performance measurement framework.

R-6: That KFM ensure that implementing agencies have, on their project team, expertise in health systems governance.

R-7: That KFM realistically set out its expectations and requirements with regard to the sustainability of results when calls for proposals are launched or no later than when project implementation plans (PIP) are being developed, and that KFM inform national partners of the duration and continuity of its commitment to them.

R-8: That KFM conduct an impact evaluation on the effects that awareness and training activities supported by MNCH projects in Haiti have had on behavioural changes among target beneficiaries.

Date modified: