Key Pathways to Developing a Participatory Theory of Change forSystem Change Tracking: A practice-oriented toolkit for enabling an evaluation culture
- Post by: IPRR
- October 9, 2023
- No Comment
B. Rajeshwari[1] and Meena Vaidyanathan[2]
[1] Senior Consultant, Niiti Consulting Pvt. Ltd.
[2] Founder and Director, Niiti Consulting Pvt. Ltd. 2/503 Jaypee CGHS Plot No. 2 Sector 22, Dwarka, New Delhi- 110075
| Title: | Key Pathways to Developing a Participatory Theory of Change forSystem Change Tracking: A practice-oriented toolkit for enabling an evaluation culture |
| Author(s): | B. Rajeshwari and Meena Vaidyanathan |
| Keywords: | Theory of Change; Participatory Process; Evaluation Culture; Power Differentials; Systemic Change |
| Issue Date: | October 9, 2023 |
| Publisher: | IMPRI Impact and Policy Research Institute |
| Abstract: | The paper demonstrates how a Theory of Change (ToC) could be developed in a participatory process, creating the space for capturing diversity and contextualities within a program and contributing to building a collaborative ecosystem to drive systemic change, through a case study. The process that evolved in building this ToC can be useful for other organizations with similar impact objectives and in similar contexts. The paper shows that a focussed, participatory-consultative approach led by self-reflection and engagement with various stakeholders and their needs, can help address power differentials between funders and the implementing partners of development programs. This approach makes it possible for ToCs to be practically adaptable documents by all stakeholders in the ecosystem, thereby building an evaluation culture. In this paper, we demonstrate seven pathways that can be contextualized to specific realities. The paper fills the existing gap in similar demonstrable cases from South Asia where a participatory approach is adapted to develop the ToC for a development programme. |
| Page(s): | 85-101 |
| URL: | |
| ISSN: | 2583-3464 (Online) |
| Appears in Collections: | IPRR Vol. 2 (1) [January-June 2023] |
| PDF Link: | https://iprr.impriindia.com/wp-content/uploads/2023/10/SA3_Key-Pathways-to-Developing-a-Participatory-Theory-of-Change_IPRR-V2I1_Jan-June-2023.pdf |
(January-June 2023) Volume 2, Issue 1 | 8th October 2023
ISSN: 2583-3464 (Online)
The paper demonstrates how a Theory of Change (ToC) could be developed in a participatory process, creating the space for capturing diversity and contextualities within a program and contributing to building a collaborative ecosystem to drive systemic change, through a case study. The process that evolved in building this ToC can be useful for other organizations with similar impact objectives and in similar contexts. The paper shows that a focussed, participatory-consultative approach led by self-reflection and engagement with various stakeholders and their needs, can help address power differentials between funders and the implementing partners of development programs. This approach makes it possible for ToCs to be practically adaptable documents by all stakeholders in the ecosystem, thereby building an evaluation culture. In this paper, we demonstrate seven pathways that can be contextualized to specific realities. The paper fills the existing gap in similar demonstrable cases from South Asia where a participatory approach is adapted to develop the ToC for a development programme.
Introduction and Conceptual Framework
Theory of Change (ToC) has, since the beginning of the 1990s, become a dominant method in the evaluation of development programs, philanthropies, governance, and anyone working to bring about social change. ToCs often describe the logic, principles, and assumptions that explain what an intervention does, why, and how it does it along with chalking out the intended results (Ghate, 2018). ToC, as an evaluation method, helps organizations understand and explain how any intervention leads to a specific development change based on a causal analysis backed by necessary evidence (UNDAF Companion Guidance, UNDG). The significant focus on ToCs is developed around how they can be represented or what should they encompass, and how connections can be built from the macro to the micro-level of change and between the inputs, outputs, and outcomes (Davies, 2018; Rasmussen, 2016). However, showing these connections between the macro- and micro-levels and the change occurring as a result of an intervention is complex and power-laden. There are apparent power differentials when those funding programs have a readymade ToC and want agencies or implementing partners to work with the same. There are ‘virtual tyrannies of ToC’ (Behar, 2022) in the social sector. ToCs have often been seen as documents that can go redundant, suffer from over-simplification, reductionism, claims of universalities, and very low weightage given to particularity (Behar, 2022). Such criticism has made development organizations and researchers address how ToCs need to be developed in consultative ways, based on robust evidence and can support continuous learning (UNDAF Companion Guidance: Theory of Change). How the power differentials can be addressed while the ToC processes evolve in development programs? Our paper addresses this question by demonstrating it through a case model of the development of a set of ToCs for tracking the stated impacts of the Oak India program and its diverse group of grantees in the state of West Bengal (a state in the eastern region of India), across two primary Focus Areas, impacting five priority communities.
Through this case, the paper demonstrates how a ToC can evolve through a participatory process and create ways for capturing diversity, and contextualities within a program that has several implementing partners contributing to building a collaborative ecosystem to drive systemic change. We have in this paper attempted to document the process of building a ToC that can, not just be used by the participating organizations, but also by others with similar impact objectives. The paper makes a case for a process focussed, the participatory-consultative approach that is self-reflective and engages with various stakeholders and their needs, to help address power differentials between the funders and the implementing partners of development programs. This approach also makes it possible for ToCs to be inclusive and practically adaptable documents to build an evaluation culture (Srinivasan et al., 2022).
Demonstrating the case of the evolution of ToC is significant given that ToCs help organizations to understand how an intervention or set of interventions shape and lead to a specific development change, becoming guiding documents for organizations to make strategic decisions about program elements, organizational policies, and policies along with providing a structure and accountability for evaluation (DuBow and Litzler, 2018). ToCs create the space for practitioners and M&E specialists to understand and explain the complex nature of the change processes in development interventions (Van Wessel, 2018).
ToCs are often analyzed in the context of specific programs centering on the benefits derived from the process and how the ToC creates opportunities to improve the evaluation (DuBow and Litzler, 2018, Van Wessel, 2018). Expectations around ToCs tend to focus on how they can help programs achieve goals, create best practices, explain the effectiveness and performance of programs, represent the complex processes of change, and how effective the approach for building ToCs (Mackenzie and Blamey, 2005; Blamey and Mackenzie, 2007; Taplin and Clark, 2012; Stein and Valters, 2012; Mayne, 2017; Davies, 2004). Solutions to creating a robust ToC are often visualized through guides that show mechanical steps to develop them (Rasmussen, 2016; Taplin and Clark, 2012). However, in recent years, emphasis has been given to participatory and consultative approaches to building a ToC so that it meets the needs of the communities and makes it most meaningful for them (Wilkinson et al., 2021; Budzyna, 2021). Despite the emphasis on participatory and consultative approaches, there are very few instances of actual cases that help understand how this happens in reality. A common criticism against ToC documents is that they are used more as a “framework that fixes agreements rather than as a living, a guiding tool that helps in reflection and adaptation (Ho et al., 2020).”
This paper primarily demonstrates the creation of a joint ToC across organizations with diverse programs through a participatory process. In this paper, we show how a ToC process evolved in a South Asian context and what were some of the key features of this process that is relatable and practically applicable to others in a similar context and address the challenges that emerge on the ground. In doing so, we fill the existing gap in similar demonstrable cases in development literature and the actual ways in which a process of ToC building takes shape in the context of a complex program with multiple partners having diverse objectives and end goals. Also, the paper shows that there are specific and conscious ways in which unequal power imbalances can be addressed in programs by creating a consultative ToC.
The Case Study and Evolving a ToC for a Complex Programme
The Oak Foundation’s India program in West Bengal supports efforts to sustainably improve the lives of marginalized people including communities in tea gardens (mainly in North Bengal) and the Sunderbans delta (in south coastal Bengal), unorganized workers and migrants (across the state), as well as indigenous communities (across the state in key clusters) through strengthening systems, enabling sustainable livelihoods and access to entitlements, amplifying community voices and their access to justice. The two focus regions, namely the tea gardens and the delta region, face a range of climate-related issues that receive a special focus from the Foundation. The secondary grant-making focus area prioritizes three community groups which include unorganized workers, indigenous communities, and migrants. The entire program has been visualized for a period of 10 years during which they hope to achieve and document significant change. A learning partnership to design and inform Oak’s tracking, reporting, and learning processes along with its ecosystem of grantees was established between initial consulting and Oak Foundation over a period of 2 years (2021-2023). A series of Theories of Change was commissioned to be developed as part of the learning partnership that could explain how a set of interventions could lead to specific changes through the interventions based on available evidence, inform gaps and enable planning of the grant portfolio of Oak, aligned to systemic outcomes and impacts.
A key objective of this exercise was to use the process of ToC development to build a culture of evaluation among the ecosystem of grantees, that enables them to leverage monitoring and evaluation systems and their enhanced capacities for improved tracking and effectiveness of their programs. The entire process was planned to be participative and consultative, driven by sound analyses, consultations, and practical learning in diverse contexts.
Several organizations, especially international donor organizations, now, visualize framing theories of change through a participatory and consultative process, particularly in the context of the #Shifthepower movement or calls for localization. However, these often do not get translated to action on the ground and examples of specific cases and steps that are taken to build theories of change under specific programs do not find articulation within the academic and development literature.
In this case study of building ToCs within the Oak Foundation India program along with their grantee organization ecosystem, the study team at Niiti aims to document the process, and to show ways in which participative and collaborative evaluation processes can be developed.
Methods
Interviews were conducted with all the members of the Niiti team who were involved in developing the ToCs through participatory workshops and individual consultations with grantees and their partner organizations. The material for this paper also emerged from the stakeholders’ workshops conducted (in August and November 2022), as well as several individual and group meetings with the grantees (between April and November 2022) where the ToC document drafts were discussed and views and concerns of the grantees, as well as Oak Foundation were heard by Niiti and feedback incorporated through an iterative process. The data from the interviews, workshops, and consultative meetings between Oak and Niiti were analyzed to bring out the main pathways adopted in building the ToC document as a guide for other organizations in building them in similar contexts. The data analysis was done through a coding method where similar themes were coded when they emerged in the data from the interviews and in documents created in the process of building the theories of change.
The authors of the paper are conscious of the subjectivities that exist while articulating this case study given that have been closely involved in the process. In a way to address this, the first author, who is part of the organization but was not closely involved in the development of the ToCs, viewed the process objectively through a fresh ‘outside’ lens. In doing so, more scope and space were given for reflection on the development of the ToCs to understand some of the challenges and critical dimensions in the approach to building the theories of change.
The Pathways to Develop Participatory and Consultative Theories of Change
We identify seven main pathways that emerged after analyzing the various steps that were key in developing the ToC for this program. For each of the pathways, we describe their importance and how they help in developing a participatory and consultative ToC. Before, highlighting the pathways, we show below through a figure (flow chart), the specific steps that led to the pathways identified in this case study.
Fig. 1 (Figure is not sourced from anywhere and made by the authors themselves)
The steps shown in the flow chart are significant to depict the organic process of ToC evolution (both for focus areas and program level) and the implementation of the evaluation culture in the Oak India program. However, these steps are not self-explanatory in themselves and only once analyzed along with specific processes adapted, led to the findings of this case study in the form of the seven pathways that can help develop a ToC through a participatory and consultative approach.
1) Process-focussed approach– The ToC in this case was built through a process-focussed approach. Specific steps were adopted in developing the process through which the ToCs evolved and were slowly built. It was viewed both by Niiti and Oak that for an evaluation culture to develop amongst organizations there was a need to create a process-driven approach to theories of change. This approach did not emerge in a vacuum but was a considered and deliberate one, based on the experiences of Niiti in other monitoring and evaluation projects over the years[1]. The idea was to create an internal culture of evaluation within organizations and to move away from an understanding of evaluation that is from the perspective of reporting alone. This need for building an evaluation culture is something that the Oak Foundation also strongly believed in, which then resulted in making the processes the focus for creating the theories of change. The Inception Report (Oct 2021) included a needs assessment of the identified 5 priority communities based on existing primary and secondary sources along with identifying the areas of further research. In addition to secondary research, grantees were consulted regularly in the process to understand their work and organization and the challenging context in which they carried out their interventions before we designed workshops to initiate the development of the TOCs. In all, 32 individual consultations were held with 13 grantee organizations, 2 reflection sessions with the entire grantee cohort along with 2 field visits, one each in the two focus areas to establish the rapport between Niiti and the grantees and also to have a first-hand understanding of how the grantees worked while navigating some of their challenges.
Making the process consultative from the beginning defined the process-focused approach to developing the ToC for this program. It was also important given the range of issues that grantees were engaged with. As articulated by one of the initial team members closely connected to developing the ToC, “We understood this was a complex program given the diverse types of organizations and the range of issues they were working on: alternative livelihood, climate resilience to sexual and reproductive health, gender-related issues and addressing patriarchy, better participation in local governance, etc. Given the varied issues and the communities, the grantees were working with, we wanted to develop a sound understanding of their work and context as a team.”
A conscious attempt was made to understand the most credible needs and evidence on them, in the focus areas, both through assessments conducted by the grantees and from other secondary sources. An initial inception report was then created to articulate the process through which the ToC will be developed and validated. Defining this process was important to make it relevant for the grantees based on their needs and capacities. How learning products and spaces can be created and used by the grantees and associated stakeholders was asked at the beginning of the process to develop the ToC and not as an afterthought which most often happens in programs. This became the stepping stone for making the process participatory and inclusive.
In defining the process based on the needs and capacities of the grantee organizations which emerged through consultations, there were choices given to all the grantees to determine their role in the process of creating the ToCs. They were also given a choice to determine the nature and engagement they would prefer, based on the time and effort they were willing to commit, as well as the availability of key resources for consultations. Some grantees participated more actively than others in the ecosystem, including new grantees who joined the Oak portfolio through the course of the ToC development process. However, all grantees, irrespective of their level of participation were allowed to provide feedback, add their concerns, many unique to their specific interventions and program areas, and provide examples to make the ToC more robust and practice-oriented. This availability of choice made the process more open for the grantees without making participation binding on them or forced.
The process approach, flexibility in terms of participation for the grantees, and transparency in the choice of processes that would be adopted helped in addressing the power differentials to a large extent between the donor organization, the learning partner, and the grantees, many of who were in various stages of their evolution in terms of adoption of monitoring and evaluation practices.
These process-centric approaches put the grantees at the center of the development of ToC, enabling them to be in charge of decision-making, as opposed to someone else making that decision for them, as it often happens in such evaluation design exercises.
2) Asking learning questions in grantee consultative workshops – A significant pathway created to develop the ToCs in this program was through the grantee consultation workshops. These workshops were not viewed as a space to guide or direct the grantees towards a specific path. Instead, a space was created where learning questions were posed before the grantees based on their focus areas and communities, making it a critically reflective process. Grantees working with the same communities and in the primary and secondary focus areas were categorized and based on their interest and ability to participate, 4 sets of workshops were designed, 1 each for all the focus areas identified. The workshops were facilitated in a consultative manner and the framework was largely driven by asking questions so that the answers to impact, outcomes, and outputs of the work that grantees were doing on the ground in the focus areas, emerged from them. This was challenging as often grantees spoke about several issues and work connected to a community and not necessarily about what they were doing in the area and several grantees working towards the same outcomes had different outputs and activities to reach them. The facilitator’s constant probing and questioning helped in making the process focused without devaluing the issues in general that grantees brought to the space and at the same time made the grantees come out with specific impacts, outcomes, and outputs of their work through the answers to the questions that were asked. For example, as articulated by one of the initial members facilitating the workshops – “the grantees working on indigenous communities would talk about several issues faced by these communities along with the ways to address them in the focus areas. However, we had to further ask them if all these issues they were addressing or which were the specific issues they were addressing and how? In this way slowly and steadily through constant questioning, we made the process reflective for the grantees and also more focussed and relevant to the development of the theories of change which all grantees could relate to while working with specific communities.”
The learning questions also helped the grantees to understand the difference between impact, outcomes, and outputs, and how to arrive at assumptions in a manner that was not overwhelming, especially for those who were not very familiar with the process. The process made them arrive at the assumptions on their own while they answered the questions asked by the facilitators related to their focus areas. The grantees could understand the differences between impact, outcomes, and outputs through discussions amongst themselves. They were encouraged to probe questions in groups and critically reflect in group activities to help them understand the ways to distinguish these concepts in Monitoring and Evaluation.
While the first three workshops had about 6-7 participating grantees each, based on their primary interest and chosen area of impact, the final workshop had almost 21 grantee participants. The third wave of the pandemic (March-May 2022) forced us to make this process hybrid, and workshops were planned both online and offline. To ensure a good learning experience for the participants as well as the facilitators, Miro was used – a visual platform that enables effective remote work and collaboration. The Miro board was effective in connecting the various views that were emerging from diverse sets of grantees and visually bringing them together to show specific impacts, outcomes, etc in the focus areas
The impact of this process can be gauged by the change in attitude and perception of participating grantees. In the first workshop, the first day of the 5-day workshop was challenging for the facilitators as the participating grantees were visibly apprehensive about the process, and were unsure how different members from different grantee organizations, each of whom was working in a different space, albeit with the same communities, as well as in different stages of evolution in terms of their understanding and adoption of monitoring and evaluation process could have a meaningful discussion. By the end of the last day of the first workshop, most participants willingly shared their positive experiences with other grantees (in the common e-groups) and also volunteered to engage more actively in the process than they had originally signed up to. The participant feedback is reflective of the experience they had.
“We were able to understand the ToC process and objective better and also others’ work. The facilitator’s patience and persevering prodding was much appreciated. Initially, it seemed like a lot of time to set aside. But it was very necessary and we are glad we invested the time.”
“We are now looking forward to an encore session and hoping to do some in person. We didn’t feel like we were disadvantaged because we were unfamiliar with the Theory of Change process. The facilitators were very supportive and we got the space and opportunity to share our views and experiences.”
“We understood how different organizations are contributing to the larger outcome. We have a clear picture of how to create a ToC now. We realize that this process helps us critically view our projects, and help us reflect better on what we can do better and differently to ensure better project outcomes.”
This locally-led approach pathway where a safe and open space for interaction was created through workshops helped in making the process participatory and building ground-level ownership for the theories of change in this program.
3) An inclusive process and conscious positionality – Making ToC development processes participatory in complex and dynamic programs with several grantees is challenging as every organization is at a different stage of its journey and with diverse types of capacities. In this program, the grantees ranged from professional organizations with skilled staff and capacities to smaller community-led organizations with lower staff capacities and skill sets. The question that was constantly reflected upon by the initial team and in Oak-Niiti meetings was around ways to ensure an inclusive approach where the voices of all grantees, as well as the communities with whom they work were given value and importance.
A key to making this process participatory was also possible because of the Oak Foundation and Niiti’s awareness and acknowledgment of their positionality. The fact that Oak Foundation was aware of its positionality as a donor and the power differentials that this position creates with the donors made it easier to make this process locally led with a participatory approach. The Niiti team, based on its own experience of Monitoring and Evaluation and community-led field work, understood the insecurities and power dynamics that often exist between those leading evaluation and the organizations doing the development work on the ground. This awareness and acknowledgment of the power differentials helped build trust with the grantees in a way that the organizations found the space to be comfortable to share their views and concerns.
The level of trust was initially built through the workshops where grantee organizations viewed the Niiti team more as facilitators of this process rather than as ‘experts’ who were there to guide or direct them. Another tool used to build trust through providing spaces for grantees to reflect and voice their concerns, feedback, and suggestions was through regular group sessions, conducted both online (in the first half of 2022) and offline (in the second half of 2022). These reflective sessions had an agenda for discussions that were shared ahead of the meeting, where updates on the stage of the ToC process were shared transparently, and plans for the next step were discussed and finalized together with the participating grantees. Consensus through online polls was built for decisions around the frequency and nature of capacity-building topics and delivery, the level of engagement that grantees preferred in workshops, and the preferred time window to conduct them. Individual grantee consultations were also a critical tool adopted to seek individual grantee feedback and concerns on the alignment of the ToC, indicators developed, and methodologies like Outcome Harvesting and Most Significant Change method, with their organizational working systems. Some of the key questions that were discussed in the reflection meetings, especially after the field visits, were very pertinent.
“Do the outcomes and outputs highlighted in the ToC relevant to your area of work, adequately represent your impact objectives and key result areas?
Do the strategies highlighted in the ToCs reflect broadly the activities that you are doing on the ground?
Do you agree with the gaps identified in the ToC? Do you suggest any other set of gaps that need to be captured?”
These discussions enabled us to ensure that the ToC was grounded in the work systems of the grantee organizations, looked at evaluating both short- and long-term impacts, and most importantly built a sense of ownership among the grantee organizations. There were two other pathways that we believe were important to capture diversity in voices and ensure the process was truly participatory and consultative.
4) Methods to enhance stakeholder involvement – During the course of developing the ToCs for this program, some specific methods were used to ensure that all the stakeholders were involved in the process of development and subsequent implementation. An important way was to create a channel for one-on-one consultations with the grantees. These consultations were scheduled regularly not just with the grantees who participated in the process of developing the ToC, but also with newer grantee organizations who were inducted into the ecosystem through the course of the year, some as late as Nov 2022. The Niiti team facilitated these discussions that were centered around various aspects of their work, the communities they were working with, and how they visualized their intervention to shape up.
These consultations were multilingual, where facilitators switched between Bangla, a language that most grantees working in West Bengal were familiar with, Hindi, and English. Language barriers often create communication gaps and apprehension to participate, and therefore vernacular language was consciously used to build greater participation and engagement.
Developing the ToC also required the participation of all stakeholders, especially to include the voice of communities and an important method to ensure this was through field visits. The field visits were conducted in two stages, once before the development of the ToCs and later after the first draft was ready. Validating the ToCs through field visits and community observations was initially planned more as a check to validate the ToCs as opposed to capturing community voices actively in the documents. However, regular consultations between Oak and Niiti resulted in bringing forward the importance of the process of including community voices, to ensure implementability and higher adoption of the ToCs and evaluation methods with the entire spectrum of stakeholders. Field visits were conducted to meet with various categories within the five communities, namely tea garden workers, communities in the Sunderbans, migrant and unorganized workers, and indigenous communities, and actively seek their inputs on the ToCs. Detailed stakeholder-oriented evaluation tools were developed, and focus group discussions and key informant interviews were conducted with all the communities and other primary stakeholders.
These field visits, conducted over a period of 3 weeks, enabled three types of validations. First, the needs of the communities that existed on the ground included in the ToC document were confirmed, and any new needs that the team learned of were recorded with evidence. Second, a better understanding of how the planned interventions were implemented on the ground enabled critical corrections and additions to the ToCs. Third, these field visits enabled us to identify some gaps that existed in the current ToC. These gaps existed as none of the grantee organizations was working on these issues and though they were important to bring a systemic change that the program aimed to achieve over the years, they did not come up either during the workshops or the individual or group consultations.
This method of stakeholder involvement through field visits and interactions with the communities was a pertinent step to ensure all stakeholder voices were captured and included in the ToC. The direct participation of the communities in this process made it more substantially inclusive and grantee organizations also could relate better to both the process and the final document that emerged from it. It also enabled us to understand the systemic gaps that existed, that could inform the donor of potential limitations or barriers, or even opportunities in their quest for systemic change. This process of conducting field visits and involving community voices, however, has its own challenges. The first challenge was to make this process time efficient while capturing the breadth of stakeholder diversity. This was mitigated through a thorough preparation that captured the various perspectives of diverse communities in a time-efficient manner. The second challenge was to convince the grantee organizations that this was a useful and important step that was not intended as an evaluation of their work or activities but to further substantiate the ToC through an outside lens which Niiti had the capacity to do. The trust developed over the months through consultations and rapport-building, was useful in addressing this challenge.
5) Capturing various types of diversities – An important pathway to the process of developing the ToC in this program was through capturing the existing diversities of a complex program involving so many grantees (32, as of Dec 2022). The diversities were in the types of grantee organizations, their capacities, the nature of work they were doing, the communities that they worked with, and the size and outreach of the organizations. This was made possible by making the process dynamic and not a linear process where one or two consultations were done and then the ToC was developed. Also, the process was dynamic because of the openness to change and incorporation of new steps, and being self-reflective in this entire process of developing the ToC. An example of this was the adaptive changes that were made during the field visits based on the situation.
In the process of developing the ToC, a space was created for organizations to join later and give feedback. Organizations were given multiple opportunities to consider the document and come forward with concerns or inputs that were discussed and then incorporated in cases found mutually valid. The process was therefore never closed-ended with no scope available to make changes or participate in a different mode or way. These steps ensured that the diversity was captured given that organizations were not coming from the same space and had the same approach to implementing the program on the ground.
In developing the methodology for participation, suggesting ways to conduct the field visits, and leading the consultative workshops, diversity was always given primary importance making it an adaptable and accommodative process rather than suggesting ‘one method fits all’. This can further be substantiated through the ways in which grantee organizations who have joined the process later have also not just embraced the ToC, but have also found it relevant and aligned to their work. The statement by one of the grantee organizations who joined the process as late as Nov 2022 (during the second capacity-building workshop) sums this up.
“Even though we didn’t participate in the ToC development workshops, the outcomes and impacts arrived at in the ToC are relevant to us. Some activities may not be adequately reflected in the ToCs currently, but we can understand the process of how our actions could lead to a larger systemic change. We also are able to assess the gaps we may have and improve the scope for collaboration with other organizations to reach the larger goal.”
The process also allowed the Niiti team to develop a program-level ToC for Oak Foundation based on the learnings during the focus area of ToC development, highlighting the gaps and opportunities for the donor organization.
6) Representation of the ToC visually – An important aspect of the robustness and relevance of any ToC is in its practical implementation. For the best adoption of evaluation systems and to enable stakeholders in any ecosystem to adopt an evaluation culture, it has to be easy to understand and appeal to their sensibilities. The visual representation of any ToC is, therefore, a significant aspect of the process of its development. The connections and dependencies between the various elements of the ToC like impacts, outcomes, outputs, assumptions, and inputs are reflected as a narrative and the visual representation makes it more appealing and comprehensive to a wide range of stakeholders, especially practitioners. This gap in representation exists even in the most robust processes of ToC development because often those creating visual representations and those working on the process of developing the ToCs work in silos. This representation is a struggle for most and was no different in this case as well. It was first addressed by adopting a visual platform like Miro to capture the process in a comprehensive and simple way. Second, a platform like Canva was used to develop a visual representation of all the complex dependencies of the logical framework of the ToC and also Powerpoint to create a simpler version of the ToC that could be used by Oak as well as all grantee organizations as a summary of their direct and indirect spheres of influence. This was accompanied by a narrative to provide detailed insights where required as an accompaniment to the visual depiction. This required a lot of flexibility and adaptability with several iterations to create the visual representations and organizations while developing this need to take this into account both in terms of time and cost needed for this purpose.
7) The way forward after developing a ToC – In the next steps after the four ToCs have been developed for the two focus areas and in relation to the five communities, the pathway is towards the implementation of these theories by the grantee organizations. In this case, since the entire process of developing the ToCs was done through a participatory approach, the organizations have a higher sense of ownership of the document and leverage it to improve their own evaluation and learning. This comment by one of the organizations shared during the workshop to discuss the ToCs, expresses the overall experience we heard from the participating grantee organizations.
“We found the process not only self-reflective but also as a step for us to create our own organization’s ToCs. We understood for the first time why it is important to develop a ToC and the practical need for the same and these emerged for us gradually as we progressed towards developing the ToC for this program with Niiti.”
The steps ahead are also consciously being co-created with the stakeholders in helping them use the ToCs in an effective and efficient way. For example – in identifying the most relevant indicators for the grantee organizations to measure their interventions in ways that will help them to understand the impact they are creating. There has been a deliberative attempt to have the process continue being consultative while deciding on ways to formulate the evaluation method and develop the capacities of the grantees so that they can use methods like Outcome Harvesting and Most Significant Change to capture the change effectively. However, this process is left open for grantees to adapt while they implement the ToC in specific focus areas ensuring that they can formulate timelines based on the stage of their intervention rather than having similar timelines and rigid ways of using these methods. The templates for the evaluations based on the ToCs are also being visualized through a consultative process to understand the possibilities and challenges ahead. Therefore, it is not just the process of developing the ToC that is of value when done through a participatory approach but the next steps and the way forward also need to be the same to ensure that the ToCs are practically used by organizations and not remain as documents which occupy their shelves.
Conclusion and lessons drawn
Co-creation, stakeholder involvement, creating safe spaces for sharing experiences, trust building, being conscious of own positionalities, and capturing diversity are all aspects that are often processes adopted in most participatory approaches. However, how these unfold on the ground, especially in complex programs where the goal is not only to develop a ToC but also to build systems and processes to practically implement it, is what the paper shows through the case of ToC development by the Niiti consulting team in the Oak India program. There is usually no method of overcoming the power differentials in development programs. However, they can be consciously addressed and the gap can be reduced particularly by adopting processes that create spaces for self-reflection and shared reflection within and amongst organizations. A key lesson that can be drawn through the process is the relevance of time and resources needed which is not possible perhaps in every program. Time was a key challenge in the Oak India program as well with specific timelines to be met and bringing organizations together for workshops at the same across geographies. The process of trust building between the facilitators of developing a ToC and organizations is a slow and dynamic process that occurs over a period of time and programs need to prepare themselves for the same. The challenge faced in the program was when new organizations were included and similar levels of trust had to be created with them as existed with the organizations that were part of the process from the beginning.
Another important learning is to keep consultations continuous and not episodic, where every step in the evolution of the ToC is consultative and becomes a part of the program. Programs and organizations often aspire to develop an evaluation culture through developing a ToC. This case shows that the emergence of an evaluation culture that can remain sustained is possible when organizations take ownership of the evaluation method and the development of the ToC for the same
References
Behar, Anurag. 2022. ‘Theories of Change are popular but could prove perilous as well’. Mint, 22 June 2022. Available at: https://www.livemint.com/opinion/columns/theories-of-change-are-popular-but-could-prove-perilous-as-well-11655915613170.html
Blamey, A., & Mackenzie, M.(2007. ‘Theories of change and realistic evaluation: peas in a pod or apples and oranges?’. Evaluation, 13(4): 439-455.
Budzyna, Laura. 2021. Theory of Change: A Tool for Participatory Design. MITD-Lab, Available at: https://d-lab.mit.edu/news-blog/blog/theory-change-tool-participatory-design, Accessed on?
Davies, R. 2004. ‘Scale, complexity and the representation of theories of change’. Evaluation, 10(1): 101-121.
Davies, R. 2018. ‘Representing theories of change: Technical challenges with evaluation consequences’. Journal of Development Effectiveness, 10(4): 438-461.
DuBow, W. M., & Litzler, E. 2019. ‘The development and use of a theory of change to align programs and evaluation in a complex, national initiative’. American Journal of Evaluation, 40(2): 231-248.
Ghate, D. 2018. Developing theories of change for social programs: co-producing evidence-supported quality improvement. Palgrave Communications, 4(1): 1-13.
Mackenzie, M., & Blamey, A. 2005. ‘The practice and the theory: lessons from the application of theories of change approach’. Evaluation, 11(2): 151-168.
Mayne, J. 2017. ‘Theory of change analysis: Building robust theories of change’. Canadian Journal of Program Evaluation, 32(2).
Rasumussen, Julz. 2016. ‘What’s wrong with the Theory of Change?’ Infitias International. Available at: http://infinitasinternational.com/category/theory-of-change/, Accessed on?
Srinivasan, D., Vaidyanathan, M. & Srivastav, S. 2022. Building an Evaluation Culture in Organizations for maximising impact. White Paper, Niiti Consulting Pvt.Ltd. Available at: https://www.researchgate.net/publication/359617032_Building_an_Evaluation_Culture_in_Organisations_for_maximizing_Impact, Accessed on?
Stein, D., & Valters, C. 2012. Understanding the theory of change in international development. London: The Asia Foundation.
Taplin, D. H., & Clark, H. 2012. Theory of change basics: A primer on the theory of change. New York NY: ActKnowledge, 844, 845.
Van Wessel, M. 2018. ‘Narrative Assessment: A new approach to the evaluation of advocacy for development’. Evaluation, 24(4): 400-418.
Ho, W., van Wessel, M. G. J., & Tamas, P. A. 2020. ‘The hidden life of theories of change’. Policy Brief, Wageningen (the Netherlands): WUR/Hivos.
Wilkinson, H., Hills, D., Penn, A., & Barbrook-Johnson, P. 2021. ‘Building a system-based theory of change using participatory systems mapping’. Evaluation, 27(1): 80-101.
[1] See Nokia India Annual CSR Report based on the report prepared by niiti consulting private limited. Available at: https://niiticonsulting.com/wp-content/uploads/2020/05/Nokia_India_CSR_Report_-_2018-19_-_V2E_24th_Nov.pdf