I’ve been writing a series of blog posts about climbing each of the 48 mountains in New Hampshire that are at least 4000 feet tall. And, each time, I’ve also been reflecting a bit on the journey since I joined Peace Corps, 33 years ago: on development, social justice, conflict, experiences along the way, etc.
So far, I’ve described climbing 35 of the 48 peaks, and covered my journey from Peace Corps in Ecuador (1984-86) through to my arrival in Sydney in 2009, where I joined ChildFund Australia as the first “International Program Director.”
Last time I described the ChildFund Australia “Development Effectiveness Framework,” the system that would help us make sure we were doing what we said we were going to do and, crucially, verifying that we were making a difference in the lives of children and young people living in poverty. So we could learn and improve our work…
This time, I want to go into more depth on one component of the DEF, the “Case Studies” that described the lived experience of people that we worked with. Next time, I’ll describe how we measured the impact of our work.
On 10 August, 2017, I climbed three 4000-footers in one very long day: Bondcliff (4265ft, 1300m), Mt Bond (4698ft, 1432m), and West Bond (4540ft, 1384m). This was a tough day, covering 22 miles and climbing three very big mountains. At the end of the hike, I felt like I was going to lose the toenails on both big toes (which, in fact, I did!) … it was a bit much!
Last time I wrote about climbing to the top of Bondcliff, the first summit of that day. This time, I will describe the brief walk from there to the top of Mt Bond, the tallest of the three Bonds. And next time I’ll finish describing that day, with the ascent of West Bond and the return to the trail-head at Lincoln Woods.
To skip the description of my ascent of Mt Bond, and go directly to my description of case studies in ChildFund Australia’s DEF, click here.
As I described last time, I arrived at the top of Bondcliff at about 10:30am, having left the trail-head at Lincoln Woods Visitor Center just after 6:30am. I was able to get an early start because I had stayed the night before at Hancock Campsite on the Kancamagus road, just outside of Lincoln, New Hampshire.
It was a bright and mostly-sunny day, with just a few clouds and some haze. The path between Bondcliff and Mt Bond is quite short – really just dropping down to a saddle, and then back up again, only 1.2 miles:
It took me about an hour to cover that distance and reach the top of Mt Bond from Bondcliff at 11:30am. The path was rocky as it descended from Bondcliff, in the alpine zone, with many large boulders as I began to go back up towards Mt Bond – some scrambling required.
This photo was taken at the saddle between Bondcliff and Mt Bond: on the left is Bondcliff, on the right is West Bond, and in the middle, in the distance, is Franconia Ridge; Mt Bond is behind me. A glorious view on an amazing day for climbing:
It got even steeper climbing up from the saddle to the summit, passing through some small pine shrubs, until just before the top.
The views were spectacular at the summit of Mt Bond, despite the sky being slightly hazy – I could see the four 4000-footers of the Franconia Ridge to the west and Owl’s Head in the foreground, the Presidential Range to the east, and several other 4000-footers to the south and south-west:
And I had a nice view back down the short path from the top of Bondcliff:
There were a few people at the top, and I had a brief conversation with a couple that were walking from Zealand trailhead across the same three mountains I was climbing, and finishing at Lincoln Woods. This one-way version of what I was doing in an up-and-back trip was possible because they had left a car at Lincoln Woods, driving to the Zealand trailhead in a second vehicle. They would then ferry themselves back to Zealand from Lincoln Woods.
Kindly, they offered to pick up my car down at Lincoln Woods and drive it to Zealand, which would have saved me three miles. I should have accepted, because finishing what became 22 miles, and three 4000-foot peaks, would end up hobbling me for a while, and causing two toenails to come off! But I didn’t have a clear sense of how the day would go, so I declined their offer, with sincere thanks…
Getting to the top of Mt Bond was my 36th 4000-footer – just 12 more to go!
I didn’t stay too long at the top of Mt Bond on the way up, continuing towards West Bond… stay tuned for that next time!
Case Studies in ChildFund Australia’s “Development Effectiveness Framework”
Jean and I had moved to Sydney in July of 2009, where I would take up the newly-created position of International Program Director for ChildFund Australia. It was an exciting opportunity for me to work in a part of the world I knew and loved (Southeast Asia: Cambodia, Laos, Myanmar and Viet Nam) and in a challenging new country (Papua New Guinea). It was a great chance to work with some really amazing people – in Sydney and in our Country Offices… to use what I had learned to help build and lead effective teams. Living in Sydney would not be a hardship post, either! Finally, it was a priceless chance for me to put together a program approach that incorporated everything I had learned to that point, over 25 years working in poverty reduction and social justice.
In the previous article in this series, I described how we developed a “Development Effectiveness System” (“DEF”) for ChildFund Australia, and I went through most of the components of the DEF in great detail.
My ambition for the DEF was to bring together our work into one comprehensive system – building on our Theory of Change and organizational Vision and Mission, creating a consistent set of tools and processes for program design and assessment, and making sure to close the loop with defined opportunities for learning, reflection, and improvement.
Here is the graphic that we used to describe the system:
As I said last time, I felt that three components of the DEF were particularly innovative, and worth exploring in more detail in separate blog articles:
- I will describe components #2 (“Outcome Indicator Surveys) and #12 (Statement of Impact) in my next article. Together, these components of the DEF were meant to enable us to measure the impact of our work in a robust, participatory way, so that we could learn and improve;
- this time, I want to explore component #3 of the DEF: “Case Studies.”
It might seem strange to say it this way, but the “Case Studies” were probably my favorite of all the components of the DEF! I loved them because they offered direct, personal accounts of the impact of projects and programs from children, youth, men and women from the communities in which ChildFund worked and the staff and officials of local agencies and government offices with whom ChildFund partnered. We didn’t claim that the Case Studies were random or representative samples; rather, their value was simply as stories of human experience, offering insights would not have been readily gained from quantitative data.
Why was this important? Why did it appeal to me so much?
Over my years working with international NGOs, I had become uneasy with the trend towards exclusive reliance on linear logic and quantitative measurement, in our international development sector. This is perhaps a little bit ironic, since I had joined the NGO world having been educated as an engineer, schooled in the application of scientific logic and numerical analysis for practical applications in the world.
Linear logic is important, because it introduces rigor in our thinking, something that had been weak or lacking when I joined the sector in the mid-1980s. And quantitative measurement, likewise, forced us to face evidence of what we had or had not achieved. So both of these trends were positive…
But I had come to appreciate that human development was far more complex than building a water system (for example), much more complicated than we could fully capture in linear models. Yes, a logical, data-driven approach was helpful in many ways, perhaps nearly all of the time, but it didn’t seem to fit every situation in communities that I came to know in Latin America, Africa, and Asia. In fact, I began to see that an over-emphasis on linear approaches to human development was blinding us to ways that more qualitative, non-linear thinking could help; we seemed to be dismissing the qualitative, narrative insights that should also have been at the heart of our reflections. No reason not to include both quantitative and qualitative measures. But we weren’t.
My career in international development began at a time when the private-sector, business culture, started to influence our organizations in a big way: as a result of the Ethiopian famine of the mid-1980’s, INGOs were booming and, as a result, were professionalizing, introducing business practices. All the big INGOs started to bring in people from the business world, helping “professionalize” our work.
I’ve written elsewhere about the positive and negative effects that business culture had on NGOs: on the positive side, we benefited from systems and approaches the improved the internal management of our agencies, such as clear delegations of authority, financial planning and audit, etc. Overall, it was a very good, and very necessary evolution.
But there were some negatives. In particular, the influx of private-sector culture into our organizations meant that:
- We began increasingly to view the world as a linear, logical place;
- We came to embrace the belief that bigger is always better;
- “Accountability” to donors became so fundamental that sometimes it seemed to be our highest priority;
- Our understanding of human nature, of human poverty, evolved towards the purely material, things that we could measure quantitatively.
I will attach a copy of the article I wrote on this topic here: mcpeak-trojan-horse.
In effect, this cultural shift had the effect of emphasizing linear logic and quantitative measures to such a degree, with such force, that narrative, qualitative approaches were sidelined as, somehow, not business-like enough.
As I thought about the overall design of the DEF, I wanted to make 100% sure that we were able to measure the quantitative side of our work, the concrete outputs that we produced and the measurable impact that we achieved (more on that next time). Because the great majority of our work was amenable to that form of measurement, and being accountable for delivering the outputs (projects, funding) that we had promised was hugely important.
But I was equally determined that we would include qualitative elements that would enable us to capture the lived experience of people who facing poverty. In other words, because poverty is experienced holistically by people, including children, in ways that can be captured quantitatively and qualitatively, we needed to incorporate both quantitative and qualitative measurement approaches if we were to be truly effective.
The DEF “Case Studies” was one of the ways that we accomplished this goal. It made me proud that we were successful in this regard.
There was another reason that I felt that the DEF Case Studies were so valuable, perhaps just as important as the way that they enabled us to measure poverty more holistically. Observing our organizations, and seeing my own response to how we were evolving, I clearly saw that the influence of private-sector, business culture was having positive and negative effects.
One of the most negative impacts I saw was an increasing alienation of our people from the basic motivations that led them to join the NGO sector, a decline in the passion for social justice that had characterized us. Not to exaggerate, but it seemed that we were perhaps losing our human connection with the hope and courage and justice that, when we were successful, we helped make for individual women and men, girls and boys. The difference we were making in the lives of individual human beings was becoming obscured behind the statistics that we were using, behind the mechanical approaches we were taking to our work.
Therefore, I was determined to use the DEF Case Studies as tools for reconnecting us, ChildFund Australia staff and board, to the reason that we joined in the first place. All of us.
So, what were the DEF Case Studies, and how were they produced and used?
In practice, Development Effectiveness and Learning Managers in ChildFund’s program countries worked with other program staff and partners to write up Case Studies that depicted the lived experience of people involved in activities supported by ChildFund. The Case Studies were presented as narratives, with photos, which sought to capture the experiences, opinions and ideas of the people concerned, in their own words, without commentary. They were not edited to fit a success-story format. As time went by, our Country teams started to add a summary of their reflections to the Case Studies, describing their own responses to the stories told there.
Initially we found that field staff had a hard time grasping the idea, because they were so used to reporting their work in the dry, linear, quantitative ways that we had become used to. Perhaps program staff felt that narrative reports were the territory of our Communications teams, meant for public-relations purposes, describing our successes in a way that could attract support for our work. Nothing wrong with that, they seemed to feel, but not a program thing!
Staff seemed at a loss, unable to get going. So we prepared a very structured template for the Case Studies, specifying length and tone and approach in detail. This was a mistake, because we really wanted to encourage creativity while keeping the documents brief; emphasizing the “voice” of people in communities rather than our own views; covering failures as much as successes. Use of a template tended to lead our program staff into a structured view of our work, so once we gained some experience with the idea, as staff became more comfortable with the idea and we began to use these Case Studies, we abandoned the rigid template and encouraged innovation.
So these Case Studies were a primary source of qualitative information on the successes and failures of ChildFund Australia’s work, offering insights from children, youth and adults from communities where we worked and the staff of local agencies and government offices with whom ChildFund Australia partnered.
In-country staff reviewed the Case Studies, accepting or contesting the opinions of informants about ChildFund Australia’s projects. These debates often led to adjustments to existing projects but also triggered new thinking – at the project activity level but also at program level or even the overall program approach.
Case Studies were forwarded to Sydney, where they were reviewed by the DEF Manager; some were selected for a similar process of review by International Program staff, members of the Program Review Committee and, on occasion, by the ChildFund Australia Board.
The resulting documents were stored in a simple cloud-based archive, accessible by password to anyone within the organization. Some Case Studies were also included on ChildFund Australia’s website; we encouraged staff from our Communications team in Sydney to review the Case Studies and, if suitable, to re-purpose them for public purposes. Of course, we were careful to obtain informed consent from people included in the documents.
Through Case Studies, as noted above, local informants were able to pass critical judgement on the appropriateness of ChildFund’s strategies, how community members perceived our aims and purposes (not necessarily as we intended); and they could alert us to unexpected consequences (both positive and negative) of what we did.
For example, one of the first Case Studies written up in Papua New Guinea revealed that home garden vegetable cultivation not only resulted in increased family income for the villager concerned (and positive impact on children in terms of nutrition and education), it also enhanced his social standing through increasing his capacity to contribute to traditional cultural events.
Here are three images from that Case Study:
And here is a copy of the Case Study itself: PNG Case Study #1 Hillary Vegetable farming RG edit 260111. Later I was able to visit Hillary at his farm!
Another Case Study came from the ChildFund Connect project, an exciting effort led by my former colleagues Raúl Caceres and Kelly Royds, who relocated from Sydney to Boston in 2016. I climbed Mt Moriah with them in July, 2017, and also Mt Pierce and Mt Eisenhower in August of 2016. ChildFund Connect was an innovative project that linked children across Laos, Viet Nam, Australia and Sri Lanka, providing a channel for them directly to build understanding of their differing realities. This Case Study on their project came from Laos: LAO Case Study #3 Connect DRAFT 2012.
In a future article in this series, I plan on describing work we carried out building the power (collective action) of people living in poverty. It can be a sensitive topic, particularly in areas of Southeast Asia without traditions of citizen engagement. Here is a Case Study from Viet Nam describing how ChildFund helped local citizens connect productively with authorities to resolve issues related to access to potable water: VTM Case Study #21 Policy and exclusion (watsan)-FINAL.
Dozens of Case Studies were produced, illustrating a wide range of experiences with the development processes supported by ChildFund in all of the countries where we managed program implementation. Reflections from many of these documents helped us improve our development practice, and at the same time helped us stay in touch with the deeper purpose of our having chosen to work to promote social justice, accompanying people living in poverty as they built better futures.
A few of the DEF Case Studies focused, to some extent, on ChildFund Australia itself. For example, here is the story of three generations of Hmong women in Nonghet District in Xieng Khoung Province in Laos. It describes how access to education has evolved across those generations: LAO Case Study #5 Ethnic Girls DRAFT 2012. It’s a powerful description of change and progress, notable also because one of the women featured in the Case Study was a ChildFund employee, along with her mother and daughter!
Two other influential Case Studies came from Cambodia, both of which touched on how ChildFund was attempting to manage our child-sponsorship mechanisms with our programmatic commitments. I’ve written separately, some time ago, about the advantages of child sponsorship: when managed well (as we did in Plan and especially in ChildFund Australia), and these two Case Studies evocatively illustrated the challenge, and the ways that staff in Cambodia were making it all work well.
One Case Study describes some of the tensions implicit in the relationship between child sponsorship and programming, and the ways that we were making progress in reconciling these differing priorities: CAM Case Study 6 Sponsorship DRAFT 2012. This Case Study was very influential, with our staff in Cambodia and beyond, with program staff in Sydney, and with our board. It powerfully communicated a reality that our staff, and families in communities, were facing.
A second Case Study discussed how sponsorship and programs were successfully integrated in the field in Cambodia: CAM Case Study #10 Program-SR Integration Final.
As I mentioned last time, given the importance of the system, relying on our feeling that the DEF was a great success wasn’t good enough. So we sought expert review, commissioning two independent, expert external reviews of the DEF.
The first review (attached here: External DEF Review – November 2012), which was concluded in November of 2012, took place before we had fully implemented the system. In particular, since Outcome Indicator Surveys and Statements of Impact (to be covered in my next blog article) were implemented only after three years (and every three years thereafter), we had not yet reached that stage. But we certainly were quite advanced in the implementation of most of the DEF, so it was a good time to reflect on how it was going.
I included an overview of the conclusions reached by both reviewers last time. Here I want to quote from the first evaluation, with particular reference to the DEF Case Studies:
One of the primary benefits of the DEF is that it equips ChildFund Australia with an increased quantity and quality of evidence-based information for communications with key stakeholders including the Board and a public audience. In particular, there is consolidated output data that can be easily accessed by the communications team; there is now a bank of high quality Case Studies that can be drawn on for communication and reflection; and there are now dedicated resources in-country who have been trained and are required to generate information that has potential for communications purposes. The increase in quantity and quality of information equips ChildFund Australia to communicate with a wide range of stakeholders.
One of the strengths of the DEF recognized by in-country staff particularly is that the DEF provides a basis for stakeholders to share their perspectives. Stakeholders are involved in identifying benefits and their perspectives are heard through Case Studies. This has already provided a rich source of information that has prompted reflection by in-country teams, the Sydney based programs team and the ChildFund Australia Board.
This focus on building tools, systems and the overall capacity of the organization places ChildFund Australia in a strong position to tackle a second phase of the DEF which looks at how the organization will use performance information for learning and development. It has already started on this journey, with various parts of the organization using Case Studies for reflection. ChildFund Australia has already undertaken an exercise of coding the bank of Case Studies to assist further analysis and learning. There is lots of scope for next steps with this bank of Case Studies, including thematic reflections. Again, the benefits of this aspect have not been realised yet as the first stages of the DEF roll-out have been focused on data collection and embedding the system in CF practices.
In most Country Offices, Case Studies have provided a new formal opportunity for country program staff to reflect on their work and this has been used as a really constructive process. The Laos Country Office is currently in the process of translating Case Studies so that they can be used to prompt discussion and learning at the country level. In PNG, the team is also interested in using the Case Studies as a communication tool with local communities to demonstrate some of the achievements of ChildFund Australia programs.
In some cases, program staff have found Case Studies confronting when they have highlighted program challenges or weaknesses. The culture of critical reflection may take time to embed in some country offices and may be facilitated by cross-country reflection opportunities. Currently, however, Country Office staff do not know how to access Case Studies from other country programs. ChildFund Australia is exploring how the ‘bank’ of DEF Case Studies would be most accessible and useful to country office personnel.
One of the uses of Case Studies has been as a prompt for discussion and reflection by the programs team in Sydney and by the Board. Case Studies have been seen as a really useful way to provide an insight into a program, practice and ChildFund Australia achievements.
At an organizational level, an indexing and cross-referencing system has been implemented which enables Case Studies to be searched by country and by theme. The system is yet to be introduced to MEL and Program users, but has potential to be a very useful bank of qualitative data for reflection and learning. It also provides a bank of data from which to undertake thematic reflections across and between countries. One idea for consideration is that ChildFund draw on groups of Case Studies to develop practice notes.
In general Case Studies are considered to be the most ‘successful’ part of the DEF by those involved in collecting information.
The second reviewer concentrated on other components, mainly aspects I will describe in more detail in my next article, not so much the Case Studies…
So the Case Studies were a very important element in the overall DEF. I tried very hard to incorporate brief reflections on selected Case Studies at every formal meeting of the International Program Team, of ChildFund Australia’s Program Review Committee, and (less frequently) at meetings of our Board of Directors. More often than not, time pressures on the agendas of these meetings led to us dropping the Case Studies from discussion, but often enough we did spend time (usually at the beginning of the meetings) reflecting on what we saw in them.
At the beginning, when we first began to use the Case Studies, our discussion tended to be mechanical: pointing out errors in the use of English, or questioning how valid the observations might be, challenging the statistical reliability of the conclusions. But, over time, I noticed that our teams began to use the Case Studies as they were designed: to gain insight into the lived experience of particular human beings, and to reconnect with the realities of people’s struggle for better lives for themselves and their children.
This was a great success, and really worked as I had hoped. The Case Studies complemented the more rigorous, quantitative components of the DEF, helping the system be holistic, enabling us to see more deeply into the effect that our work was having while also enhancing our accountability.
Next time, I will describe getting to the top of West Bond, and all the way down the 11 miles from there to the Lincoln Woods parking lot, where I staggered back to my car with such damage to my feet that I soon would lose toenails on both my big toes! And I will share details of the final two components of the DEF that I want to highlight: the Outcome Indicator Surveys and Statements of Impact were probably the culmination of the whole system.
So, stay tuned!
Here are links to all the blogs in this series. There are 48 articles, including this one, each one about climbing one of New Hampshire’s 4000-footers, and also reflecting on a career in international development:
- Mt Tom (1) – A New Journey;
- Mt Field (2) – Potable Water in Ecuador;
- Mt Moosilauke (3) – A Water System for San Rafael (part 1);
- Mt Flume (4) – A Windmill for San Rafael (part 2);
- Mt Liberty (5) – Onward to Colombia, Plan International in Tuluá;
- Mt Osceola (6) – Three Years in Tuluá;
- East Osceola (7) – Potable Water for Cienegueta;
- Mt Passaconaway (8) – The South America Regional Office;
- Mt Whiteface (9) – Empowerment!;
- North Tripyramid (10) – Total Quality Management for Plan International;
- Middle Tripyramid (11) – To International Headquarters!;
- North Kinsman (12) – Fighting Fragmentation and Building Unity: New Program Goals and Principles for Plan International;
- South Kinsman (13) – A Growth Plan for Plan International;
- Mt Carrigain (14) – Restructuring Plan International;
- Mt Eisenhower (15) – A Guest Blog: Max van der Schalk Reflects on 5 Years at Plan’s International Headquarters;
- Mt Pierce (16) – Four Years At Plan’s International Headquarters;
- Mt Hancock (17) – Hanoi, 1998;
- South Hancock (18) – Plan’s Team in Viet Nam (1998-2002);
- Wildcat “D” Peak (19) – Plan’s Work in Viet Nam;
- Wildcat Mountain (20) – The Large Grants Implementation Unit in Viet Nam;
- Middle Carter (21) – Things Had Changed;
- South Carter (22) – CCF’s Organizational Capacity Assessment and Child Poverty Study;
- Mt Tecumseh (23) – Researching CCF’s New Program Approach;
- Mt Jackson (24) – The Bright Futures Program Approach;
- Mt Isolation (25) – Pilot Testing Bright Futures;
- Mt Lincoln (26) – Change, Strategy and Culture: Bright Futures 101;
- Mt Lafayette (27) – Collective Action for Human Rights;
- Mt Willey (28) – Navigating Principle and Pragmatism, Working With UUSC’s Bargaining Unit;
- Cannon Mountain (29) – UUSC Just Democracy;
- Carter Dome (30) – A (Failed) Merger In the INGO Sector (1997);
- Galehead Mountain (31) – What We Think About When We Think About A Great INGO Program;
- Mt Garfield (32) – Building Strong INGO Teams: Clarity, Trust, Inspiration;
- Mt Moriah (33) – Putting It All Together (Part 1): the ChildFund Australia International Program Team;
- Owls’ Head (34) – Putting It All Together (Part 2): ChildFund Australia’s Theory of Change;
- Bondcliff (35) – ChildFund Australia’s Development Effectiveness System;
- West Bond (36) – “Case Studies” in ChildFund Australia’s Development Effectiveness System;
- Mt Bond (37) – Impact Assessment in ChildFund Australia’s Development Effectiveness System;
- Mt Waumbek (38) – “Building the Power of Poor People and Poor Children…”
- Mt Cabot (39) – ChildFund Australia’s Teams In Cambodia, Laos, Myanmar, Papua New Guinea, and Viet Nam;
- North Twin (40) – Value for Money;
- South Twin (41) – Disaster Risk Reduction;
- Mt Hale (42) – A “Golden Age” for INGOs Has Passed. What Next?;
- Zealand Mountain (43) – Conflict: Five Key Insights;
- Mt Washington (44) – Understanding Conflicts;
- Mt Monroe (45) – Culture, Conflict;
- Mt Madison (46) – A Case Study Of Culture And Conflict;
- Mt Adams (47) – As I Near the End of This Journey;
- Mt Jefferson (48) – A Journey Ends…