A Story Approach for monitoring change in an agricultural extension project

A paper presented at the Conference of the Association for Qualitative Research
Melbourne, July, 1999

Jessica Dart

Institute of Land and Food Resources
University of Melbourne
Parkville, Victoria 3052
Ph (03) 9344 8356 Fax (03) 9344 4665

Email: j.dart@pgrad.unimelb.edu.au

Abstract

In the field of program evaluation, recognition of problems associated with the use of quantitative performance indicators has set the stage for alternative or supplementary approaches. There have been strong murmurs of interest about the use of "performance stories" for monitoring social change programs. To date little research has been done in this area.

A Story Approach was implemented across a statewide dairy extension project in an attempt to overcome some of the difficulties associated with monitoring the project impact. This process was adapted from the ‘evolutionary approach to organisational learning’ (Davies 1998). The Story Approach is participatory, in that all the project stakeholders are involved in deciding the sorts of change to be recorded. Essentially the process involves the collection of stories of change, emanating from the field level, and the systematic selection of the most significant of these stories during regional and statewide committees meetings.

This approach goes beyond merely capturing and documenting client stories; each story is accompanied by the storyteller’s interpretation, and after review the stories are also accompanied by the reviewers’ interpretation. One of the ideas behind the process is that it promotes a slow but extensive dialog up and down the project hierarchy each month.

This paper describes the method of the Story Approach and highlights some experiences gained during the 12-month trial of the process with the Target 10 dairy extension project. It is argued that this approach can constitute an appropriate and credible process for monitoring change, can help to promote organisational learning, and can be a rewarding and enjoyable process for the participants.

Introduction

Between May 1998 and May 1999 the Target 10 Dairy Extension project implemented a novel approach to participatory monitoring and evaluation referred to as ‘the Story Approach’, which as far as we know, has never been attempted in Australia before. The purpose was twofold: to collect data about the impact of the project as a whole; and to promote organizational learning within the project team.

Background to the Target 10 project

The Target 10 dairy extension project aims to enhance the viability of the dairy industry through programs that profitably increase consumption of pasture by cows. Information from research on pasture utilisation is extended to farmers through courses, discussion groups, newsletters, target graphing, feed budgeting, comparative analysis, field days, focus farms and demonstrations and other media. In a concerted effort towards evaluation since its inception in 1992, the Target 10 dairy extension project had completed extensive benefit-cost analysis and individual programs have been evaluated against their objectives. However, in 1998 there was still a feeling that some of the project impact and outcomes were not being captured.

It was agreed to experiment with some unconventional forms of monitoring and evaluation, and one of these ‘experiments’ was to implement a ‘story-based’ approach to monitoring and evaluation. After introducing the story concept to key project stakeholders, an agreement was made that the approach would be implemented across the whole project for a period of one year. While many sceptical voices were heard at the start of this ‘experiment’, there is now growing enthusiasm for the approach and already several other extension projects across Australia are adopting modified versions of this approach.

The evolutionary or Story Approach to M&E

The ‘story ’, or ‘evolutionary’, approach was developed by Rick Davies in Bangladesh in 1994 (Davies 1996). It is a participatory approach, as all the levels (committees in this case) of the project are involved in deciding the sorts of change to be recorded. The process is illustrated in Figure 1. Unlike conventional approaches to monitoring, the Story Approach does not employ quantitative indicators, but is a qualitative approach. Conventional approaches to monitoring extension programs are heavily influenced by a planning ethos that places substantial emphasis on prediction and control. The Story Approach to monitoring, in many respects, is the opposite. Davies (1996) outlines seven main differences between these approaches, which are presented in text box below.

• Objective indicators versus subjective iterative process: A central feature of planning-based methods of monitoring is the use of ‘indicators’. Within planning-based approaches, it is believed that differences in the subjective perspectives of events, and the underlying value concerns of different observers, need to be controlled or ignored. Under the Story Approach, agreement on meaning of events is an outcome at the end of a process (a month’s cycle or more), never final in its form, and subject to revision in the light of new experience. The identification of differences in interpretation is central to the whole process; they are to be brought to the surface and explored, not ruled out.
• Inclusion versus selection: Planning-based monitoring systems are mostly quantitative in their content. Quantitative analysis is based on the ability to enumerate things or events. At the very basic level of counting, quantification is also about the homogenisation of experience. Within the daily experience of an organisation, those events that are countable are summarised by a process of inclusion. Within the Story Approach, experience is summarised by selection rather than by inclusion; it focuses on the exceptional rather than the common experience; and it seeks to differentiate rather than to homogenise. It is about defining the meaningful edges of experience rather than identifying a central tendency.
• Predicting outcomes versus open-ended outcomes: Under the planning-based approach to monitoring, events of concern are specified before their occurrence rather than afterwards. In conventional monitoring systems, ‘indicators’ are established at the beginning of a project, and data in the form of statistics are gathered repeatedly throughout the life of the project. The process is strongly deductive in orientation: it starts with a conception of the desired outcomes and works down from there. The opposite is an inductive approach, where indicative events are abstracted out of recent experience, and this process is renewed with each new reporting period of the monitoring system. Instead of being predictable, it is open-ended.
• Who defines the indicators? In most monitoring systems, events of concern are defined by people distant from the events that are to be monitored. Typically, senior staff in organisations carry out the identification of indicators. In the Story Approach, those closest to the experience being monitored (e.g., the field staff) are given the right to pose a range of competing interpretations of those events.
• Where is the data analysed? Normally, the analysis of events documented by an organisational monitoring system is carried out on a centralised basis, at senior levels of the organisation. Typically, field-level workers do not analyse the data they collect; rather they simply forward information up their hierarchies for others to analyse. In the Story Approach, information is not stored or processed on a centralised basis but is distributed throughout the organisation and processed locally. Staff not only collect information about events, but also make their own evaluation of that information, according to their own local perspective.
• Statistics versus ‘thick’ description: When conventional monitoring data is analysed, it is in a form and location that strips it of context. Typically, few text comments accompany statistics sent up from field workers. As in many qualitative approaches the Story Approach makes use of ‘thick description’, closely textured accounts of events, placed in their local context, and where the role and subjectivity of the observer is visible. In the world of ordinary people, these often take the form of stories or anecdotes. Within the Story Approach to monitoring outlined here, these ‘stories’ are accompanied by their reviewers’ interpretations.

• Static versus dynamic: Most monitoring systems are largely static structures. Indicators remain essentially the same for each reporting period, and the same questions are asked again and again. The focus remains the same. With the Story Approach, the contents of the monitoring system are potentially far more dynamic and adaptive. Events reported reflect both a changing world and changing sets of perceptions within the members of the organisation about what is important within their world.

• Paraphrased from Davies (1996)

Method of the Story Approach

There are three main parts to the approach (as practised in the Target 10 dairy extension project):

• Establish domains of change
• Set in place a process to collect and review stories of change over a 12-month period
• Monitor the process.

Stage One: Establishing domains of change

In the first stage of the process, the evaluation audience (Target 10 stakeholders) identified three ‘domains’ of changes that they thought needed to be monitored at the project level; for example, changes in profitability. These domains were established using the Delphi technique, which is a form of interactive (postal) surveying that utilises an iterative questionnaire and feedback and provides participants with an opportunity to revise earlier views based on the response of other participants, until some desired level of consensus is reached. Unlike ‘performance indicators’, these ‘domains’ of change are not precisely defined but are left deliberately fuzzy; and it was initially up to field staff to interpret what they felt was a change belonging to any one of these categories.

Stage Two: Collecting and reviewing the stories of change

The next stage involved the collection and review in each region, of stories demonstrating significant change (according to the nominated domains of change). The stories were told by those most directly involved (e.g., farmers, extension staff, and field workers). Each level of the Target 10 project hierarchy (i.e., regional committees, the statewide Central Executive Committee) was then involved in reviewing a series of stories and selecting those that they thought were the most significant accounts of change.

The various committees were required to document which stories they selected and what criteria they used. It was intended that the monitoring system should take the form of a slow but extensive dialogue up and down the project hierarchy each month. At the end of the trial period, a document was produced containing all the stories that had been selected by the Central Executive Committee over the period of the year. The stories were accompanied by the criteria that the Central Executive Committee used to select the stories.

Finally, a panel of "key influencers" and funders were asked to review this document and score the stories on the basis of the extent to which the stories represented the sorts of outcomes that they would want to purchase. They were also asked to document the criteria used to score the stories.

Stage Three: Monitor the process

In addition to the production of a document containing selected stories and readers’ interpretations, the story process itself was monitored and additional analysis carried out.

Figure 1 Main steps of the Story Approach

Fine-tuning the process

It would be misleading to suggest that the Story Approach was implemented smoothly and easily across the project. At various stages in the 12-month trial, problems arose and where possible these were addressed. However, as the process was an iterative one, it was possible to modify each ‘round’ on the basis of feedback provided from the previous ‘round’ of stories. Over the year, various changes were made to the process to address the informational needs of the Statewide Executive Committee (who were reviewing the stories on a statewide basis every 2-3 months).

An example of feedback concerning the content of the stories was that the Executive Committee strongly valued stories directly written by farmers. After this point was fed back to the regions, the number of stories collected directly from farmers (and reported in the first person) increased. In the same way, problems with the process were addressed. In one region voting became very competitive and it was found that story reviewers, at the regional level, were actually judging the stories more for who wrote them rather than for their content. This was brought to the attention of the relevant committee and the process was modified to avoid this outcome.

Constructivist epistemology and the Story Approach

Rick Davis developed the approach as part of his doctoral research that examined the use of evolutionary theory to aid organizational learning. In this case, I used a modified version of the evolutionary approach (referred to as the Story Approach), coupled with different theoretical focus. I considered the Story Approach under a constructivist epistemology that appears to be congruent with the evolutionary perspective as described by Davies (1996), (although he does not explicitly refer to constructivism in his research). Constructivism claims that meaning is constructed by human beings as they engage with the world they are interpreting.

In the Story Approach, project stakeholders interpret their experiences with the project and select instances of significant change to record as a story. They are also required to record why this change is significant to them. Thus when a farmer tells a story of significant change, she/he interact with the world and draw meaning from it, and it is in the telling of the story that meaning is constructed. Then when reviewers and read and evaluate the story, they engage with it and construct a further new meaning. When this is done in a group, this construction may be shared. In the Story Approach the criteria that are used to interpret the story are documented, made transparent and attached to the story itself. It is this transparency that makes the whole process even more open to new and more sophisticated constructions of meaning.

In the Model of Fourth Generation Evaluation, Guba and Lincoln (1989) employ a constructivist approach to evaluation, contending that realities are social constructs of the mind, and that there is no one objective reality. The key emphasis of their model is on the process of negotiation, incorporating various stakeholders more centrally into the evaluation process. This description could equally be applied to the Story Approach, although there are substantial differences in method. However, a critical distinction between the Story Approach and Fourth generation is the method of collecting and selecting stories of significant change.

Stories are particularly promising as a medium for helping stakeholders to make sense of impact for several reasons. In organisational learning literature, stories are valued and referred to as the preferred sense-making currency (Boje 1991). In complex organizations, part of the reason for storytelling is the working out of those differences in the interface of individual and collective memory (Boje 1991). Thus, if stories are to be considered an indigenous sensemaking systems in organisations, this medium would seem to be an ideal one to also collectively make sense of impact. Just as staff use stories to make sense of surprises (such as a story about someone being fired in a business firm), it is suggested that the natural storytelling process can be harnessed to help practitioners and farmer-clients to make sense of impact and outcomes in agricultural extension programs, through participatory evaluation processes.

Findings

Describing the ‘results’ of this process is a difficult task. The first problem is that there is never a ‘final’ outcome, as the aims of the process are to:

• Move towards a better understanding between all the various project stakeholders as to what is occurring for the individual farmer clients.
• To explore and share the various values and preferences of the project stakeholders.
• To gain a clearer understanding (as a group) of what it is and is not being achieved by the project and to clarify what they are really trying to achieve, so that the project can move towards what is desirable and move away from what is undesirable.

Secondly, unlike conventional evaluation approaches that tend to reduce the complexity of the client experience into numbers and averages, the Story Approach attempts to keep an element of the ‘rich picture’. Therefore, it would go against the ethos of the approach to dissect the stories and summarize them in the name of the ‘final results’. The ‘final results’ of this process are really the feelings and the judgements that are made when reading the stories and deciding whether they represent the sorts of outcomes that the reader finds merit-worthy for a project such as this.

Thus the audience of the evaluation were encouraged to read the story booklet that was produced at the end of the year’s trial and to evaluate the stories. The booklet contained 24 stories selected by the Central Executive Committee of the project over the twelve-month period. The stories were accompanied by comments explaining the reason for their selection. In the booklet I encouraged the readers to engage with the stories and add their own comments in the spaces provided. For this reason, the stories in the booklet were deliberately not ‘analysed’ by an external expert, but were left in their raw form for the reader to interpret and draw their own conclusions. In a sense then, this document aimed to be interactive.

Nevertheless, for the purposes of accountability and transparency, the process was monitored and the stories were examined for overall trends in content and origin.

What this process revealed about the impact of the project

In total 134 stories were collected as part of this trial. The stories concerned significant changes brought about by project activities. During this period the Statewide Central Executive Committee of the Target 10 Dairy Extension selected 24 of these stories as the most significant accounts of change. The stories originated from all of the four regions of Victoria where the Project operates. These stories were written by staff from the Department of Natural Resources and Environment (NRE), farmers, industry representatives and educators.

En masse the stories present a picture of many farmers implementing part or all of the Target 10 message, and of farmers gaining from the programs in unexpected ways. The most frequent theme amongst all stories (including the selected stories) concerned farmers who had changed to recommended practices and subsequently gained an increase in production. The second most common theme concerned farmers who had adopted recommended practices and as a result experienced short-term financial savings. These findings are consistent with the aims of the project and support the other evaluation findings, which together suggest that the Target 10 recommended practices do have an impact on increased production and short-term profitability.

However, in addition to stories concerning production and profit outcomes, many stories concerning other types of change were collected and interpreted. An example of other themes running through these stories were that farmers, after attending Target 10 programs, felt more in control of their business and empowered to challenge their consultant. Some of the themes to come out of the stories were largely unexpected outcomes, which would be very difficult to monitor using an indicator-driven process. While it is rhetorically impossible to measure long-term impact in the short-term, it is nevertheless possible to capture and share stories about farmers who have experienced intermediate outcomes, that they believe may eventually lead to long-term impact.

About 10% of all stories collected concerned some element of ‘bad news’. There was no formal system for implementing changes in the project as a result of these stories. Feedback from the Central Executive Committee suggested it was extremely beneficial to read and discuss bad news stories. It is planned that in the second round of the project, a system of encouraging ‘bad-news stories’ will be implemented and incorporated into the continual improvement process of the project. However, all the ‘bad-news stories' were read widely and it is possible that these lessons have been stored in the organisational memory of the project, to inform future action.

Interpretation of results in relation to the project environment

The organisational structure under which the Target 10 project operates is complex. The project was developed with considerable collaboration of industry, university and other NRE providers and opperates under the new environment of the purchaser-provider model. As a result, the project has several purchasers and co-providers interacting in a dynamic project environment (Mcdonald and Kefford 1998). Taking this organisational complexity into account, it is vitally important that time is allocated for the various stakeholders to enter into a meaningful dialogue about what is happening in the field, and whether these experiences represent the sort of outcomes that are desirable. It is also important that projects under this new organisational structure are able to demonstrate that they have the capacity for reflective practice, organisational learning and the ability to capture and interpret evidence of changes that they are trying to achieve.

During the story review process it became apparent that different stakeholders interpreted the stories in differing ways and held different things to be of value. This was especially true of the purchaser group. This finding supports the concept that modern evaluation is conducted in a value-pluralistic context (Toulemonde et., al 1998). Thus, negotiation and dialogue between the various evaluation stakeholders (including the purchaser and the provider) is essential.

Impact of the process on organisational learning

This ‘experiment’ has been viewed as a positive learning experience by those who participated most actively in the process (the members of the regional and central executive committee) . It is planned to continue the process (in a modified form) into the next phase of the project. Also, it is likely that the method will be adopted by other projects within NRE.

The process of collecting and analysing stories has seen farmers, collaborators and NRE staff sitting together at committee meetings discussing and interpreting qualitative data, casting evaluative judgements and negotiating about what constitutes a significant change. Feedback from the project committees suggests that learning has also occurred in terms of increased skill in conceptualising and capturing impact; over the year the storytellers became better at capturing impact and responding to the suggestions that were provided in the feedback from the story review process.

Conclusion

The Performance Story Approach appears to meet the project management’s need for ‘thick- description’ (Geertz 1973) about the variety of farmer experiences. Without underrating the power of the Story Approach to produce data which contributed to describing the impact of Target 10 project, it is suggested that the most significant impact lies in the intangible area of organisational learning. There have been noticeable improvements in terms of gaining a richer and more shared understanding of what has been achieved as a project and what is valued as a positive outcome by the project. Further research is currently being conducted on the impact of the Story approach on the project.

References

Boje, David- M. (1991). The Storytelling Organization: A Study of Story Performance in an Office-Supply Firm. Administrative Science Quarterly 36 (1), 106-126.

Davies, R. J. (1996). An evolutionary approach to facilitating organisational learning: An experiment by the Christian Commission for Development in Bangladesh. A WWW publication at http://www.swan.ac.uk/cds/rd/ccdb.htm. Centre for Development Studies: Swansea.

Geertz, C. (1973) Thick Description: Toward an Interpretive Theory of Culture, In Interpretation of Cultures (pp3-30). Basic Books: New York.

Owen, J. M. (1993) Program Evaluation, Forms and Approaches, Allen and Unwin, St Leonards, NSW, Australia.

Shaw, G., R. Brown, and P. Bromiley. (1998) Strategic Stories, Harvard Business Review, 1998, 41-50.

Winston, J.A. (1991). "Linking Evaluation and Performance Management", review paper. Paper read at National Evaluation Conference, at Adelaide.