Ensuring rigour from a multi-site, multi-discipline,

multi-researcher study

M.L. Leybourne1, H.K. Crawford1, A. Arnott2 & R. Benson2

1 Agriculture Western Australia; 2 Tropical Savannas Cooperative Research Centre

Abstract

Qualitative research has often been criticised for its lack of rigour. In order to overcome this, measures of trustworthiness, dependability and reliability have been suggested. A study of how pastoralists learn to incorporate sustainable farming systems in the tropical savannas of Australia employed multiple-researchers, working in three States and from a variety of disciplines. To ensure rigour a framework for the study was developed by the researchers prior to commencing interviews. This was followed by regular teleconferences to ensure that the framework was valid and to adjust for any problems encountered along the way. Every interview was analysed independently by all researchers before a workshop was conducted to bring the ideas together. Categories and ideas within the data were synthesised to create an overall understanding of the learning process within the confines of "landcare" in the Tropical Savannas. These processes were undertaken in consultation with the pastoralists and the process has been explicitly documented to enable readers to follow the research process easily. The rigour in this project is shown in the clear documentation of the research process carried out by individual researchers and by the team when it met. The understanding of pastoralists’ learning processes is our interpretation; it is up to the reader to decide whether s/he agrees with that interpretation, but from the description of the process it is easy for the reader to see where and why her/his interpretation differs.

Introduction

The purpose of the research project was to identify, describe and share knowledge and understanding of the learning processes of pastoralists in the tropical savannas of Australia through a largely phenomenological approach. The intention was to assist the Tropical Savannas Cooperative Research Centre and other education and service agencies to improve their communication and extension work with pastoralists. It was also hoped that pastoralists would learn from experiences of their peers and perhaps help them to manage change more easily themselves.

In-depth interviews aimed to explore how a group of pastoralists or ‘pastoral families’ was stimulated to learn, how they went about learning and how they gained and developed knowledge. It concentrated in particular on learning processes concerned with making changes to their management practices, and these ranged from changing a mustering system, or modifying a fire regime, to introducing cell grazing or diversifying their sources of income.

It was evident at the start of the project that a process would be required to ensure rigour in the research, particularly as the intention was to use different field researchers, in each of the three States (Western Australia, Northern Territory and Queensland), who all came from different disciplines and backgrounds. The research framework for the study was developed at the same time as the study’s objectives, in consultation with the researchers.

Many authors have debated the issue of rigour in qualitative research. This paper shows how a multi-site, multi-disciplinary, multi-researcher study can maintain rigour, particularly as defined by Lincoln & Guba (1989). In addition, we suggest that the rigour of our study is strengthened by the explicit reporting of how we achieved what we did and that this is a crucial element maintaining rigour within qualitative research irrespective of the methodology you might be using.

Our research process

The broad objectives of the research project were developed at the Tropical Savannas CRC and field researchers were selected for each of the three States. The whole team participated in a three-day workshop in Darwin when the framework, style and research method, including the "checklist" of questions to be used by each researcher, was determined and the project proposal further developed.

The researchers in the project all had either experience with pastoralists or experience in education research, and the researchers often knew the pastoralists selected. This helped to develop the project as something less than top-down and helped remove any barriers of mistrust to allow for more in-depth and effective information.

The field researchers conducted a trial interview which was followed by a teleconference with the whole team to discuss strengths and weaknesses in the checklist of questions and any differences in the application of the research method used. This allowed field researchers to more closely use the same techniques during the research process to follow.

Prior to the formal interview, each pastoralist or family, was contacted by the field researchers to engage their interest in the project. A total of 18 semi-structured in-depth interviews were then held with pastoralists or pastoral families in the Western Australia, Northern Territory and Queensland. At each interview, specific learning episodes were identified and the discussion revolved around these in order to study them in depth, rather than briefly touch on a series of separate episodes. The episode was not identified beforehand; it depended on what issues were raised by the pastoralist him or herself during the first part of the interview.

Another teleconference was held part way through the interviewing period, which allowed an opportunity for the researchers to bring up any issues or problems they were having. In the event, all the interviews proceeded smoothly.

Following the interview, and before they were analysed, the transcript was sent to each family for comment. Contact was maintained with the pastoralists during, and after, the data analysis part of the research for their continued input into the project outcomes.

Once pastoralists had agreed that the transcript was an accurate record of each conversation, a full set of the transcripts were sent to each member of the research team for analysis. Several teleconferences were held during this time to discuss issues the researchers felt were important in the transcripts and to ensure that the data was being analysed in a complementary, if not similar, fashion by each person. All the teleconferences were recorded and transcribed.

A second workshop was then held for the whole research team to bring the individual analyses together. During plenary sessions in this workshop, the research team identified three major themes: perceptions, informal learning and sources of information. Each of these three themes was further subdivided into sub-themes. For example, informal learning was subdivided into observation, experience, trial and error, talking/listening/discussion, questioning, reflection, transfer of knowledge/principles, networking/mentoring, transposing positions, information exposure in terms of positions the pastoralists may have held, groups, opportunistic/timing and triggers.

In break-out groups, researchers working in pairs highlighted data, from each interview, that related to each of the sub-themes. In addition, the discussion that took place during the workshop was recorded and transcribed in order to capture other issues raised during the workshop.

To enhance the phenomenological nature of the research a number of stories were also identified that could be developed from the transcripts and would highlight the learning processes as described by the pastoralists or pastoral families. The researchers began developing these in the weeks following the workshop. This assisted individual researchers look individually at each theme and ensure that all issues raised in the interviews were correctly covered by the workshop. These were further discussed in another teleconference.

Finally four members of the research team spent four days together to start writing the research report. During this stage the data collected for each theme was further consolidated to help focus the writing team members on the issues and develop cohesive research outcomes.

Discussion, and how we ensured rigour

The project used a phenomenological approach, which is a focus on understanding the meaning events have for persons being studied (Patton, 1991). Broadly this is, by nature, qualitative research within the constructivist inquiry paradigm, as it looks to understanding a situation as the participants construct it. It sets out to capture what people say and do and how they interpret the world. As researchers, we need to be able to capture this process of interpretation through an empathic understanding or the ability to reproduce the feeling, motives and thoughts behind the actions of others, while at the same time remaining as objective as possible (Bogden & Taylor, 1975; Maykut & Morehouse, 1994).

Crotty (1996) suggests that all qualitative research is unique. This we acknowledge. Whilst the underlying principles of our phenomenological approach did not change, we are preared to accept that we may not have remained entirely true to the hermeneutic priciples of phenemenological research throughtout the project. The research therefore might be considered by some to be more accrately labels natualistic rather than phonomenological.

It has been argued that the conventional criteria for judging the rigour or trustworthiness of qualitative research include internal validity, external validity, reliability and objectivity are not always appropriate (Guba & Lincoln, 1989; Lincoln & Guba, 1985). These have been described as follows:

· Internal validity: the extent to which variations in an outcome or dependent variable can be attributed to controlled variation in an independent variable

· External validity: inference that the presumed causal relationship can be generalized t and across alternate measures of cause and effect and across different types of persons, settings and times

· Reliability: consistency of a given inquiry - is generally a precondition for validity. It refers to a study’s consistency, predictability, dependability, stability and/or accuracy. Reliability typically rests on replication.

· Objectivity: neutrality, a demonstration that the inquiry is free of bias, values and/or prejudice.

Guba and Lincoln (1989) offer the following criteria as alternatives:

· Credibility: parallel to internal validity. The focus is more on establishing the match between the constructed realities of respondents and those realities as represented by the evaluator and attributed to various stakeholders. Credibility can be "verified" by 1) prolonged engagement; 2) persistent observation; 3) peer debriefing; 4) negative case analysis; 5) progressive subjectivity; 6) member checks.

· Transferability: Parallel to external validity or generalisability. Transferability is relative and depends entirely on the degree to which salient conditions overlap or match. This is mostly verified through "thick" description. The constructivist does not provide the confidence limits of the study, but instead provides as complete a data base as possible in order to facilitate transferability judgements on the part of others.

· Dependability: parallel to reliability. Concerned with the stability of the data over time. Need to be able to demonstrate any changes or shifts in the way in which the inquiry was conducted.

· Confirmability: parallel to objectivity. Need to show that data, interpretations and outcomes inquires are rooted in contexts and persons apart from the evaluator and are not simply figments of the evaluator’s imagination. All data needs to be able to be tracked to its source and that the logic used to assemble the interpretations into structurally coherent and corroborating wholes is both explicit and implicit in the narrative of the case study.

We have used these to assess the rigour of our research.

1. Credibility

The outcomes of our research can be considered credible through a number of mechanisms. The researchers, in the main, knew the pastoralists that participated in the study which allowed for a measure of trust between the researchers and those being researched. The research team included four "field" researchers that undertook the actual research and a further two full time researchers with expertise in adult education and learning processes. This provided an opportunity for "disinterested" peer reviews of the interviews across all three States and extensive discussion and analysis with them throughout the research project. A further four researchers had peripheral involvement in the study and provided feedback to the research team at various stages of the project. An additional aid to ensuring rigour through credibility was to include the pastoralists in later stages of the study, by sending the transcripts of the interviews for verification and then providing them with drafts of the research outcomes. A final stage will be for the field researchers to make presentations of the research findings to various pastoral groups.

2. Transferability

This measure of rigour has been enacted through making this a multi-site project. One of the intentions at the beginning of the study was to undertake an analysis of differences between the three States, but the results have shown very minimal differences. Most of these differences came from the sources of information that would be expected. However, even these results did not differ much; for example, pastoralists from all three States stated they used sources of information from Queensland (although Queensland pastoralists did not go outside their State to seek further information).

3. Dependability

This measure was ensured through maintaining constant contact between members of the research team at all stages of the study, with three workshops (although the third one consisted of only four of the core team of six) and around 15 teleconferences as well as contact through e-mail (sending of documents). The teleconferences, in particular, were used to ensure that the research was being conducted in a similar manner across the three States, and then that the data was being consistently analysed. It was also important for all the researchers to analyse all the data, not just that they had collected themselves.

4. Confirmability

It was difficult at times for the researchers to remain fully objective. This was seen in some of the interview transcripts when the pastoralists and researchers are well acquainted. Although it is accepted that interviews become conversations, in some instances researcher coments could be construed as leading, offering opinions or explanations that the pastoralists hapily took on as their own. However, these instances were culled through having a research team of six. The discrepancies were recognised and removed. As much of data analysis also took place as a workshop, any individual biases were pushed aside. In fact, the fact that this was a multi-site, multi-discipline and multi-researcher study ensured rigour in this context.

Conclusions

Measuring our research against Guba and Lincoln’s 1989 criteria suggest that we have been able to maintain rigour within our research despite the difficulty of the different sites, researchers and backgrounds. Continual communication, a simple adherence to the principles of the phenomenological research methodology, and listening to our fellow farmer researchers has strengthened this. However, we would also like to postulate that by the straightforward description of what we have undertaken also offers and guarantees rigour for our research. In short we have described to you our research process and through that description allow the research to be repeated. That is all we can do as the final understanding, whether agreed to be our co-researchers or not, of the learning process is ours. It is up to you the reader to agree with the interpretation and when reading the full report to decide if that is as you see it or not. Our rigour is maintained in the full description of what and how we did the research.

References

Arnott, Allan, Rebecca Benson, Ken Crawford, Stephanie Herbert, Marnie Leybourne, Greg Shaw and Robert Speirs (1999). "Learning processes and the impact of change on pastoralists in the tropical savannas of Northern Australia", IV International Rangeland Congress, Townsville.

Bogdan, R. and S.Taylor (1975). Introduction to qualitative research methods: a phenomenological approach to the social sciences, Wiley, New York.

Crotty, M. 1996. Phenomenology and nursing research, Churchill Livingstone, Melbourne

Guba, E.G. and Y.S. Lincoln (1989). Fourth generation evaluation, Sage Publications, California.

Lincoln, Y.S. and E.G. Guba (1985). Naturalistic inquiry, Sage Publications, California.

Maykut, P. and R. Morehouse (1994). Beginning qualitative research: a philosophic and practical guide, Falmer Press, London.

Patton, M.J. (1991). "Qualitative research on collage students: Philosophical and methodological comparisons with the quantitative approach" Journal of College Student Development, Vol.32, pp.389-96.