Some Principles
for
'Quality Control'
in Qualitative Research
a Phenomenographic
Treatise

by

Dr Robert M. Herschell

Dean, School of Education
Christian Heritage College
Mansfield, Brisbane
Queensland

9th July 1999

 

Abstract:

Researchers using quantitative approaches from positivist paradigms have argued the need for principles and strategies for quality control in their studies. Highly defined theoretical concepts such as replication, variance, validation, reliability and discrimination, together with tests, such as those for sampling, correlation and significance, have been developed as a direct result of quantitative researchers’ concerns for quality control .

• What principles and strategies are available to and appropriate for qualitative researchers?
• How can these be defended?
• How and when ought these be applied?
• Is there a case for a dismissal of the need for quality control in qualitative studies?

In this article, it is argued that researchers ought to employ a range of quality control measures when undertaking qualitative studies. Researchers need to ensure that data collection, organisation, and analyses, as well as the processes and outcomes of such studies, are trustworthy and believable.

A new term Process Believability is described, defined and defended as an appropriate technical term to encompass the meanings that are needed as researchers of qualitative phenomena address issues associated with quality control.

Researchers are invited to critique a set of principles that are also proposed to guide the use and implementation of Process Believability.

Robert is Dean, School of Education, at Christian Heritage College, Mansfield, in Queensland. He co-ordinates four preservice undergraduate and four inservice / post-graduate education programs. He has recently completed a doctoral study on ‘Conception of Thinking Legally: an interpretative approach’ through the University of New England’s Ph.D. research program. He can be contacted by:

(Phone): (07) 3343 8011 (work) or 0408-789-616 (mobile)

(Snail Mail): PO Box 2246, Mansfield DC, Brisbane, QLD, 4122

(E-mail): r.herschell@uq.net.au or chc@chc.qld.edu.au

The Issue:

The following extracts were taken recently from an International Qualitative Researchers’ Email ‘list-serv’:

Respondent A:

I have just been reading along…. I smiled when I read what Robert wrote: "I have been concerned that researchers using quantitative approaches have developed protocols for quality control of issues such as sampling, validity, reliability, discrimination and so on. Here are my questions: (i) How have researchers, or how might they, develop 'process' approaches to quality control in qualitative research? (ii) Does anyone have any definitive advice as to how this question might be answered?"

At this stage I was tempted to respond that I could not appreciate the question because I failed to understand why and how 'quality control' of these issues could be a legitimate concern. But then I figured it was a joke. Then I was not sure. I was about to plan some serious reply when I read what (name) wrote -then figured that this MUST be a kind of leg-pull? You have gotta be kidding!!!!! Right???

I note that the Market Research Society of Australia (M.R.S.A.) has a quality accreditation scheme which addresses quality issues relating to the recruitment of respondents for qualitative research. Organisations that recruit respondents for qualitative research are audited annually and given accreditation if they adhere to the recommended processes. If you contact the M.R.S.A. you will get details of the specified requirements.

But then I know it's summer in the southern hemisphere and who knows what madness that may induce (now I'm kidding!) BUT if this is a serious topic then please answer the following question: What are the outcomes of phenomenographic -research and how could 'quality control' affect the legitimacy of these outcomes?

…

Respondent B:

Sorry (Respondant A), not a leg-pull!

I work full-time as a qualitative researcher and often need to answer challenges to:

(a) data collection methods

(b) data analysis procedures

(c) validity of interpretation of data

Quality control procedures arise out of facing these issues on a regular basis.

I agree with (name) that in a perfect world this might not be necessary, but in this real world it is not a bad idea to record the preferred standards for data collection and interpretation, whether in physics, education or social research.

It REALLY is not good enough to say, "It means that because I, the researcher, says it does."

Some Methodological Perspectives:

The previous statements by Respondents A and B represent two very polarised perspectives on the need for principles of quality control with which qualitative researchers need to grapple. Some researchers have argued that to entertain notions of quality control is to impose quantitative principles on qualitative studies. Others search for ways to ensure that the processes and outcomes of qualitative work trustworthy and believable. Svensson described some of the history of these developments as:

The explorative and interpretative character of the (phenomenographic) data collection and analysis meant a radical shift from the quantitative methodological tradition. This shift was inspired by hermeneutic, ethnographic and phenomenological traditions within a general concern for paradigms and methodological traditions. The reporting of the phenomenographic research results also met questions concerning the character of the method and the rationale behind the research. These questions were raised first mainly from representatives from the dominant traditions of positivist and quantitative research, but later on also from representatives of ethnographic, hermeneutic and phenomenological research traditions. From one side the analytic characteristics of explicating results in the form of categories and relations was appreciated but the explorative and interpretative methods of arriving at the results were questioned. From the other side the explorative and interpretative character of data collection and analysis was accepted but the analytic character was questioned. (Svensson, 1994, 13)

While ontological and epistemological assumptions need to be declared by qualitative researchers when undertaking their studies, there are additional sets of assumptions that relate to each methodological approach being used in particular studies. Key areas to which sets of assumptions relate include: (i) the exploratory nature of the data collection processes and (ii) the contextual character of the treatment and (iii) analysis of the data (Svensson, 1994, 18). I believe these need to be complemented by (iv) the conclusion - drawing processes, and (v) the application and usefulness of the research findings.

The nature of the data collection processes relates directly to the type of the qualitative investigations being undertaken and the usual desire to maintain openness rather than closure in the data collection, analysis and conclusion-drawing phases of qualitative research. Many qualitative studies often view data collection as an exploratory activity. Since phenomenography, as an example of a qualitative research approach or orientation, uses interpretative approaches, a degree of uncertainty exists typically in the mind of the researcher about the meanings being generated through the discourses that are undertaken with each participant.

Since the character of qualitative inquiry is inductive rather than deductive in nature, it is expected that some variation in the use of the research approach will occur. While this may be of some concern to some, the qualitative researcher is confident that delaying premature judgments is actually in the best interests and traditions of the research.

Similarly, as the researcher and participants explore both the breadth and the depth of the phenomenon under study, they must move towards the participant’s external horizon of knowledge where conclusions tend to be both more tentative and abstract. External horizon is a concept that describes the boundaries, extremities or limits of an individual’s knowledge and experience bases. In other words, at one’s external horizon, meanings are made rather than found. This applies not just to the constituent parts of the phenomenon but also to what Svensson (1994, 18) refers to as the ‘wholes-qualities of the entities’.

In the data analysis processes, similar openness needs to be maintained as the researcher seeks the commonalities of meaning that are induced from the collected data obtained from each individual as well as from the whole cohort of research participants. The inductive process of identifying and interrelating meaning units in the data is both a cognitive and a meta-cognitive task. There seems to have been an emphasis on rejecting the tendency to make premature judgments about the content, concepts and conceptions included in the data. In a sense, an holistic understanding of each participant’s contributions as well as the meanings gained from all the participants’ research data must be developed prior to any conclusions being made. This, in one sense, might be construed as a clumsy and elongated process. But this is an essential process of research with qualitative data, especially when the data are highly conceptual and abstract.

In the conclusion-drawing processes (especially those associated with, for example, phenomenographic outcomes such as meaning units, categories of description, conceptions and outcome space, especially in their structural and referential dimensions), unique combinations of individual and collective lived world experiences are dependent on the researcher, the participants and the study topic. For example, the research data used to form a conclusion are usually derived from various participants who have each experienced a different aspect of their lived world. No attempt is made either to standardise or to compare these experiences. It is the participants’ perceptions and resulting understandings rather, than the experiences themselves, which are the object of study.

When, for example, the phenomenographic researcher is attempting to integrate various identified aspects of each meaning unit, the integration of these occurs in the mind of the researcher. The key ideas and outcomes developed by a study, and in phenomenographic terms: the expression of their structural and referential relationships in the form of outcome space, are being constructed in the researcher’s mind. Not every aspect of each participant’s conversational discourse contributes to these processes. Indeed, through processes such as phenomenographic interpretation, induction and reduction, unrelated aspects of the data are discarded while related aspects are coalesced as the meaning units arise from the data. This could lead to a situation where different researchers might arrive at very different interpretations of the same data.

It is the coherence and rigour of the data collection principles and procedures, together with strategies for ensuring what in this article is referred to as process believability (Herschell, 1997, 102), that provide the qualitative researcher with confidence in the data collection processes and trustworthiness of interpretations (Glesne and Peshkin, 1992, 147-148).

Another key ingredient in this process of reduction, abstraction and conceptualisation is that of contextual analysis (Ferris, 1996, 108-111). The analyses of context, together with Svensson’s conclusion (1985, 1) that ‘since 1970 phenomenographic researchers have taken descriptions of people’s conceptions or understandings of certain messages, phenomena or problems as their research focus’, requires that the description of shared meaning of the phenomenon is central to a research study. The focus is, therefore, set within the participants’ experiences of their lived world so as to identify their ‘common sense conceptions in which they explain the physical and social world’ (Tesch, 1990, 49).

A contextual analysis, then, must not only mean an aggregation of specific data with generally given interpretations, but a delimitation of specific data related to each other as aspects of the same phenomena. (Svensson, 1985, 6)

Discussion:

Quantitative researchers, such as Leedy (1974), Fetterman (1988), Burns (1990), and Best and Khan (1993), use terms such as replication, verification and validation as a basis for their strategies to attest to the accuracy, truthfulness, integrity and reliability of the researcher’s data and analytical processes being used in a particular study. These terms, together with their related definitions and meanings, however, are of little relevance to qualitative researchers because of the epistemological, axiological, ontological and paradigmatic differences between quantitative and qualitative approaches to research studies.

In order to establish processes for confirming the authenticity, reliability and dependability of their data and analytical approaches, a recent Doctoral study (Herschell, 1997, 102) proposed that the term of process believability be used and defined for these purposes.

Sandberg (1997, 1-19) raised the question: ‘Are Phenomenographic results reliable?’ in challenging the usefulness of inter-judge reliability or agreement in phenomenographic research because of procedural, theoretical and methodological inconsistencies associated with objectivist epistemologies. He posed, as a solution to this dilemma, the use of reliability as interpretative awareness (Sandberg, 1997, 1). He proposed the epistemology of intentionality that underpins the phenomenographic approach. In this, one acknowledges and explicitly deals with the personal subjectivities in the research processes rather than ignoring them. This concept is related to Kvale’s (1991) discussion of the contrasts between biased subjectivity and perspectival subjectivity. However, there appear to be limitations in Sandberg’s approach to this question in that he seems to have either rejected, or forgotten to acknowledge the role of, reliable protocols and processes in phenomenographic research. In Sandberg’s work, little attention is paid to data acquisition and organisation strategies or analytical processes and tactics that are essential if process believability is to be legitimised.

Marshall and Rossman (1989) suggested that qualitative researchers should respond to the traditional social science concern for replicability by:

(a) asserting that qualitative studies by their very nature cannot be replicated because of real world changes;

(b) keeping thorough notes and a researcher’s diary which records not only each research design decision but also the rationale behind it, thus allowing others to inspect their procedures, protocols and decisions; and

(c) keeping all collected data in a well-designed retrievable form, making them available readily, either if findings are challenged, or if another researcher wants to reanalyse the data. (Marshall and Rossman, 1989, 148)

In suggesting a means of rectifying this inconsistency identified in Sandberg’s work, the notion of process believability is understood to be the design and implementation of principles and procedures to ensure that the qualitative data, analytical tactics and outcomes are authentic, believable, trustworthy and reliable. This notion is also closely related to Kvale’s (1991) perspectival subjectivity, although this too is somewhat distant from an inclusive perspective of process reliability .

Consequently, process believability is considered to be a more inclusive concept than those used by Sandberg or Kvale, as it integrates their concepts of reliability as interpretative awareness (Sandberg, 1997) and perspectival subjectivity (Kvale, 1991) in accordance with the principles proposed by Marshall and Rossman (1989).

To illustrate this notion of process believability, the author (Herschell, 1997, 100-101) designed and implemented the following strategy when undertaking the analysis of the qualitative database generated in his Doctoral research study:

1. a series of pilot studies attested to the suitability and usability of the study’s research questions and the interview protocols to be used in the data collection conversations with participants;
2.  
3. the oral interrogation of the research conversations during its recording;
4.  
5. in the preparation of the transcripts, the recording of each conversation was replayed many times to ensure that any idiosyncrasies of both the context (such as laughter, pauses, and phone interruptions) and the text were understood and meaningful in the context of the transcripts were acknowledged;

(d) the text of each transcript was prepared in draft form;

(e) the recorded conversation was replayed, the draft transcript text was checked for authenticity and accuracy;

(f) following the return of the edited and authorised version of the draft transcript from each participant, the final version of each transcript was prepared and printed;

(g) once all the transcripts were finalised, these were read numerous times and, on each occasion, notations were made of the potential ideas that were, through cognitive and meta-cognitive analysis processes, becoming foci for both the developing meaning units and, subsequently, the conceptions. The researcher attempted to keep his personal theories and prejudices in check (e.g. [bracketing] ) as he attempted to interpret the participants’ conceptions of the topic being investigated;

(h) these developing conceptions were drafted and discussed with two independent observers (i.e. triangulation) in relation to their reading of the transcripts;

1. 1. the segments of the each transcript of conversation were then coded on the basis of their ability to represent a key idea contained in the transcripts;

10. 1. a textual statement of each of the seven identified conceptions was drafted and edited several times;

(k) the formal statements of the conceptions were then finalised, edited and printed;

(l) the transcripts were read again in the light of the codings (intentional interpretations) in order to check their reliability in relation to the final textual statement of each conception;

(m) on the basis of this coding, each of the thirty transcripts, in computer file form, was reformatted so that all statements coded as related to Conception A were copied into a separate computer file. Similar processes were used for Conceptions B-G. This produced a reduced set of collations in which statements from all thirty participants related to Conceptions A - G were in seven separate computer files;

(n) each of these files was then analysed using an iterative and reduction processes, similar to that outlined above, in order to produce sub-categories of meaning for each conception. These sub-categories were used as the basis for sequencing the discussion of each conception; and

(o) once each sub-category had been decided, the transcript excerpts were once again analysed in order to ascertain the key idea being developed. On this criterion, the excerpts in each selection were sequenced and used as a basis for writing the description and discussion of each conception.

Marshall and Rossman (1989) supported the notion that qualitative researchers should answer concerns that the natural subjectivity of the researcher will shape the research. Ferris (1996, 113) supported the Marshall and Rossman assertion about the strengths of qualitative studies as well as gaining some understanding, even sympathy, for the study participants so as to gain entry into their experiential world and to the meaning they have made about the phenomenon being investigated. However, qualitative researchers, in particular, must provide contemporaneous controls for bias within the interpretations by ensuring that:

(a) a partner plays ‘devil’s advocate’ and questions the researcher’s analysis critically;

(b) there is checking and rechecking of the data;

(c) value-free note taking is practised; and

(d) the guidance of previous researchers is followed in order to control the data quality. (Marshall and Rossman, 1989, 147)

Gerber (1993) asserted that:

… the analysis of the data is said to maintain this same sense of truthfulness if it obeys a number of hermeneutic rules, namely:

(a) The analysis is oriented toward the phenomenon;

(b) The phenomenon must be described as it appears to the interviewees. The interviewees’ own understandings ... must be obtained during the interviews, without the researcher seeking explanations of these understandings;

(c) All aspects of the responses must be given equal importance, i.e. horizontalising the data. The researcher has to elicit a pool of understandings that come from the stakeholders in their own terms, without giving priority to any one response over another...

(d) The data should be then checked for structural features, i.e. for either different conceptions or meanings of a phenomenon.

(Ferris, 1996, 114)

The study had also relied upon others for both formative feedback and critical comment during each stage of the study. This has been achieved through conducting seminars and presenting conference papers (Herschell, 1992 & 1995) during the progress of the study as well as obtaining feedback from experienced researchers. The Marshall and Rossman (1989) recommendations for the control of bias in the interpretation of the research data were implemented. In order to ensure further the believability of the analysis of the data, the hermeneutic rules described above by Gerber (1993) were applied to the analysis process.

The study was also able to articulate a phenomenographic approach to the analysis of the research data that had been obtained and presented in verbatim transcripts of interview form. This analysis approach is outlined below in Figure 1.

Figure 1: An Iterative Approach to Phenomenographic Analysis:

Consequently, one outcome of my Doctoral research study (Herschell, 1997, 102) emphasised that:

Process believability means to: (i) investigate, (ii) check, (iii) question, (iv) embed one’s self in the study data, (v) mine the data, (vi) categorise, (vii) generalise, (viii) articulate, and (ix) communicate the research presuppositions and assumptions, data collection processes, analytical processes and strategies as well as the study outcomes.

Conclusion:

Qualitative researchers should not be afraid of comments, questions and criticisms from other research traditions and paradigms. By identifying, discussing and debating the quality control procedures being addressed, clarity will be developed in relation to the appropriate measures that should be considered in qualitative studies.

Based on the previous discussion, the following proposed set of principles for qualitative researchers is offered as a starting set for elaboration, debate and clarification by qualitative researchers who are interested in notions of process believability that has been introduced in this article.

Proposed Set of Principles for

Qualitative Researchers

1. The declaration of ontological, epistemological, … and methodological assumptions;
2. A justification of the relationship between the topic or nature of the study and the research approach(es) being used;
3. Grounding of a study within the context of appropriate quality literature;
4. Critical discussion of all aspects of the study proposal, procedures, … outcomes and reports with critical friends;
5. Conduct of a pilot study to massage the research question(s) and establish appropriate data collection protocols;
6. The development of research partnerships and contracts between researchers and participants so that all own the study processes, data and outcomes;
7. In phenomenographic terms, application of principles such as triangulation in the data collection and analysis phases of the research;
8. Declaration of the methodological processes being used in the data analysis;
9. Storage of the original research data for periods of up to seven years or the inclusion of the data base in the research report;
10. Debate and critical discussion of the processes and outcomes of the study through all possible communication genres;
11. The development and critical evaluation of theoretical frameworks relating to, and resulting from, the study; and

The integration of theory and practice through the application of each study’s outcomes to practical situations in appropriate contexts.

References:

Best, J.W. & Kahn, J.V., 1993, Research in Education, (7th edition), Boston: Allyn and Bacon.

Burns, R.B., 1990, Introduction to Research Methods in Education, Melbourne, Victoria: Longman Cheshire.

Ferris, J.A., 1996, A Question of Quality: The TAFE Stakeholders' Conceptions of Quality in Student Learning, unpublished Ph.D. thesis, Armidale, New South Wales: University of New England.

Fetterman, D.M., 1988, Qualitative Approaches to Evaluation in Education : the silent scientific revolution, New York: Praeger.

Gerber, R.G., 1993, A Sense of Quality – Qualitative Research Approaches for Geographical Education, in H. Jager, (ed), Liber Amicorum Prof Niemz, Frankfurt am Main: Goethe University Press.

Herschell, R.M., 1992, Thinking Legally, Paper and Seminar presented at the 1992 Commercial Teachers Association of Queensland State Conference, All Hallows' School, Brisbane, Australia, Queensland.

Herschell, R.M., 1995, Towards the Development of a Schema for ‘Thinking Legally’, Seminar presented at the 1995 Commercial Teachers Association of Queensland State Conference, Queensland University of Technology, Brisbane, Australia.

Herschell, R.M., 1997, Conceptions of Thinking Legally: an interpretative approach, unpublished Ph.D. thesis, University of New England, Armidale, New South Wales.

Glesne C., and Peshkin, A., 1992, Becoming Qualitative Reasearchers: an introduction, London: Longman.

Kvale, S., 1991, Ten standard responses to qualitative research interviews, paper presented at the Nordic research course, Qualitative research on learning and cognition, Grebbestad, Sweden.

Leedy, P., 1974, Practical Research: planning and design, New York: Macmillam.

Marshall, C. & Rossman, G.B., 1989, Designing Qualitative Research, Newbury Park, California: Sage.

Sandberg, J., 1997, Are Phenomenographic results reliable?, to be published in a special edition of Higher Education Research and Development.

Svensson, L., 1989, The conceptualisation of cases of physical motion, in European Journal of Psychology of Education, Volume IV, No. 4, 529-545.

Svensson, L., 1994, Theoretical Foundations of Phenomenography, (Keynote Address), in Ballantyne, R. & Bruce, C., (eds), Phenomenography: philosophy and practice, Conference Proceedings, 7-9 November 1994, Brisbane, Australia: Queensland University of Technology.

Tesch, R., 1990, Qualitative Research: analysis types and software tools, Hampshire, New York: Faraday Press.