The Process of Synthesising Qualitative Research:
A Case Study
Harsh Suri
D.S.M.E., University of Melbourne
“Issues of Rigour in Qualitative Research”
Annual Conference of the Association for Qualitative Research,Melbourne, 6-10 July, 1999
The process of synthesising qualitative research: a
case-study
In
many areas, particularly Education, economic constraints can
restrict the scale of any single research study (Draper et al.,
1992). As a consequence, the comprehensive investigation of
an area, such as internet-based courses, may require the
combination of findings from several individual studies. At
times different individual studies provide conflicting findings
or contradictory narrative accounts which can have confusing
implications for practitioners and policy makers (Wolf, 1986).
Synthesis of primary research is also important to transmit the
accumulated knowledge to lay persons and to determine the areas
that need further investigation (Cooper & Rosenthal, 1980;
Sandelowski, Docherty & Emden, 1997).
Research review
plays an important role in dissemination of knowledge and in
shaping further research and practice. Therefore the issues
of rigour in a good research synthesis are as consequential as in
a primary research study (Glass, McGaw & Smith, 1981; Dunkin,
1996). Contemporary methods of research synthesis
include traditional narrative reviews, meta-analyses and
best-evidence syntheses. Traditional literature reviews are often
narrative reports of an intuitive aggregate of individual
research findings (Johnson, 1989; McGaw, 1997). The criteria for
the inclusion of particular studies and the techniques used for
synthesis in a narrative review have not always been made
sufficiently clear, which makes it difficult for the reader to
fully appreciate the effect of the reviewer’s theoretical
position on the review’s findings.
Glass
(1976) argued that variability and uncertainty of data in
research synthesis are as evident as in the data analysis of
primary research. Hence research synthesis requires the
same rigour as is demanded in the data analysis of an empirical
study. With these views in mind, he proposed a statistical
method of research integration that he called
“meta-analysis”. Meta-analysis is the
quantitative integration and analysis of the findings from all
the empirical studies relevant to an issue and amenable to
quantitative aggregation (Glass, 1976; Wolf, 1986). In a
meta-analysis, the criteria used for selecting the findings
included in the synthesis are explicitly stated to remove any
unstated ambiguity (Hunter, Schmidt & Jackson, 1982). However,
meta-analyses ignore all the qualitative research.
To overcome the limitations of the methods of traditional narrative review and meta-analysis Slavin (1986) proposed the method of “best-evidence synthesis”. In this method, a meta-analytic integration of quantitative findings is followed by a narrative review of qualitative research. The method is freed from unacknowledged subjectivity by including well-justified and well-described inclusion criteria for empirical studies (Slavin, 1986). However, Slavin’s method fails to provide guidelines for systematic and rigorous methods of synthesising qualitative research.
Although qualitative research reports do not easily lend
themselves to synthesis, some form of synthesis is essential to
enhance the practical value of qualitative research in policy
making. Synthesis of individual reports can facilitate a
fuller understanding of the phenomenon, context or culture under
consideration. However, synthesis of qualitative research
should be interpretive rather than aggregative. While
preserving the integrity and holism of individual studies,
inductive and interpretive techniques should be used to
sufficiently summarise the findings of individual studies into a
product of practical value (Jensen & Allen, 1996; Noblit
& Hare, 1988; Sandelowski, 1997; Sandelowski, Docherty, &
Emden, 1997).
The purpose of an
interpretive synthesis of qualitative research is not to generate
predictive theories, but to facilitate a fuller understanding of
the phenomenon, context or culture under consideration (Jensen
& Allen, 1996; Sandelowski, 1997). The nature of
the synthesis product should be guided by the purpose of the
synthesis, and the relationships between the findings and
methodological positioning of individual reports. As an
illustration, content analysis could be effectively used if the
purpose of the synthesis is to categorise research literature
available in a particular area of interest, or to identify the
aspects of a particular phenomenon that need further
investigation. “Reciprocal translational
synthesis” (Noblit & Hare, 1988) is appropriate to
synthesise reports with similar findings and methods. Individual
reports with contradictory narratives are amenable to
“refutational synthesis”(Noblit & Hare, 1988) where
the synthesis attempts to explain the refutations using an
interpretive approach. When individual reports examine
different aspects of a phenomenon or culture,
“lines-of-argument” (Noblit & Hare, 1988) synthesis
is likely to be appropriate form of synthesis method. Thus,
synthesis of qualitative reports should be an interpretive,
inductive, hermeneutic and eclectic process at every stage (Jensen
& Allen, 1996).
This paper is a
report of work in progress that is part of a pilot study for my
doctoral project. The purpose of the main project is to
comprehensively synthesise research, both quantitative and
qualitative studies, in order to facilitate an understanding of
interactions between students and faculty and among students when
asynchronous (non-real time) computer-mediated-communication is
used as means of interactions in post-secondary (credit or
non-credit) courses. Although I have conducted a
meta-analysis in the past, this is my first attempt at
synthesising qualitative research. This paper will
highlight the issues, concerns and criteria that contributed to
my decision-making at various points of this part of the pilot
study. This pilot study involved carrying out a synthesis
of ten qualitative research studies (listed in Appendix A)
that were randomly selected from a pile of studies selected for
the major project.
To begin with, I realised that the research questions that can be
addressed by meta-analyses and qualitative syntheses are very
different. Examples of questions addressed by meta-analytic
synthesis are: “Is asynchronous computer mediated
communication an effective means of communication among students
and between students and faculty in post-secondary courses? What
are the variables that moderate the effect sizes?” The
purpose of qualitative research synthesis in this context was:
“to facilitate an understanding of the dynamics of
interactions among students and between students and faculty in
post-secondary courses via asynchronous computer mediated
communication.” In other words, the purpose of a
meta-analytic synthesis is more likely to accept or reject a
hypothesis while a qualitative research synthesis is better
suited to promote an understanding of the phenomenon at hand.
In a typical
meta-analysis, the findings of each report are generally recorded
on a coding sheet designed apriori and the coding process
is linear. Often meta-analyses address well-defined
hypotheses and the synthesists know exactly what is the
information that they wish to extract from individual reports
before they commence the synthesis. In this qualitative
synthesis, I tried to hear the voices of the authors from
individual reports. First of all, I read each report to
familiarise myself with the data at hand. This was followed
by an identification of context, major themes and findings
emerging from each report by using an open-coding scheme. While
coding the second paper, I had to constantly go back to the first
paper and recode it in the light of the information obtained from
the second report. Likewise, I moved back-and-forth between
individual reports while coding each new report. The coding
process in a qualitative research synthesis is eclectic rather
than sequential.
A meta-analysis aims at a
representative sample of individual empirical research studies by
including all the accessible reports. In a qualitative research
synthesis, purposeful sampling may be justifiable. One can
stop looking for more research reports on reaching the stage of
data-redundancy when every new report included in the synthesis
is likely to tell the same story rather than provide another
perspective. Relevance of reports to the current times is
another criterion to be considered for a qualitative research
synthesis. For instance, in the area of Internet based
courses standard methodologies for conducting quantitative
research have been used over the last decade. However, with
the growth of research literature in this area, qualitative
researchers have thought of innovative ways of research
methodologies that promote a better understanding of the context.
In this case, the research reported several years ago may not be
as rigorous as the current research and hence may be excluded
because it has become anachronistic.
Content analysis
was conducted to identify the general contexts and research
methods employed in individual reports. Table 1 illustrates
a part of the preliminary matrix that lists participant
demographics, content method taught, methods of data collection,
methods of data analysis and major findings of the individual
reports. Such a matrix is useful to identify the scope of
the synthesis product. Once all the relevant studies are
entered in this matrix, the individual columns can be categorised
using broader categories and the frequencies of each category
counted. This matrix is useful for identifying major themes
emerging from these studies as well as contexts that need further
examination.
All these studies
can then be divided into subsets that examined similar contexts
using similar methods. For example, reports by Thomas,
Clift & Sugimoto (1996), Schlagal, Trathen & Blanton
(1996), and Souviney, Saferstein & Chambers (1995) were
clustered into one subset. Each one of these reports used
similar methods to examine the utility of asynchronous
computer-mediated communication for teacher students to interact
with the university faculty member and the supervising teacher
while the interns were on their teaching rounds. Reciprocal
translational synthesis method was used to synthesise the common
findings from each of these three reports.
Table
1: Part of a preliminary matrix used for content analysis
| Authors
(Year) |
Context
of the study |
Data
source & analysis |
Major
findings |
| Thomas,
Clift & Sugimoto (1996) |
Teacher
Education; 11 Ss & 2 Profs; 6 months; most emails
sent to a listserv, Usual Assignments: lesson plans;
Purpose: … |
Archived
emails, participant interviews & surveys; Unit
of analysis: each msg; 6 categories: Course related,
Personal, Request for information, Pedagogical, Content,
& Confirmatory; … |
Primary
use: meeting task demands; Filtering information giving
high priority to instructors’ msgs; impersonal;
quick response from instructors; “valuable
enhancement to communicating with Profs”; feedback
to instructors enabling them to adapt the curriculum
according to students’ needs; most interactions
between Ss-faculty; few interactions between Ss-Ss; few
msges in spontaneous categories; most msgs in deliberate
categories; no participation by supervising teachers;
… |
| Schlagal,
Trathen & Blanton (1996) |
Teacher
Education; 16 Ss & 5 Profs; one year; most emails
sent to a listserv, Requirement: at least 2 assignments /
wk; Assignments: open-general themes, students’
observations; emails sent to specific lists with the
particular focus; Purpose: … |
Archived
emails; 4 categories: Response to class assignments;
Socioemotional exchanges, Housekeeping queries and
bulletins, & Spontaneous, sustained exchanges of
ideas, … |
“a
significant strand of professional conversations occurred
spontaneously on important themes”; quick response
from instructors; most interactions between Ss-faculty;
few interactions between Ss-Ss; no participation by
supervising teachers; Course requirements influence the
nature of interaction; 3 factors in structure that
promote reflective dialogue: open, thematic prompts, the
direction & focus of msges, & time to write; Ss
showed interest in what was happening in their
peer’s classes; volume of email exchanges was a
challenge for profs; Missing data:
information about filtering; … |
| Souviney,
Saferstein & Chambers (1995) |
Teacher
Education; 18 Ss (100%) & faculty in 1st
yr & 18 Ss(70% of 26) & faculty in 2nd
yr; 1yr; no requirements or assignments; Purpose: … |
Participant
interviews; Archived emails collected from a complete
information network not just a listserv; Unit of
analysis: each msg; 4 categories: Procedural, Academic,
Clinical, & Personal; incomplete data; … |
Participation
by supervising teachers; quick response from instructors
& supervising teachers; majority of msges were
procedural; As in f-t-f communication, different personal
preferences lead to different usages of the network by
individual Ss; Large number of interactions between
Ss-Ss; Large number of interactions between Ss-faculty;
… |
… |
… |
… |
… |
… |
… |
… |
… |
| Synthesis
categories |
Souviney,
Saferstein & Chambers (1995) |
Schlagal,
Trathen & Blanton (1996) |
Thomas,
Clift & Sugimoto (1996) |
| Procedural |
Procedural |
Housekeeping
queries and bulletins |
Course
related |
| Socioemotional |
Personal |
Socioemotional
exchanges |
Personal |
| Spontaneous |
Clinical |
Spontaneous,
sustained exchange of ideas |
Confirmatory;
Pedagogical |
| Responses
to course requirements |
Academic |
Responses
to class assignments |
Request
for information; Content |
Note: Four categories of Thomas, Clift &
Sugimoto (1996) were collapsed and recoded as two categories in
the last two rows.
This method assumes
that the individual reports are addressing similar issues and can
be integrated. To begin with, the key metaphors, themes,
perspectives, or concepts emerging from individual reports that
can capture the essence of that report in a reduced form are
identified. The findings of each report are then tested for
their abilities to translate the findings of other reports.
Thus we select those terms or findings that can more succinctly
describe the findings of all the reports within the subset.
(Noblit & Hare, 1988). For instance, in each study the
students tended to perceive “telecommunication primarily as
a valuable enhancement to communicating with professors”
(Thomas, Clift & Sugimoto,1996, p.172). Thus a finding
of one report could be used to express the findings of the other
two reports.
At times, the terms
employed in individual reports may not be suitable to portray
concisely all the reports. In those cases, new terms were
introduced that adequately describe the major findings from all
the reports. For example, each of the reports identified in Table
1 used different categories to analyse data from archived
emails. As illustrated in Table 2, terms used to
define the categories of individual studies were used to
reciprocally translate in the first three categories in order to
express the categories of individual studies in terms of a common
classification scheme. As none of the terms used in
individual studies succinctly defined the fourth category, a new
term was coined to describe it. This common classification scheme
enabled a direct comparison between the findings of the three
studies and facilitated the observation that in all the three
studies, majority of interactions could be classified as
“procedural”.
When individual
reports give conflicting representations of the same phenomenon,
they are not amenable to a reciprocal translational synthesis.
These reports lend themselves to a "refutational
synthesis" (Noblit & Hare, 1988) where the relationship
between individual studies and the refutations become the focus
of synthesis process. This process begins with the
identification of key findings of individual reports followed by
an examination of the relationships between individual reports.
The contradictions between individual reports may be explicit or
implicit. The implicit refutations are made explicit using
an interpretive approach. This is followed by an attempt to
explain the refutation.
As an illustration,
Thomas, Clift & Sugimoto (1996) and Schlagal, Trathen &
Blanton (1996) found a general reluctance among the supervisory
teachers to contribute to the email traffic. This
contrasted with the findings of Souviney, Saferstein &
Chambers (1995) who found that emails were a popular means of
communication between the interns, faculty members and
supervisory teachers. A closer inspection of the individual
reports revealed that the former two studies collected data from
listservs where every member of the listserv had an access to
each contribution made by any member. In contrast with
this, the third study analysed the emails archived on a network
which included personal emails sent on a one-on-one basis. In
this report, the examples of emails sent by a supervisory teacher
were addressed to individual faculty member or intern rather than
the entire listserv. This might explain the contradictory
findings. Perhaps supervisory teachers feel comfortable
with the notion of sending emails to an individual intern or a
faculty member rather than an entire cohert of interns, faculty
members and supervisory teachers enlisted on a listserv. This
example also illustrates the interpretive rather than aggregative
nature of this qualitative research synthesis.
At some level, if
the individual reports examine different aspects of the same
phenomenon, "lines-of-argument" synthesis method
(Noblit & Hare, 1988) could be used. The main purpose
of a "lines-of-argument" synthesis is to make
inferences. In this method, findings from individual
reports are used as pixels to get a fuller picture of the
phenomenon at hand. The method involves a grounded theory
approach for open-coding and identifying the categories emerging
from the data. The key categories that are more powerful in
representing the entire data-set are identified by constant
comparisons between individual accounts. These categories are
then linked interpretively to create a holistic account of the
whole phenomenon.
The broad purpose
of each study included in this pilot study was to facilitate an
understanding of some aspect of the dynamics of asynchronous
computer-mediated-communication in post-secondary courses. Yet
each study had its individual context, method of investigation
and purpose. An interpretive and grounded theory approach
was used to identify the major themes emerging from individual
reports and to link all these themes to address the broad purpose
of the synthesis.
Unlike a typical meta-analysis, this qualitative synthesis did
not follow a rigid sequential procedure. An inductive
approach was used at every stage of the synthesis process where
the decisions made were guided by the purpose of the synthesis at
that stage and the relationship between the individual findings
at hand.
Figure1 illustrates the eclectic and hermeneutic nature of
this synthesis process. Let Subset 1, Subset 2 and Subset 3
be collections of studies where asynchronous
computer-mediated-communication is used for Teacher Education,
Learning English as a Second Language, and Learning a Language
other than English respectively. Let Report 1.1, Report
1.2, and Report 1.3 be the individual reports included in Subset
1, Report 2.1, Report 2.2, and Report 2.3 be the individual
reports included in Subset 2, and Report 3.1 and Report 3.2 be
the individual reports included in the Subset 3. As
mentioned earlier, coding of each report was an eclectic rather
than a sequential process. For instance, while coding
Report 1.2, I had to go back to sections of Report 1.1 and
vice-versa.
Odman and Kerdeman (1997) define a “hermeneutic circle”
in terms of “preunderstanding” and
“understanding”(Odman & Kerdeman, 1997, pp.
186-187). Familiarity with the entire phenomenon leads to a
preunderstanding of the general context which facilitates an
understanding of the specific context. A better
understanding of specific contexts improves the understanding of
the entire phenomenon. This notion of a dialectic
hermeneutic circle was extended in this synthesis to facilitate a
comprehensive understanding of the phenomenon. I started
with reading each report to familiarise myself with different
aspects of the phenomenon and to create a preunderstanding.
This preunderstanding enhanced my understanding of individual
reports. Understanding of individual reports shed light to
my synthetic interpretation of all the reports. Thus I
often moved back-and-forth between my understanding of the whole
phenomenon and individual reports.
Issues of rigour and validity are as important in a research synthesis as in a primary research study (Jensen & Allen, 1996). An attempt was made to be
consistent and thorough at every stage of the synthesis to attain high levels of rigour. Validity was tested by checking if my synthetic interpretations resonated with experiences of the practitioners in the field.
Research synthesis
plays an important role in dissemination of knowledge and in
shaping further research and practice. Although qualitative
research reports do not easily lend themselves to synthesis, some
form of synthesis is essential to enhance the practical value of
qualitative research in policy making. The synthesis of
qualitative reports should be an interpretive, inductive,
hermeneutic and eclectic process at every stage.
REFERENCES
| Cooper,
H. M., & Rosenthal, R. (1980). Statistical Versus
Traditional Procedures for Summarizing Research Findings.
Psychological Bulletin, 87(3), 442-449. |
|
| Draper,
D., Donald P. Gaver, J., Goel, P. K., Greenhouse, J. B.,
Hedges, L. V., Morris, C. N., Tucker, J. R., &
Waternaux, C. M. (1992). Contemporary Statistics (1).
Combining information: statistical issues and
opportunities for research. Washington, D.C.:
National Academy Press. |
|
| Dunkin,
M. J. (1996). Types of errors in synthesizing research in
education. Review of Educational Research, 66(2),
87-97. |
|
| Glass,
G. V. (1976). Primary, secondary, and meta-analysis of
research. Educational Researcher, 5(10), 3-8. |
|
| Glass,
G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis
in social research. Beverly Hills, CA: Sage. |
|
| Hunter,
J. E., Schmidt, F. L., & Jackson, G.B. (1982). Meta-analysis:
Cumulating research findings across studies. Beverly
Hills, CA: Sage. |
|
| Jensen,
L. A., & Allen, M. N. (1996). Meta-synthesis of
qualitative findings. Qualitative Health Research, 6(4),
553-60. |
|
| Johnson,
B.T. (1989). DSTAT: Software for the
meta-analytic review of research literatures. Hillsdale,
NJ: Lawrence Erlbaum. |
|
| McGaw,
B. (1997). Meta-analysis. In J. P. Keeves (Ed.), Educational
research, methodology, and measurement : an international
handbook (2nd ed.), pp. 371-380. Cambridge: Elsevier
Science Ltd. |
|
| Noblit,
G. W., & Hare, R. D. (1988). Meta-ethnography :
Synthesizing qualitative studies. Newbury Park: Sage
Publication. |
|
| Odman,
P. J., & Kerdeman, D. (1997). Hermeneutics. In J. P.
Keeves (Ed.), Educational research, methodology, and
measurement : an international handbook ( 2nd ed.,
pp. 185-192). N.Y: Pergamon. |
|
| Sandelowski,
M. (1997). "To be of use": enhancing the
utility of qualitative research. Nursing Outlook, 45(3),
125-32. |
|
| Sandelowski,
M., Docherty, S., & Emden, C. (1997). Focus on
qualitative methods. Qualitative metasynthesis: issues
and techniques. Research in Nursing and Health, 20(4),
365-71. |
|
| Slavin,
R. E. (1986). Best-evidence synthesis: An alternative to
meta-analytic and traditional reviews. Educational
Researcher, 15(9), 15-11. |
|
| Wolf,
F. M. (1986). Meta-analysis : quantitative methods for
research synthesis. Beverly Hills, CA: Sage. |
|
Appendix A: List of studies included in
this qualitative synthesis
Ahern,
T. C., & et al. (1992). The Effects of Teacher Discourse in
Computer-Mediated Discussion. Journal of Educational Computing
Research, 8(3), 291-309.
de
Vries, L., & et al. (1995). On-Line Professional Staff
Development: An Evaluation Study. Distance Education, 16(1),
157-73.
Eastmond,
D. V. (1994). Adult distance study through computer conferencing.
Distance Education, 15(1), 128-152.
Leppanen,
S., & Kalaja, P. (1995). Experimenting with Computer
Conferencing in English for Academic Purposes. ELT Journal, 49(1),
26-36.
Mowrer,
D. E. (1996). A Content Analysis of Student/Instructor
Communication via Computer Conferencing. Higher Education, 32(2),
217-41.
Rogan,
J. M. (1997). Online Mentoring: Reflections and Suggestions. Journal
of Computing in Teacher Education, 13(3), 5-13.
Schlagal,
B., & et al. (1996). Structuring Telecommunications To Create
Instructional Conversations about Student Teaching. Journal of
Teacher Education, 47(3), 175-83.
Souviney,
R., & et al. (1995). InternNet: Network Communication and
Teacher Development. Journal of Computing in Teacher
Education, 11(4), 5-15.
Thomas,
L., & et al. (1996). Telecommunication, Student Teaching, and
Methods Instruction: An Exploratory Investigation. Journal of
Teacher Education, 47(3), 165-74.
Waugh,
M. (1996). Group Interaction and Student Questioning Patterns in
an Instructional
Telecommunications Course for Teachers. Journal of Computers
in Mathematics and Science Teaching, 15(4), 353-82.