The Use of Case Stories to Examine the Instructional Designer and the Subject Matter Expert Interaction
Mike Keppell
Biomedical Multimedia Unit
Faculty of Medicine, Dentistry and Health SciencesThe University of Melbourne
Australia
m.keppell@medicine.unimelb.edu.au
Abstract: This paper examines a qualitative study which developed and began testing a process to assist instructional designers in eliciting unfamiliar content from subject matter experts and in conceptualizing that content. Called by the working designation "Content Production Process" (CPP) the approach draws upon a variety of theories, constructs and methods including advance organizers, schema theory, consultation practices, current elicitation procedures in instructional design, knowledge acquisition strategies in constructing expert systems, interviewing strategies and knowledge mapping. This qualitative study focussed on how one instructional designer aided by the CPP interacted with, elicited and conceptualized unfamiliar content. Through the process of the study it was possible to examine the situation in depth, detail and from a holistic perspective. Case stories as opposed to case study were used to investigate the instructional designer–subject matter expert (SME) interaction within the real-world context of a military setting. They attempted to focus on discovery, insight and understanding from the perspectives of the instructional designer and the SME’s being studied. In this way they attempted to "illustrate conclusions to which the author ... was ... already committed" (Biddle & Anderson, 1986, p. 238). Such case stories are intended to provide new insights and understandings and "... not provide conclusions, however, that reflect evidence" (p. 238). Three major insights were gained into the conceptualization of unfamiliar content and the use of the CPP. Firstly, the novice instructional designer found that knowledge map construction was a useful tool for assisting him to think carefully about the unfamiliar content from the interviews. It is reasoned that transforming the content into a visuo-spatial representation such as a knowledge map may promote deeper processing of the content. Secondly, the knowledge map appears to have been a useful "prop" for assisting the instructional designer in interacting with the subject matter experts. It seemed to provide a common focus of attention for both designer and expert. Thirdly, the instructional designer found that combining a "teachback" procedure with the knowledge map provided a powerful means of checking his understanding of the content against that of the expert.
Nature of the Problem
Imagine attempting to design an instructional unit in an area in which you have no understanding or expertise. How would you approach the unfamiliar materials? What are your choices? You can read textbooks or reference materials or you could talk to experts in the area. Subject matter experts, however, are often busy people. How can you optimize your time spent with the subject matter expert? How can you elicit a sufficient quantity and quality of information so that instructional design is possible? What questions would you ask of the subject matter expert? This paper addresses the relationship between the instructional designer and subject matter expert and examines more particularly a possible method for eliciting and conceptualizing unfamiliar content knowledge from the subject matter expert so that efficient and effective instructional design can proceed.
This crucial skill of an instructional designer is a potential "bottleneck" for the development of instructional materials in project-based multimedia environments. For example, an instructional designer may be required to design multimedia in content areas as diverse as mining, engineering, taxation, alumina processing, beef production and medicine. It is not possible to be conversant in all of these diverse content areas, and the instructional designer must rely on subject matter experts to assist with this content. Without an effective strategy for interacting with the subject matter expert, valuable time will be lost understanding and organizing the content, hence the need for an efficient and effective method to assist in the instructional designer–subject matter expert interaction (Keppell, 1997).
Instructional Designers and Subject
Matter Experts
Instructional Designer
Instructional designers tend to be process-oriented individuals as they can apply instructional design principles to a wide range of content areas. The designer usually begins by analyzing goals, needs and learner characteristics in an attempt to understand the instructional problem. The scope and content of the subject are then identified. These areas represent the "problem space" of the instructional designer. One of the designer's main jobs is to select, sequence, synthesize and summarize the content for instructional purposes and deliver the desired product. "Despite having no content expertise, the designer has a set of representations based on a "design model" which can guide the development of an effective training program" (Nelson, Magliano & Sherman, 1988, p. 32) (italics added). The design model is analogous to a script (Schank & Abelson 1977). Scripts are knowledge packages in memory which allow individuals to understand routine activities (e.g. eating in restaurants, visiting the dentist). They are prototypical or stereotypical information useful in everyday occurrences.
Subject Matter Expert
As the name suggests the subject matter expert (SME) or the content expert is an authority on a particular domain of knowledge from whom the designer is attempting to elicit knowledge. The SME may be the client in the business, academic or military setting. Nelson, Magliano and Sherman (1988) suggest that "experts'" knowledge structures are more highly organized and well integrated" (p. 30) than those of novices. "Experts tend to "chunk" or organize information into more highly structured patterns and to complete the task more quickly than novices. Experts also appear to represent problems differently than novices because of their superior ability to recognize patterns, infer relationships, disregard irrelevant information, and recall similar problems from past experience" (Nelson, Magliano & Sherman, 1988, p. 33). One of the SME’s functions in training is to provide accurate content to the design team. In this study SME’s have two roles: (1) providing a clear description and explanation of the content area being examined, and (2) assisting the designer's conceptualization by clarifying and verifying the content.
Phases of the Study
Phase 1: Development of the Process
The CPP was developed by using an eclectic approach. Ausubel’s concept of "intellectual scaffolding" forms an important part of the CPP (Ausubel, 1960, 1963, 1968). When instructional designers are working with SME's their aim is to create a conceptual scaffold and then attach content elicited in the subsequent interactions with the SME. As designers create this conceptual scaffold (knowledge map) and elaborate the content they begin to conceptualize the relationships within the content. It seemed that some form of graphic organizer might be an appropriate tool for representing successive iterations of the instructional designer's attempts to conceptualize unfamiliar content, and of the various forms available the knowledge map was selected (Breuker, 1984; Dansereau, 1991a, 1991b). In addition to helping the instructional designer conceptualize unfamiliar content (Wedman, 1987), it was thought that there would emerge a second advantage of making the instructional designer’s perceptions public for the subject matter expert to examine.
Phase 2: Personal Trial of the CPP
The personal trial of the CPP helped to crystallize the researcher's conceptualization of the process. Initially, it was necessary to keep pace with the SME and attempt to obtain an overview of the content area. A second phase began with the SME viewing the map and understanding the role of the instructional designer within the interaction. At this point in the interaction the SME appeared to become more cooperative and provided more detailed explanations because a certain rapport had been developed. A third phase involved both the designer and SME collaborating to achieve the goals of the interaction. The fourth phase centered on cultural understanding with the instructional designer feeling more effective in conceptualizing the content when the content could be viewed from the SME's point of view. After the trial an effort was made to formalize the process to allow it to be taught to another instructional designer.
Phase 3: Training a Novice Designer to Use the CPP
The training program addressed both the declarative and procedural knowledge required by a novice designer when working with SME’s in a "real" setting. The novice designer (John) was trained for a total of thirty-three hours in all aspects of the CPP. Personally teaching the process to John provided the researcher with the opportunity to observe the process in action and document the proceedings. John achieved competency in all aspects of the CPP as it was conceived at that point and was ready to work with SME’s in a real setting. He had shown competence in both declarative and procedural knowledge in relation to the CPP.
Phase 4: Case Stories
The researcher used a case stories approach as John applied the CPP to investigate the designer–SME interaction within the real-world context of a military setting. Case stories focus on discovery, insight and understanding from the perspectives of the designer and the SME’s being studied. In this way they attempted to "illustrate conclusions to which the author ... was ... already committed" (Biddle & Anderson, 1986, p. 238). Such case stories are intended to provide new insights and understandings and "... not provide conclusions, however, that reflect evidence" (p. 238).
This case focussed on how one instructional designer aided by the CPP interacted with, elicited and conceptualized unfamiliar content. Through the process of the case it was possible to examine the situation in depth, detail and from a holistic perspective. This portrayal provided a solid basis for examining what took place in the interaction. The following important areas emerged during the case: (1) Strategies for conceptualizing the content, (2) the knowledge map as a communication prop, (3) the teachback function and the knowledge map.
Strategies for Conceptualizing the Content
John’s preferred approach to gaining an initial understanding of the content was to obtain an overview of the area. He appeared to need to see the conceptual terrain of the content area before examining the area in more depth. This was consistent with the author's own application of the CPP. John commented that he needed to do this so that he could see the extent of the information that needed to be covered. It may represent a heuristic that designers need to use in the process of interacting with SME’s.
John also found that the knowledge map was a useful tool which assisted him when wrestling with unfamiliar content and displaying the content to the SME. John suggested that the construction of the knowledge map "forced me to think about what was said" and "focuses my thinking". The use of the knowledge map appeared to assist John in thinking more carefully about the content. Holley and Dansereau (1984, p. 8) suggest that the creation of a knowledge map may force the constructor to process the content in "greater semantic depth". Furthermore, Holley and Dansereau also suggest that reorganizing the information in the form of a knowledge map may activate both the spatial and verbal processing systems, allowing the developer to gain a deeper understanding of the information.
The Knowledge Map as a Communication Prop
The use of the knowledge map in the designer–SME interview may act as a communication "prop" to improve the interpersonal nature of the interview. The knowledge map may help to focus the attention of the SME and designer on the most important parts of the content. Accurate content on the map may not require further attention. It may not be necessary to further explore this information because the SME has verified the accuracy of the content. John used the map in this sense by check-marking accurate information on the map as it was examined by the SME. In this sense the map may act as a communication device that confirms the knowledge shared by the designer and SME. The focus of the interview can then be concerned with the areas that are most in need of attention.
Lambiotte, Dansereau, Cross and Reynolds (1989, p. 332) suggest that knowledge maps are "computationally efficient" in that they "facilitate faster search and recognition of relevant information" (p. 332). Efficiency is defined in terms of how the knowledge map representation assists attention focusing, knowledge assimilation and knowledge searching of new information. The presentation of the knowledge map to the SME may activate both the spatial and verbal systems thus increasing the processing efficiency of the information.
The Teachback Function and the Knowledge Map
In the initial conception of the CPP it was envisaged that the designer would stop at certain points throughout the interaction and teach the material back to the SME. This proved far too ambitious for a designer who is totally unfamiliar with a content area. It was not possible for the designer to gain sufficient familiarity with the content to teach back the material to the SME. John also felt uncomfortable with this practice. "I still think the toughest nut to crack for me is the teachback, I could just go over it and kind of lecture him, but I really felt uncomfortable doing that". Therefore it seems more reasonable to assume that the designer must concentrate on obtaining the big picture and must postpone paraphrasing and asking in-depth questions until a later point in the interaction. A great deal of reflection and problem-solving was required to develop a conceptualization of the SME's content area.
It would appear that the teachback procedure is more appropriately used at the beginning of a subsequent interview. With the aid of the knowledge map, John found this a powerful means of checking his understanding of the content against the expertise of the SME. John gathered the information in the interview, reflected on the content, constructed the map and then taught the material back to the SME at the beginning of each interview. This proved an effective adaptation of the CPP over what was originally conceived.
Implications
This study set out to develop a possible method for assisting the elicitation and conceptualization of unfamiliar content by a designer. It examined a trial by the researcher and the application by another designer. The aim of the study was to address a potential "bottleneck" in the instructional design process which had not been addressed in the research literature, and to provide a starting point for further research into the elicitation and conceptualization of unfamiliar content.
The introspective analysis associated with the personal trial of the CPP by the researcher provided a rich source of information in preparation for teaching the method to another instructional designer, an experience which might be described as "reflection-in-action" and "reflection-on-action" (Schon, 1987). Schon suggests that "knowing-in-action" "… attempts to put into explicit, symbolic form a kind of intelligence that begins by being tacit and spontaneous" (p. 25). By examining his own application and the adjustments that were required before training another designer, the researcher could make some initial refinements. Further, it helped in interpreting John's experience.
However, this introspective study also has a large number of risks. It raises more questions than it answers. The researcher’s application may have been an idiosyncratic account of the interaction with the SME. He has a preference for visual representations like the knowledge map for representing content. In fact, he now works in the multimedia field which is a visually oriented area. Content within this area is portrayed by specialised visual representations. This visual preference may not be as strong for all designers. The researcher also has a preference for obtaining a "bird’s eye view" of content before examining the content in more detail. Again, this may not be the preference of other designers who prefer to examine a particular concrete instance or example rather than obtain the big picture of a content area. The method was useful to the researcher; however, it is not known at this point how applicable the method is to the instructional design field in general.
Conclusion
This study has examined the front-end investigation of eliciting and conceptualizing unfamiliar content by an instructional designer. The work is significant as it has carefully documented the interactions of a novice designer with SME’s in a real context. The process appears promising in assisting the conceptualization of unfamiliar content and improving the designer–SME interaction. Further research needs to be undertaken with a variety of designers and SME’s to determine the applicability of this process for the field of instructional design.
The use of case stories provides a means to contribute authenticity and bridge the gap between theory and practice. Case stories may allow the development of rich insights into professional practice. It is essential to understand the designer–SME interaction due to its role within the project driven environments of multimedia design, but it is difficult to understand and improve this practice "while immersed in it" (Ackermann & Maslin-Ostrowski, 1995, p.4). "The case story is viewed as a form of "situated knowledge", and, as such, represents a vital yet often misunderstood dimension of learning and sharing of a practice" (Maslin-Ostrowski & Ackermann, 1997, p.1).
References
Ackerman, R. & Maslin-Ostrowski, P. (1995). Developing case stories: An analysis of the case method of instruction and storytelling in teaching educational administration. (ERIC Documentation service No. ED 390 132).
Ausubel, D. P. (1960). The use of advance organizers in the learning and retention of meaningful verbal material. Journal of Educational Psychology, 51, (5), 267–272.
Ausubel, D. P. (1963). The psychology of meaningful verbal learning. New York: Grune & Stratton.
Ausubel, D. P. (1968). Educational psychology: A cognitive view. New York: Holt, Rinehart & Winston.
Biddle, B. J. & Anderson, D. S. (1986). Theory, methods, knowledge, and research on teaching. In M. C. Wittrock (Ed.), Handbook of research on teaching (3rd ed.) (pp. 230–252). New York: Macmillan.
Breuker, J. A. (1984). A theoretical framework for spatial learning strategies. In C. D. Holley & D. F. Dansereau (Eds.), Spatial learning strategies: Techniques, applications and related issues (pp. 21–46). Orlando: Academic Press.
Dansereau, D. F. (1991a, December). Dumbells and dyads: Structured experiences that expand thinking. Paper presented at the University of Calgary, December 5, 1991.
Dansereau, D. F. (1991b, December). Knowledge prototypes (derived structural schemas) and the facilitation of transfer. Paper presented at the University of Calgary, December 6, 1991.
Holley, C. D. & Dansereau, D. F. (1984). Spatial learning strategies: Techniques, applications and related issues. Orlando: Academic Press.
Keppell, M. J. (1997). Development and pilot-testing of a method to assist instructional designers elicit unfamiliar content from subject matter experts. Unpublished doctoral dissertation, University of Calgary, Alberta, Canada.
Lambiotte, J. G., Dansereau, D. F., Cross, D. R. & Reynolds, S. B. (1989). Multirelational semantic maps. Educational Psychology Review, 1, (4), 331–367.
Maslin-Ostrowski, P. & Ackerman, R. (1997). A case for stories: Toward further understanding of situated knowledge and practice. (ERIC Documentation service No. ED 408 265).
Nelson, W. A., Magliano, S. & Sherman, T. M. (1988). The intellectual content of instructional design. Journal of Instructional Development, 11, (2), 29–35.
Schank, R. C. & Abelson, R. (1977). Scripts, plans, goals and understanding. Hillsdale, N J: Lawrence Erlbaum Associates.
Schon, D. A. (1987). Educating the reflective practitioner. San Francisco: Jossey–Bass.
Wedman, J. F. (1987). Conceptualizing unfamiliar content. Journal of Instructional Development, 10, (3), 16–21.