USING QUALITATIVE COMPUTER ANALYSIS TO INFORM YOUR LITERATURE SEARCH WHEN THE LITERATURE IS ONLY VAGUELY RELEVANT - A CASE STUDY ON THE TRAINING NEEDS OF SCHOOL GOVERNORS

Paper presented at Inaugural Conference of

Association for Qualitative Research,

Melbourne, July 14 - 17, 1999

Research findings about the learning needs of those charged with school governance are rare. This paper describes a computer procedure that was devised in order to complete an initial analysis of literature on governance so that available materials could inform an investigation into the learning needs of Boards of Trustees (BOT's). It is concluded that despite the absence of specific research into this matter, tentative hypotheses concerning BOT learning needs were able to be generated by using a range of computer applications in tandem. By using EndNote, a word processing programme and QSR NUD*IST interactively, it became possible to clarify issues contained within the literature about governance and training needs. It is proposed that this computer strategy can be usefully applied to a range of research projects thereby aiding the quality of the literature review which informs the research output.

Dr. Jens J. Hansen
School of Education
UNITEC Institute of Technology
Auckland, New Zealand
Email: jhansen@unitec.ac.nz

Overview

Research findings about the learning needs of those charged with school governance are rare. This paper describes a computer procedure that was devised in order to complete an initial analysis of literature on governance so that available materials could inform an investigation into the learning needs of Boards of Trustees (BOT's). It is concluded that despite the absence of specific research into this matter, tentative hypotheses concerning BOT learning needs were able to be generated by using a range of computer applications in tandem. By using EndNote, a word processing programme and QSR NUD*IST interactively, it became possible to clarify issues contained within the literature about governance and training needs. It is proposed that this computer strategy can be usefully applied to a range of research projects thereby aiding the quality of the literature review which informs the research output.

When available literature is akin to a mouthful of feathers

Contrary to expectations, researchers are not necessarily overjoyed when they find that a plethora of literature exists about matters they contemplate investigating. Instead, their pulses are more likely to quicken when they discover that their intended research domain is, relatively speaking, uncharted. Put another way, this means that ascertaining that seemingly everyone else has already plumbed the depths of a phenomenon is not nearly as nice as learning that an issue beckoning investigation is, at the time of calling, relatively unexplored. But irrespective of whether there is an abundance of literature on a topic or a paucity of references, convention demands that data contained within existing literature be discursively sorted. That accepted academic ordinance is the literature review.

Such an exercise inevitably seeks to inform the researcher and academic peers not only of what has previously been said about the matter under review, but also provides critical commentary about how thoroughly a subject has been traversed and how such information can be extended. Thus do some research proposals that endeavour to extend or confirm current epistemology become justified. And, moreover, it can also be argued that a critique of literature that is worth it's salt, will almost always identify shortcomings. The reviewer will highlight aspects about the matter being examined that have, at best, not yet been adequately researched, or, at worst, have not yet been investigated at all. Thus do other research projects that address epistemological deficits sometimes become justified.

All this assumes, of course, that there is an actual availability of useful literature. However, in the case of this study it became apparent very early that there has been very little specific research into the training needs of Boards of Trustees – this issue is characterised by an almost total absence of specific literature. There is, however, a fairly extensive body of information on Boards of Trustees and governance that has emerged since the advent of the educational reforms of the late 1980's. Broadly speaking, that literature describes the emergence of, and the early impact on education, of the establishment of Boards of Trustees. It also describes the nature of governance and describes accountabilities and roles which Boards of Trustees must assume once they have accepted the mantles of governance.

In sum, given the specificity of this research project, it appeared on the surface that there was little literature that might illuminate possible research pathways to explore. There was, in effect, little relevant flesh in the literature, but rather, a mouthful of feathers.

A Rationale for this Study and Parameters of the Project

Before describing the computer assisted procedures that were developed for facilitating a literature review for this study, it is useful to briefly describe the context of this research project which is being conducted by educational management staff at the School of Education at UNITEC Institute of Technology. The investigation has been made possible because funds were secured in mid 1998 from a competitive pool which the institute makes available to staff.

Funding was initially sought because it was felt that the 1998 elections for Boards of Trustees would give rise to a diverse range of incoming trustees who would have an equally diverse range of learning needs as trustees. The board elections, the third such to be held throughout New Zealand since the advent of the 1989 education reforms, would see fresh boards formed whose composition would, in all likelihood, comprise a blend of experienced and novice trustees. Those who had previously served as trustees for one or more terms, might simplistically be thought of as experienced, whereas members who were being elected for the first time might, with equal simplicity, be thought of as inexperienced. Hence it was felt that the reservoir of prior experience and knowledge pertaining to the roles and accountabilities of trustees, and the capacity to practice the range of skills needed to effectively perform governance duties, would be extremely varied. Accordingly, it was felt that an investigation into the learning needs of School Boards of Trustees would be both timely and informative.

At the outset, it was intended that the research would be completed in one year. However, it has already become clear that the project will take close to two years to complete if it is to be accorded an appropriate degree of practical thoroughness and academic rigour. Even the discovery that there is an absence of specific literature has meant that extraordinary time consuming procedures have had to be devised and tested. The development of this paper is directly attributable to this. The initial research objectives, however, together with attendant research questions and contemplated data collection and management strategies are outlined in Table One.

TABLE ONE: Research Objectives, Questions and Procedures

Source: Jens. J. Hansen, 1999

An examination of Table One reveals that the objectives as initially stated, the ensuing research questions, and the strategies to be employed for completing this research, are relatively straight forward. However, the variables which can impact upon a study such as this are anything but straight forward, instead, they are manifold and complex. For instance, the socio-economic status of individual board members and the socio-economic status of the school (even when expressed oversimply by ascribed decile ranking) are important variables to consider. Similarly, the individual and collective history of formal and informal education of board members, the amount of available time they can devote to governance, their relationships with each other and with the school's senior educational managers, are variables which will impact upon the workings of boards.

These variables will also drive the thrust of training needs. Moreover, given even minimal variability within and between boards, the range of expressed as well as unexpressed or perceived learning needs can be expected to be diverse – board members will be variously aware of, and will variously be able to express their individual and collective learning needs, and equally, they will variously remain unaware of other training and development needs.

Given these comments, the completion of a comprehensive literature review assumes an even greater degree of importance because an investigation that has been well informed by surrounding literature will help investigators probe matters that might not otherwise have been discussed. Thus might both expressed and perceived learning needs of boards of trustees be explored and become confirmed.

Although such comments are largely theoretical, they remain important because they contribute to the raison d'tre for what follows. At the same time, it should be recognised that at the stage of writing this paper the literature analysis for this study has been commenced but has not been completed. And as indicated earlier, the initial foray into the literature unearthed a paucity of data specifically pertaining to the training needs of trustees irrespective of whether they have been elected to boards for one, two or even three terms. Thus for all intents and purposes, the learning needs of those who govern our schools are largely unknown. Certainly, they have not been widely reported. It is, however, reasoned that inferences about learning needs can be drawn from the surrounding literature. The argument goes something like this:

The answer, it is suggested, involves the sequential invoking of functions from three academic software applications – EndNote, MS Word and QSR NUD*IST.

• First, EndNote, a bibliographic database programme is used to generate a body of bibliographic data together with abstracts and/or researcher generated reviews. It is essential that the abstracts/reviews be included for it is these that form the basis of ensuing content analyses.

• Second, the EndNote data are copied into Microsoft Word so that the format of the database can be prepared for introduction into QSR NUD*IST. The important point here is that the data must be converted into a format that enables content analysis to proceed smoothly in QSR NUD*IST.

• Third, once entered into QSR NUD*IST, data are analysed by completing a series of searches using key words and/or synonyms. Resultant finds can then used to inform the researcher about which items of literature warrant even further probing. Finds can also be used to generate hypotheses which will inform interviews and focus groups schedules.

What follows is a more detailed description of the procedures used.

An Account of the Procedures used

At the outset, a research assistant and staff at the UNITEC library assisted with on-line data base searches using key words, e.g. Trustees, Not for Profit, School Councils, etc. As a result some 400 references, spanning the decade from 1988 when Boards were first mooted, through to 1998 were found. A manual inspection indicated that only a handful of these (approximately a dozen) referred to training. However, many other issues confronting boards had been reported. It was therefore decided that the print out of all of the references would be manually sorted into two simple categories, Reject and Explore Further. Bibliographic details from the Explore Further category (n=118) were subsequently entered into EndNote. Later, using Boolean operators to search the EndNote data base, it was confirmed that only 10 of the 118 citations (8%) explicitly mentioned training needs or continuing professional education or professional development. It could, therefore, clearly be demonstrated that the matter of the training needs had not been reported within the body of literature that had been sorted as worthy of further exploration.

Two MS Word print outs were then generated from EndNote. The first showed all of the 118 references sorted by year (to achieve this select the number menu from the Styles Menu and choose the year of publication field from the sorting option in the Analysis menu). This chronological representation enabled a simple and rapid scan of how the literature had unfolded over a decade. An interesting finding was that a good deal of the early literature clearly came from New Zealand although contributions from the Netherlands, the UK, the USA, Australia and Canada were also apparent. The Table below shows the general thrust of the literature, which included newspaper articles, by year (see Table Two). Notably, initial writings tended to focus upon the impact of the educational reforms and some manuals of practice were developed. During the early 1990's conference proceedings were produced and the issue of Maori participation was discussed – albeit not widely (1973; 1979; Bernstein, 1980). By the mid 1990's, however, the writing agenda had shifted so that the roles of governors and the impact of Boards of Trustees on school effectiveness became the foci. Also emerging at that time was a tendency to stock take the performances of boards and principals and to highlight difficulties that boards and schools were encountering. Finally, a comprehensive review of the effectiveness of trustee training was completed by Gilmore (1998).

TABLE TWO: An Overview of BOT Literature. 1988 –1999

The second print out comprised all of the bibliographic details of the 118 references together with available abstracts. To achieve this, the Styles menu of EndNote must be edited so that the abstract field becomes included in the print out. (This option, incidentally, is very useful for generating annotated bibliographies.) It is worth noting that the print out was printed in landscape form so that preliminary margin notes could be manually introduced. Such a procedure is useful as it helps to inform subsequent content analysis.

It is important to point out here that QSR NUD*IST analyses non-numerical unstructured data, i.e words, in blocks. These blocks of data are called text units. Specifically, the programme allows researchers to either search and code many documents concurrently, or to code documents and their text units one at a time. In other words, searches enable users to retrieve each of the text units that have been found to contain words that conform to the category or categories being searched for and these can then be coded for subsequent retrieval from within a database of coded finds. Furthermore, searches can be directed to include or exclude specific categorical elements such as year of publication, or gender, or whatever is appropriate to the analyses.

The researcher can also elect to capture a spread of text units on each side of finds or can elect to keep as their spread, the subsection of the document within which each find occurs given that such subsections are separated by an asterisk *. Such sections are known in QSR NUD*IST vernacular as sub-headers. Moreover, depending upon the degree of specificity of search required, a text unit can be defined by preference as either a line or a paragraph. When a single line is selected as the text unit, only the line containing the word or words being searched for is shown but where a paragraph is selected as the text unit, the entire paragraph containing the find is shown. Each option has advantages and disadvantages but their discussion falls outside of the parameters of this paper.

However, what had been achieved in the conversion from EndNote to MS Word was not yet amenable to conducting a QSR NUD*IST analysis because the print out of the 118 references was akin to one large disorganised block of text. In short, the disorganised format meant that effective and efficient content analysis would be well nigh impossible to achieve. Finds, if derived from the abstract section of each citation entry, and if kept in the form of the paragraph within which the find was contained, would eliminate the accompanying bibliographic details; conversely, if the find was derived from the bibliographic details, the abstract would be left out. And if the spread of the text containing that find was set at one (paragraph) text unit on each side, the resultant finds would include either a superfluous abstract or a superfluous citation.

It was, therefore, decided to massage the citations and abstracts which had, it will be recalled, been saved as an MS Word document. This was achieved by invoking a series of find and replace searches which enabled materials to be formatted so that each citation became prefaced by an asterisk *. Furthermore, each abstract became linked (by a soft return) to the citation data which effectively meant that each citation and it's accompanying abstract became one, rather than two paragraphs. It also meant that each of the 118 citations and their associated abstract became individual sections of the document that was to be introduced to QSR NUD*IST. Hence, the results of searches could be kept using a spread that was confined to the subsections within which the data had been found, i.e. to individual citations and their abstract. The Figures below, which shows one data entry only, demonstrates the appearance of the bibliographic and abstract data before and after this procedure had been completed.

Creese, M. (1995). Effective Governors, Effective Schools: Developing the Partnership. London: David Fulton Publishers. ^#

^## The purpose of this book is to help school governing bodies in Britain to work more effectively in partnership with staff towards improving the quality of education provided by the school. Increased responsibilities mean that the way in which the governors carry out their duties can now have a significant impact upon the management of the school. This book is intended to help school governors explore some key aspects of their role, and to also help teachers understand the work of the governing body more clearly, enabling them to work more closely in partnership with the governors. The book is divided into six main chapters: (1) The Role of the Governing Body; (2) Building the Team; (3) Governor - Staff Relationships; (4) Managing the Work of the Governing Body; (5) Governors and the Effective School; and (6) How Effective is your Governing Body? and faith to their particular schools. Each chapter is followed by a "signpost", an interlude that summarises the lessons learned in that chapter. (ERIC)

FIGURE ONE: Sample Reference & Abstract from BOT Database

prior to preparation for NUD*IST

* Creese, M. (1995). Effective Governors, Effective Schools: Developing the Partnership. London: David Fulton Publishers.
The purpose of this book is to help school governing bodies in Britain to work more effectively in partnership with staff towards improving the quality of education provided by the school. Increased responsibilities mean that the way in which the governors carry out their duties can now have a significant impact upon the management of the school. This book is intended to help school governors explore some key aspects of their role, and to also help teachers understand the work of the governing body more clearly, enabling them to work more closely in partnership with the governors. The book is divided into six main chapters: (1) The Role of the Governing Body; (2) Building the Team; (3) Governor - Staff Relationships; (4) Managing the Work of the Governing Body; (5) Governors and the Effective School; and (6) How Effective is your Governing Body? and faith to their particular schools. Each chapter is followed by a "signpost", an interlude that summarises the lessons learned in that chapter. (ERIC)

 

FIGURE TWO: Sample Reference & Abstract from BOT Database

after preparation for introduction to NUD*IST

Three points of difference, therefore, exist between the first and the second versions of the same data. First, the re-formatted version (Figure Two) enables the entry to be considered as a discrete text unit because the introduction of the asterisk converts it into a document subsection; second, the replacement of the hard return with a soft return at the end of the citation merges the citation with the ensuing abstract; third, the changing of the font to courier (12 point) together with the operation of saving it as text only, is a standard procedure that prepares data for introduction to QSR NUD*IST.

Once all of the bibliographic da had been transformed so that they were, in appearance, the same as shown in Figure Two above, they were entered into QSR NUD*IST for thematic analysis. And at the point of writing, this analysis remains ongoing. As those who are familiar with the use of QSR NUD*IST are aware, the procedure for completing content analysis from a data base of non numerical unstructured data involves iterative searches for key words and/or their synonyms.. These searches are informed not only by previously prepared hard copy margin notes but also by the emergent finds themselves.

It is clear, even at this early stage, that the results of these searches will indeed provide useful leads with which to shape the design of focus group discussions and semi-structured interviews. And if necessary, the codes of the categories discerned can be exported from QSR NUD*IST into a mind-mapping programme such as Inspiration so that a visual model of a possible interview schedule can be achieved . But irrespective of the findings, it is apparent that the procedure which has emerged is a useful research tool.

Tentative Findings

It is tempting now to elaborate on what has emerged from the execution of this experimental research procedure, but realistically, detailed discussion is premature. Instead, it is more appropriate to make tentative observations about emergent trends and about issues that have not emerged. Such issues may still be submerged either because they remain undiscovered in this partially completed literature analysis, or because they have not yet been dealt with in depth by any other researcher. In addition, and in order to reinforce these comments, it was considered useful to show categories (nodes in QSR NUD*IST vernacular) that were derived from initial searches of the data. These are shown overleaf in Table Three.

From the partial analysis completed at the time of writing, it appears that clarifying the roles of boards of trustees and determining their responsibilities has been a regular issue within the literature. The matter of to whom boards are accountable (and for what) also appears to be a feature of the literature and it seems clear that relationships between Boards of Trustees and school management staff has been internationally explored, especially with respect to the governance management divide and, more recently in New Zealand at least, with respect to the matter of performance appraisal. Finally, it seems that recognising their own training needs may be an issue for some trustees although there does seem to be a range of resource materials available to boards.

TABLE THREE: Early Nodes derived from text searches.

(1) /By Year

(1 1) /By Year/1988

(1 2) /By Year/1989

(1 3) /By Year/1990

(1 4) /By Year/1991

(1 5) /By Year/1992

(1 6) /By Year/1993

(1 7) /By Year/1994

(1 8) /By Year/1995

(1 9) /By Year/1996

(1 10) /By Year/1997

(1 11) /By Year/1998

(2) /By Theme

(2 1) /By Theme/Governance

(2 1 1) /By Theme/Governance/Status of BOT

(2 1 1 1) /By Theme/Governance/Status of BOT/power

(2 2) /By Theme/School Committee

(2 3) /By Theme/STA

(2 4) /By Theme/Appraisal

(2 5) /By Theme/Gender

(2 6) /By Theme/Equity

(2 7) /By Theme/SES

(T) //Text Searches

(T 1) //Text Searches/1988

(T 1 1) //Text Searches/1988/Governance 1988

(T 1 12) //Text Searches/1988/School Committees

(T 2) //Text Searches/1989

(T 3) //Text Searches/1990

(T 4) //Text Searches/1991

(T 5) //Text Searches/1992

(T 6) //Text Searches/1993

(T 7) //Text Searches/1994

(T 8) //Text Searches/1995

(T 9) //Text Searches/1996

(T 10) //Text Searches/1997

(T 11) //Text Searches/1998

It also seems that the literature is deficient in some areas. It appears to be bereft of information on the strong possibility that training needs of boards of trustees and the socio-economic status of schools are linked. This variable clearly needs to be explored. Furthermore, although some data are available on matters pertaining to Maori participation in governance, this matter also demands further investigation, especially if pre-recruitment training opportunities engender higher levels of Maori participation in governance. In a similar vein, issues pertinent to gauging how well boards understand the concept of schools being responsive to community needs and to equity (gender, student, and community representation) warrant further research for that too may unearth learning needs. Finally, there is the matter of how best to deliver effective training to trustees and it is suggested that this matter definitely requires inclusion in completing a learning needs analysis for trustees.

The next stage of the research will obviously involve the completion of the literature search – documents for detailed consideration will be selected, read and analysed with commentary documents and/or EndNote citations and reviews once again being added to the QSR NUD*IST document data base.

Ultimately I am confident that these data will inform the design of the interview schedule and will guide focus group agendas. Time and money permitting, it is hoped that focus groups and interviews will be completed in Wellington and Auckland across a stratified sample of schools. Emergent data will be transcribed for analysis via QSR NUD*IST. Again, with time and money permitting, it is anticipated that findings will be validated via further focus groups.

But above all, as this account has revealed, a useful research procedure has been devised that enables computer assisted analysis of vaguely relevant literature to proceed speedily and, I believe, effectively. Moreover, I believe that this procedure is transportable across projects irrespective of whether bibliographic data are highly useful or not especially helpful. However, the message that this strategy should proceed in tandem with traditional literature review procedures must be strongly underscored. This strategy is not a substitute procedure to manually examining selected literature. It is an augmentive procedure. Conceivably though, the content from selected literature could be entered into a computer by using flat bed scanning devices which convert paper based text into digital data. Such data then become amenable to computer analysis. Now there is another story....

References

Cohen, L. & Manion, L. (1993). Research Methods in Education. London: Routledge.

Creese, M. (1995). Effective Governors, Effective Schools: Developing the Partnership. London: David Fulton Publishers.

Gilmore, A., MacGibbon, L. & Besley, S. (1998). Evaluation of the Effectiveness of Training for School Boards of Trustees: Final Report Prepared for the Ministry of Education University of Canterbury.

Glaser, B. & Strauss, A. (1968). The Discovery of Grounded Theory. London: Nicolson.

Hansen, J. J. (1995). So what's with this NUD•IST stuff? Confessions from one who's been there. University of New England, Armidale:

Kilmister, T. (1989). Brilliant Boards: The Art of Governance. Wellington: NFP Press.

Kilmister, T. (1990). The Effective School Board: Policy Governance in Action. Wellington: NFP Press.

Kilmister, T. (1993). Boards At Work: A New Perspective on Not-For-Profit Board Governance. Wellington: NFP Press.

Johnston, P. (1991) A Fair Measure of Influence?: Maori Members on School Boards of Trustees. MA, The University of Auckland.

Johnston, P. (1992a). Enabling, Encouraging or Empowering?: Maori Members on School Boards of Trustees. Access: Critical Perspectives on Cultural and Policy Studies in Education,11, 1 - 17.

Johnston, P. (1992b). A Fair Measure of Influence?: kei hea te mana Maori o nga poari kaitiaki? In Australian Association for Research in Education/ New Zealand Association for Research in Education Joint Conference, (pp. 26). Deakin University, Geelong:

Lange, D. (1988). Tomorrow's Schools: The Reform of Education Administration in New Zealand. Wellington: Government Printer.

Lincoln, Y. S. (1995). Emerging Criteria for Quality on Qualitative and Interpretive Research. Qualitative Inquiry,1, 275 - 289.

Miles, M. B. & Huberman, A. M. (1984). Qualitative Data Analysis - A Sourcebook of New Methods (1st ed.). Newbury Park: Sage.

Richards, L. (1997). QSR NUD•IST User Guide. Melbourne: Qualitative Solutions and Research.