Running up the hill: Contract Research, Qualitative Methods and Rigour

Dr Maree V Boyle
Graduate School of Management
University of Queensland
St Lucia Qld 4072
Tel: 61 7 33656751 Fax: 61 7 33656988
Email: m.boyle@gsm.uq.edu.au

Paper presented at AQR ’99
held at theDuxton hotel, Melbourne,
July 10, 1999.

Abstract

Academic researchers in all disciplines are now under pressure to work more closely with industry and the community through contract research. These alliances are expected to produce increased quantities of research output for both the individual academic and institution, as well as successful and tangible outcomes for the organisation. The design and execution of contract research, however, is somewhat different to that of funded research. It often involves tight time lines, a strict adherence to the requirements of the written and verbal contract between all parties concerned, and intensification of research work practices. While permission to publish, access to sites and informants, academic freedom and ownership of intellectual property are challenges that most qualitative researchers face at some stage, those who wish to utilise contract research data with a view to producing rigorous and high quality scholarly work face additional hurdles. Issues concerning rigour and validity may present many particular challenges to the contract qualitative researcher.

This paper examines this tension between opportunities to engage in qualitative research via contractual arrangements with organisations and the desire to produce intellectually rigorous outcomes. Two case studies that illustrate the difficulties inherent in conducting contract-based qualitative research will be discussed. The first case study examines the development of qualitative research design as part of a larger project on innovation, technology and change management within a multinational corporation. The second case study analyses the relationship between length of contract, grounded theory and methodological rigour within a contract based multiple case study project on public sector service delivery.

Introduction

Significant change within the Australian tertiary education sector in recent years has meant that contract research as a primary way of funding research is becoming more common. The typical university-based research career now involves raising private as well as public sector funds and the integration of contract or "applied" with pure research. Changes to how research is funded has led to increasing collaboration with the various industry and community sectors. This increasingly complex scenario then raises many issues. Several of the main issues include the concern surrounding ownership of intellectual property, particularly in relation to what the researcher can do with data, and the role of the researcher during the active research phase. Researchers are often required to grapple with the challenge of being cast in the change agent role, being charged with the claim of adopting the role of servant of power and struggling with the role impartial observer all within the same research project.

The juggling of these often disparate roles in qualitative contract research is closely linked to the issue of rigour, particularly in business and management research, where good qualitative research has to fight for legitimacy within a discipline that is dominated by a positivistic paradigms (Bate 1997). The nature of contract research itself therefore places demands and restrictions upon academic researchers. This paper then focuses on the tensions inherent in conducting qualitative research on a contractual basis within organisations or contexts that have a business or public administration focus. Two case studies will be used to illustrate the challenges qualitative researchers face when conducting contract research within these types of organisations or contexts.

Academic or Consultant?: Juggling Multiple Research Roles

The adoption of particular roles within a research context and the impact this has upon research quality is central to issues of rigour and validity. Academic business researchers who take on consultancies with a view to producing sound scholarly work as well as meeting the requirements of the contract often are torn between the demands of two different roles – that of independent and free thinking researcher, and that of management consultant (Gummesson 1991).

While many business academics may feel comfortable wearing the "academic as consultant" hat, others who conduct research within similar settings for the first time often struggle with the instrumental nature of contract research, where research is no longer valued for its own intrinsic worth (Becker 1996). In terms of qualitative research, the level of discomfort rises when the qualitative researcher is often forced to make a choice about research techniques and strategies based on expediency and cost effectiveness rather than paradigmatic or personal preference. Hammersley and Atkinson (1994) aptly point out the often tenuous relationship that exists between the expectations of ethnographic research and the practical realities of social and political practice. In this era of the postmodern or reflexive turn in ethnographic research, the gap between the two appears to be widening.

In qualitative contract research, data collection and analysis no longer remains an end in itself, a contribution to disciplinary knowledge. Instead, there is often an expectation that it will somehow magically develop into a transformative mechanism (Atkinson and Hammersley 1994 ). It appears that qualitative research carries with it expectations that quantitative research does not –that is, it possesses a greater propensity to effect genuine and lasting change within an organisation. Taken to an extreme this expectation may turn a simple piece of qualitative data analysis into a highly political tool.

The degree to which this occurs does depend on the kind of role a contract qualitative researcher may adopt during the course of research. A typology of these roles is documented in Table 1 (See Appendix 1). These roles can be located on a continuum that ranges from the analyst role adopting a "hands-off" approach right through to the highly interventionist management-for-hire role. While Gummesson’s typology provides a description of the different consultant roles, it does not indicate the political aspects inherent in intervention levels.

Table 2 (See Appendix 2) indicates that each role can be identified as taking the political position of muse, mediator, or manipulator within a contract research context. Timely and appropriate selection of a particular position may significantly influence access to informants and amount time spent within the research context through enhanced political standing within the organisation.

However, if a researcher chooses a position that he or she is neither equipped nor skilled to handle, the overall positioning of the research project within the organisation will suffer. Ergo, this will be reflected in the data quality. Timely adoption of the right researcher/consultant role may also lessen the effect that stereotypes of both consultants and academic researchers have upon how the organisation perceives the use value of the research. For instance, Gummesson (1991) cites a tendency towards esoterica and non-productive clannishness as a stereotype that plagues academics who attempt consultancy-based research. Similarly, pure consultants are charged with the claim that they are totally consumed by short term interests, thus ignoring the developmental and long term concerns of organisations.

The ascent gets steeper: Negotiating the qualitative research contract

Qualitative research within a business or management context has a history almost as long as qualitative research within a sociological or anthropological context (Linstead 1997). Many early researchers, including Mayo and his infamous Hawthorne studies, used the services of anthropologists, industrial psychologists and sociologists to enhance their organisational studies (Schwartzman 1991). However, this role has not always been formally recognised. Even today where some of the best quality qualitative business research has developed out of consultancy research, a strong bias still exists both within the typical academy and management consultancy against qualitative research (Sutton 1997).

Similarly, many organisations are still wary of research proposals that solely based on the collection and analysis of qualitative data. Concerns about validity, reliability and generalisability are thrown at prospective researchers hoping to gain access to a particularly good research site. One of the main reasons for this is that organisations often expect measurement rather than description of organisational effectiveness and its impact upon service delivery and profitability. In addition to this rather daunting hurdle, other kinds of consultants such as organisational psychologists and business strategy and development consultants are seen to be able to produce positivistic "results" within a relatively short time span, and are therefore viewed as a more cost effective option than an organisational sociologist or corporate anthropologist.

Thus, the academic who hopes to develop a consulting career that runs parallel to his or her more scholarly endeavours needs to consider how to do both without compromising methodological rigour. To do this successfully, the researcher/consultant needs to be actively involved in setting the research agenda. This needs to occur during the design and contract negotiation phase of the project. The following summaries of two case studies describe how rigour can be seriously compromised if the research design and contract negotiation phase is not properly managed. Inadequate handling of this component of the research process can lead to poor quality data, analysis and research outcome.

Case I: Industrial Ethnography: Technological Innovation, Change Management and the Politics of Fieldwork

This case describes a study of the politics of technological innovation at one industrial plant within a major Australian multinational manufacturing firm. The project centred around an experimental trial of an advanced manufacturing technology. A group of Australian academics from a regional university had been contracted by the firm to develop a change management strategy which would reduce the incidence of user resistance to the trial, and hence, implementation of the new technology. The project team consisted of several sociologists whose main area of interest was the sociology and philosophy of science and technology, two industrial psychologists whose main task was to facilitate the change process at the shopfloor level, and several doctoral students with backgrounds in engineering and management. The official aim of the team was to access appropriate data in order to fulfil the requirements of the contract. Unofficially, however, disciplinary and professional differences meant that the unofficial aim of the project for the academic team differed. Hence, there was considerable debate about what the team was actually doing within the organisation and what the eventual outcomes would be.

The manufacturing firm under investigation had a long and history of failing to integrate new and advanced systems into the manufacturing process. This high rate of failure was not only costly, it also continually reduced the status of the company’s R&D department. Thus, when members of the R&D department arrived at a site, the majority of the shopfloor workers, technicians, and middle managers often expressed a high level of cynicism about their ability to facilitate the implementation of technological innovation.

However, in 1996 a risk review workshop was held between R&D and senior executives about changes to the early stages of technology projects. They identified a number of human factors that may have impeded the successful completion of the AMT trial. It was after identification of these factors that the current group of academics were approached to address the human factor issues in technological innovation within an existing manufacturing site located in another city.

The negotiations between the academics as consultants and the firm lasted over eighteen months before a final contract was settled. One of the main features of this contract was that the academic group were to produce a number of deliverables within a specified amount of time. These deliverables included workshops, informal reports to R&D, and formal reports to senior management responsible for organisational change and development. However, despite a contract, the specific nature of these deliverables remained unclear for quite some time, and protracted discussions about seemingly basic but highly political aspects of the study continued. For example, the leader of the technology team rejected the use of the term "sociotechnical" in favour of the more benign human factors". In terms of data analysis, this language difference is pivotal.

From the point of view of traditional ethnographic research, the workshops provided a unique challenge. On the one hand, participants saw all academics that attended the workshops as part of the change process. Despite attempts to convince them otherwise, company participants and technology developers had very little understanding of the ethnographic research process outside the change management process. At least one academic and one postgraduate student who were conducting ethnographic observations were cast as "observers" and not researchers who had a definite role to play within the project. In this instance, the qualitative data collection process was, for the most part, covert.

In a sense then, two studies were being conducted simultaneously. One study, which centred around production of the deliverables, involved the implementation of two different sociotechnical "expert" approaches. Although it could argued that these approaches adopted could be located within an action science paradigm, informal interviews with the facilitators of these approaches indicated that their adherence to the expert nature of their processes excluded the possibility of either of them being located within this paradigm.

Therefore, several issues arose which stymied a rigorous study of the politics of technological innovation within this particular site. First, data collection outside this expert focus was considered of secondary importance by both the academic project leader and the technology group. Second, access by a single researcher to the site was extremely difficult to organise. The site was at least three hours drive from the University. Funding had not been allocated for the "basics" of ethnographic fieldwork that requires the researcher to spend days at a time away from home base. The "basics" here included taping equipment, access to cheap accommodation, "hot" desking at a nearby university, and safe travel arrangements. Instead, the role of the ethnographer was to be "covert" in the sense that she was to be introduced as an assistant to the academic team, rather than as a full-time researcher.

This would have placed extensive limitations on the ability of the ethnographer to fully access appropriate data. The site itself proved also to be an obstacle. Physical entry to the site was difficult. Security arrangements were oppressive and the researchers were subject to some form of surveillance for most of the time. Special clothing and protective apparatus had to be worn at all times. Researchers were to be never left alone or given access to areas deemed unsafe. As there were no women employed on the site, private facilities for women were initially non-existent. This also meant that many of the company participants at the workshops viewed the female researchers as pleasant diversions from a tedious work routine rather than serious scholars conducting important research.

Third, the project leader, while obviously captured by the power of ethnographic research, had neither the will nor the understanding to enable the proper climate for ethnographic research within this particular research site to develop. One of the main reasons for this was that the continuation of contract funding was dependent upon the performance of the academic research team as change agents rather than as analysts or distant observers. In other words, the contractual obligations put in place were not conducive to conducting rigorous qualitative research that required a certain level of distance and objectivity on the part of the fieldworker. The project leader, when pressured by senior management about the quality of deliverables, saw the likelihood of a team member being "let loose" for a considerable length of time at a manufacturing site as too politically risky. In the end he could not reconcile the paradigmatic and practical differences between the reality of traditional ethnography and the political demands of quasi action science.

Thus, the ethnographic component of the study was scrapped and replaced by a variety of qualitative methods, including historical analysis of company documentation and single interviews with the technology and manufacturing site team members. Although academic team members continued to collect qualitative data, their main complaint was that a grounded theory approach was almost impossible to achieve. This was somewhat influenced by lack of access to appropriate data. Thus, an opportunity to produce several very good quality, rigorous ethnographies on the implementation of technological innovation within the Australian manufacturing industry was considerably reduced.

Case 2: What worked, what didn’t: Case studies in government service delivery and problems of rigour

This research project was sponsored by the Queensland chapter of a national research and advocacy public administration institute. This organisation acted as a broker for the seven public sector agencies that participated in the study. The main aim of the project was to document innovative changes to service delivery within the Queensland public sector. This project was entirely funded through contract research funds over a period of four months. During that time, the research team was required to produce several deliverables, the main one being a 20 000 word report documenting the findings of nine case studies. The research team consisted of four full-time political science and public sector management academics and two research assistants. One of these assistants worked on the project on a full-time basis. Apart from the project leader, the three academics allocated less than one day a week to the project over the time span of the study.

The project team won the tender to conduct the research from two other bids. After the initial contract was awarded, several significant changes to the funding levels, number of case study participants and project team membership occured. Initially, four state government departments and one local authority were to be the only participants in the study. However, a week after the contract was awarded two other state government departments joined the project. In total, nine case studies of various approaches to changes to service delivery in the Queensland public sector were completed and included in the final report. In addition to this, a fourth academic joined the team.

Funding of the project was tied to the production of four deliverables. The final date for the completion of the major deliverable, the report, was early August, 1998. From early April until late July, 86 public sector senior executive and middle managers, senior union officials, industry and community representatives and Directors-General were interviewed. With the exception of two interviews, all interviews were conducted by at least two members of the team.

As the timeframe was extremely tight, the project leader issued the following guidelines for interviews:

• No taped interviews
• No more than six interviews per case study, including non-public sector interviewees
• Preference for interviewing elites within each organisation
• Use a semi-structured format
• Refuse to let political climate interfere with the use of the semi structured questionnaire guide
• At least two project members must be present at each interview
• The paradigmatic and ideological framework of the chief investigator must prevail at all times
• During the analysis phase, adhere strictly to the pre-coded categories developed by the sponsor

For the most part, access to public servants and relevant documentation was forthcoming. In some ways the project was awash with data. However, gaining access to appropriate data was less successful than it might have been. Restrictions to access included stonewalling by several departments, the cancellation of interviews at the last minute, excessive use of jargon and insider language, an attitude of mistrust towards academics from schools of management and commerce doing qualitative research and not adopting an expert stance, raised expectations with regard to quality and outcome of the final document, blancmange or vanilla interviews, no time to conduct rigorous analysis, and conducting a research project on a highly sensitive topic during an election campaign.

In addition, several major policy decisions made by the incumbent State government had effectively stymied any serious attempts to privatise or commercialise government service delivery in Queensland. This last point set the tone for many interviews and, given the time frame, it was difficult to get beyond this point. Thus, the project team ended up being in the position of having too much poor quality "official" data and not nearly enough good quality "unofficial" data.

The structure and management of the research team also added to the problem of poor and inadequate analysis. At no stage in the project history did all members of the research team meet together to discuss the project as a whole. The full-time research assistant was expected to attend all interviews and contribute to the analysis and write-up of all nine case studies, as well the initial literature reviews of both the local and international context. Political and ideological differences began to emerge between the research assistant and at least one team member. This affected the assistant’s ability to conduct proper data analysis, because she had to locate all the analysis within one ideological framework.

What initially appeared to be an exercise in grounded theory methodology actually turned out to be a procrustean exercise. The data was discarded and trimmed to "fit" an anti-public choice theoretical stance. However, because the research assistant was in the best position to give an overall view of what the data was saying, her own analysis did not fit that of the other members of the team. This meant that the executive summary and conclusion reflected a superficial and biased reading of the data.

Three issues emerged within this study that seriously affected the quality of the final deliverable. The first one is concerned with the research design phase. During the tendering process and subsequent negotiating process, the research team leader was explicit about utilising a qualitative approach. The data collection process that the team planned to adopt was clearly outlined in the tender document. During the tender interview and presentation one member of the selection panel made several inquiries about issues relating to validity and reliability. He also inquired about how the team was going to access financial data and subsequently analyse that data which some see as crucial to making any generalisations or assumptions about local conditions. Given the nature of this research, some quantitative analysis was a necessary stage in the later stages of the project. Access to financial data other than that available in the public domain, however, was not forthcoming. Therefore, the financial analysis needed to confirm several of the generalisations made in the conclusion did not happen.

Second, insufficient time was spent with the brokerage agency negotiating a reasonable timeframe in which to complete the research. As there was only one person working full-time on this project, other team members’ involvement ebbed and flowed. Several factors were not considered when the original proposal was developed. The availability of all four academic team members was restricted considerably due to overseas conference travel and academic service commitments. An election campaign was called half way through the project.

While this was surprisingly less of a problem than it might have been during the campaign itself, it did restrict access to public servants during the ten days it took for a new government to be formed. At least one participating agency underwent significant change immediately the new minister was appointed. This change included a restructuring of the positions which had been previously occupied by key informants within that agency. At least one participating agency went into a very defensive mode of operation and "changed" their original suggestions made to the researchers. This included a request that the case study be "rewritten" to reflect these changes. Another agency commenced a literal dismantling of their structure in preparation for a total absorption into a different "parent" agency.

While these changes in themselves did not affect the overall data analysis process, they did slow down the whole process of meeting the deadline for the completion of the final deliverable. The time available for systematic and thorough data analysis was virtually non-existent. However, even if such data analysis had occured, the poor communication between research team members meant that much of this analysis would have been shelved because it did not support the prevailing ideological viewpoint of the chief investigator.

Third, the imposition of an analytical framework by the brokerage agency severely hampered a "grounded" approach. One of the major findings that did not figure in the final report but which was "grounded’ in the data concerned the "business" skills levels of senior executives and senior officers within the Queensland public sector. In every case study there were clear examples of how senior executives lacked the skills to design and implement changes to service delivery that involved commercial or non-public sector practices. Many of the senior executives interviewed were quite comfortable with a return to traditional public administration principles and practices, and were obviously nervous about the demands that practices of transparency and accountability placed upon them. This finding did not "fit" into the categories of "drivers" which structured the analysis of the report.

Discussion

Both case studies demonstrate a number of factors that effected the quality of both the data analysis and the final research outcome. These include the following:

• Inflexible agenda set by the sponsor
• Pre-coding by the sponsor
• Time constraints
• Intellectual property issues Who owns the data
• Pro-positivist approach
• Inadequate understanding of the research process

• Unrealistic expectations of the research outcome

• Excessive vetoing or vetting of final "deliverable" or research outcome
• Poor collaborative research practices

While many of these factors can be minimised or eliminated, the reality is that qualitative research does not fit neatly into the consulting framework. Data collection is time consuming, analysis can be labour intensive, and the language of the qualitative researcher may differ from that of the typical consultant.

Despite the problematic nature of these two cases discussed here, significant success has been achieved by qualitative researchers conducting management consultancies. Linstead (1997) and Bate (1997) present a convincing case for the increased use of fieldwork methods in academic management research, which he sees as providing the researcher with good opportunities to engage in grounded theory in a management or business context. More importantly, he argues that companies can be convinced of the value of corporate ethnography.

Conclusion

In terms of maintain a high level of rigour, the case studies presented here indicate that there are three phases of the qualitative research process that are important. First, Morse (1994) cites the quality of the research design phase as an important contributor to maintaining rigour. A major part of this process is the selection of an appropriate research strategy. In the case of contract research, time limitations mean that fieldwork is often condensed.

Second, the contract negotiation phase can determine the degree to which a project is methodologically sound and rigorous. Anticipation of possible obstacles and negotiating a reasonable timeframe are both essential to producing quality outcomes for the contract project and for any scholarly work that may result from the data. Allocation of enough time for rigorous analysis is central to the production of high quality deliverables. In these days of user friendly data analysis software packages, data analysis can commence from day one of data collection. Further development of voice recognition software and hand held computers will also assist in diminishing the problem of inadequate time for data analysis.

Finally, if the work of qualitative contract researchers is to be appropriately recognised within both academic and industry circles, they need to make good "presentational" choices (Fine 1993). Such choices not only involve how a final deliverable or book looks and reads, but also the quality of the formal and informal feedback they provide to sponsors and clients.

References

Atkinson, P. and Hammersley, M.(1994) Ethnography and Participant Observation in Lincoln and Guba (eds) The Handbook of Qualitative Research, Newbury Park, Sage.

Bate, P. (1997) Whatever happened to organisational anthropology?, Human Relations.

Becker, H. (1996) The Epistemology of Qualitative Research. http://weber.u.washington.edu/hbecker/qa.html

Fine, G.A. (1993) Ten Lies of Ethnography: Moral Dilemmas of Field Research, Journal of Contemporary Ethnography, 22, 3: 267-294.

Gummesson, E. (1991) Qualitative Methods in Management Research, Newbury Park, Sage,

Linstead, S. (1997) The Social Anthropology of Management. British Journal of Management, 8:85-98.

Lyon, E. (1997) Applying Ethnography, Journal of Contemporary Ethnography, 26,1:3-27.

Morse, J. (1994) Designing Funded Qualitative Research. In Lincoln and Guba (eds) The Handbook of Qualitative Research, Newbury Park, Sage.

Schwartzmann, H. (1993) Ethnography in Organizations, Newbury Park, CA, Sage.

Sutton, R. (1997) The Virtues of Closet Qualitative Research. Organization Science, 8,1:97-106.

TABLE 1

CHARACTERISATION OF RESEARCHER/CONSULTANT ROLES

APPENDIX 2

TABLE 2

POLITICAL POSITIONS OF RESEARCHER/CONSULTANTS