Consistency and inconsistency in PhD thesis examination.(Report)

This is a mixed methods investigation of consistency in PhD examination. At its core is the quantification of the content and conceptual analysis of examiner reports for 804 Australian theses. First, the level of consistency between what examiners say in their reports and the recommendation they provide for a thesis is explored, followed by an examination of the degree of discrepancy between examiner recommendations and university committee decisions on the theses. Two groups of discrepant recommendations are identified and analysed in depth. Finally the main sources of inconsistency are identified. It was found that the comments of a small minority of examiners were inconsistent with each other or with the committee decision in a significant way. Much more commonly the texts of examiner reports were highly consistent and were closely reflected in the final committee decision.

Keywords

doctoral examination

higher education

examiner consistency

graduate education

thesis assessment

Introduction

When a PhD thesis is examined, whatever the process, examiners typically require something more from the student, whether it be the correction of typographical errors in the written document or more substantial changes. It is also in the nature of doctoral candidature that there is the expectation that the new researcher can always learn more, and that the thesis is but a step on this path (Mullins & Kiley, 2002). As one commentator notes it is an 'apprenticeship in the art of discovery' (Kwiram 2006, p. 141).This explains why most examiners provide some comment that is instructive or formative at the final point of a thesis examination (Holbrook, Bourke, Lovat & Dally, 2004a; Tinkler & Jackson, 2004). Furthermore, there is evidence that the type, extent and tone of this comment reflect the overall evaluation of the qualities of the thesis (Bourke, Hattie & Anderson, 2004). This paper focuses on the standards that examiners apply and how consistently they apply them.

The literature on examination and doctoral thesis quality has mostly emerged from the UK and Australia and on the whole indicates that, except in all but general terms (Morley et al., 2002; Shaw & Green, 2002), doctoral 'qualities' and 'standards' have proved very difficult to 'tie down' (Tinkler & Jackson, 2004, p. 8). In the past when there were relatively few candidates destined for scholarly pursuits, thesis quality was not a public or pressing issue, as evidenced by the lack of research and publication on the issue.With the rapid 'massification' of doctoral degrees and the growing need for cross-disciplinary work to tackle emerging social and scientific problems, there has come the realisation that not only is there an absence of benchmarks, but an absence of information about the degree and its evaluation (Morley et al., 2002; Shaw & Green, 2002; Jackson & Tinkler, 2001; Tinkler & Jackson, 2004; Lovitts, 2007). A heavy reliance in Australia and elsewhere is placed on the collective experience of examiners and knowledge of 'peer review' (Hoddell et al., 2002 p. 64; Grabbe, 2003 p. 130; Tinkler & Jackson, 2004 p.119; Holbrook, Bourke, Fairbairn & Lovat, 2007). But what is the correspondence between peer review and thesis examination? Evidence is also emerging from the USA that few students publish in peer-reviewed journals (Nettles & Millett, 2006). For most students, thesis examination constitutes the only instance of independent scholarly review of their complete project.

While supervisors and examiners play a pivotal role in defining and shaping the practices in their disciplines, including how and what candidates need to learn to be successful, there is very little in the literature that explores the connection between expectation, judgement and outcome (Mullins & Kiley 2002; Denicolo, 2003; Powell & McCauley, 2002, 2003). The lack of a formally articulated 'curriculum' in relation to assessment (Gilbert, 2004) prompts concerns about consistency in procedures and judgements (Sloboda & Newstead, 1997; Tinkler & Jackson, 2000; Morley et al., 2002; Shaw & Green, 2002; Lawson et al., 2003; Denicolo, 2003). Although procedures for adjudication where examiners differ have been described (Tinkler & Jackson, 2004; Lawson et al., 2003), until the study on which this paper is based, there has been no sustained analysis of examiner recommendations on the same thesis and the relationships between examiner recommendations and the official committee decision.

Some of the most powerful mythology in doctoral examination derives from stories of substantial differences in judgement between examiners of the same doctoral thesis (Morley et al., 2002). High levels of inconsistency would cast doubt on the reliability and quality of doctoral assessment and, in this current international climate of quality assurance and research quality assessment, there is intensified interest in the outcomes of doctoral research. So, are examiners frequently inconsistent in their judgements as anecdote would have us believe?

The paper deals with three questions:

* How consistent are PhD examiner comments and recommendations across institutions and disciplines?

* If inconsistency is evident, what forms does it take?

* What aspects of the written examiner reports influence the subsequent decision of the committee when it is faced with conflicting recommendations on a thesis?

Most Australian universities (58 per cent) use three external examiners, others use two; the process does not normally involve a viva voce or oral examination. Almost half of Australian doctoral examiners are drawn from other countries, and the majority of these are located in the USA and the UK. This information has been published elsewhere together with other details related to supervision, and other individual information such as candidate and examiner gender (Bourke et al., 2007a).

In this paper the level of consistency between what examiners say in their reports and the recommendation they provide for a thesis are explored, followed by an examination of the …