4 2007 POSTGRADUATE CONFERENCE Paradigm-Independent Meta-Criteria for Social & Behavioural Research Professor Ray W. Cooksey (School of Business, Economics & Public Policy) Abstract In this paper, I flesh out a network of inter-related meta-criteria for evaluating the quality, coherence and value of social and behavioural science research, independent of the paradigm(s) guiding the research. The network emerges, in part, from principles of complexity science and considerations of multimethodology. I argue that the ultimate test of research quality and contribution lies in its power to convince those who stand to be influenced by it. ‘Convincingness’ emerges from the confluence of the research act and the telling of the story about that act and therefore forms the central meta-criterion to be satisfied. Judgments of ‘Convincingness’ are influenced by a set of 12 inter-connected meta-criteria, each targeting an aspect of research act or story. The network is displayed in both matrix and mindmap formats to assist researchers and reviewers in its application. The network is explicitly designed to move researchers’ thinking beyond the boundaries of specific research traditions or paradigms and their localised assumptions and definitions of research ‘validity’. I argue that focused consideration of the meta-criteria can greatly assist both the planning and the evaluation of social and behavioural science research. In the social and behavioural sciences, there is a continuing debate about the criteria one should use to judge the research quality, impact and contribution. The crux of this debate has centred on the different meanings held for the criteria of ‘validity’ and ‘generalisability’ within various research traditions or paradigms (see, for example, the discussions in Beer 1993, Crotty 1998 and Thomas 2006). The debate has yet to arrive at a consensus view. Furthermore, in emerging conceptualisations of social and behavioural research in terms of multiple paradigms, multimethodology and triangulation (see, for example, Brewer & Hunter 2006, and Onwuegbuzie & Teddlie 2003), lines of distinction are becoming increasingly blurred. Meanings for ‘internal validity’ and ‘external validity’, two of the dominant criteria in the positivistic or ‘normative’ paradigm, have been borrowed, distorted and recast to fit different expectations and paradigm assumptions. LeCompte and Goetz (1982) and Healy and Perry (2000), for example, demonstrated this reshaping process when they generated meanings for internal and external validity to fit the contexts of interpretive ethnographic research and qualitative marketing approaches within the realism paradigm. Cooksey (2001) showed that it was beneficial to apply complexity science considerations and principles to social and behavioural science research. Research, from a complexity science perspective, is an emergent activity evolving from the dynamic and contextualised intersection of the researcher and the researched in the context of one or more sets of guiding assumptions (i.e., paradigms). This encourages a 'multi-' mindset for research: multidisciplinary, multi-paradigm, multi-methodology (see Cresswell & Plano-Clark 2007 and Mingers & Gill 1997). An important question that emerges from this complexity perspective is how should the quality, coherence and value of research evaluated and judged? Criteria typically applied within one 5 UNIVERSITY OF NEW ENGLAND specific research tradition or paradigm cannot and should not be expected to hold or even to be meaningful in another (Thomas 2006). What is required is a set of meta-criteria that can be used to evaluate the quality, coherence and value of social and behavioural science research, irrespective of the particular guiding paradigm(s) and in light of the attendant complexities. The Inherent Complexity of Judgments about Research Quality There have been relatively few attempts to establish a unified set of coherent criteria that could be used to assess the quality of research independently of the particular paradigm assumptions underpinning the research while being sensitive to the contexts in which that research is situated and produced. Most discussions of research quality criteria tend to be paradigm-bound, involving concepts such as construct validity, statistical conclusion validity, internal and external validity, generalisability, situatedness, objectivity and subjectivity, transparency trustworthiness, authenticity, that are difficult to meaningfully navigate and apply across paradigm boundaries (see, for example, Campbell & Stanley, 1966; Guba & Lincoln 2005; Neuman 2006). This creates difficulties for researchers wishing to work across paradigms using mixed methods and approaches as well as for consumers or reviewers of research who may be tempted to apply criteria from within their own paradigm to make potentially ill-founded judgments about the quality of research arising from within other paradigms (e.g., Armstrong 1997; Hojat, Gonnella & Caelleigh 2003). A number of researchers have attempted to develop more general criteria for judging research quality either in the context of mixed methods research (e.g., see Brannen 1992; Cresswell & Plano Clark 2007; Greene, Kreider & Maye 2005; Johnson & Onwuegbuzie 2004; Onwuegbuzie & Teddlie 2003 and Sale & Brazil 2004) or by extending the ‘reach’ of initially positivistic criteria (such as reliability and validity) to more qualitative research domains (e.g., see Golafshani 2003; Healy & Perry 2000; LeCompte & Goetz 1982; Seale 1999). However, these efforts are constrained in important ways. For example, criteria established for evaluating mixed-method research are generally not sufficiently holistic and they tend to ignore or downplay the impact of context (both of the researcher and of the participants) on the research. The framework developed by Johnson and Onwuegbuzie (2004) rests almost entirely on pragmatic assumptions focusing on strengths and weaknesses rather than specific judgment criteria. Onwuegbuzie and Teddlie (2003) focused on ‘inference quality’ but again stopped short of specifying appropriate judgment criteria. Moves to extend the reach of positivistic criteria to qualitative research seek are inherently flawed simply because they cannot shake the historical ‘baggage’ that accompanies the original meanings of the criteria, leading to a ‘Procrustean bed’ approach that forces a degree of fit to new paradigms and data types. Krathwohl (1985, 1998) was arguably the first to develop and implement an integrated framework for educational researchers purporting to offer a more general system of criteria that could be applied to judging research quality across paradigm boundaries. Krathwohl’s system of criteria included sensitivity to contextual constraints on the research, including institutional constraints, resource limitations and ethical standards as well as criteria such as resource allocation, information yield and audience credibility. However, Krathwohl was limited by his own paradigmatic ‘baggage’. Two of his central criteria, ‘Linking Power’ and ‘Generalising Power’, failed to satisfy his cross-paradigm intent when he equated them, respectively, with the positivistic concepts of ‘internal validity’ and ‘external validity’. Furthermore, he reified the ultimate positivistic goal of establishing cause and effect relationships through the sub-criteria 6 2007 POSTGRADUATE CONFERENCE he associated with ‘Linking Power’. This, in effect, negates the potential meaningfulness of such criteria to non-positivist paradigms and assumptions. The Meta-Criteria My goal here is to set out a network of criteria by which one may genuinely judge the quality, coherence and value of any type of social and behavioural science research, irrespective of paradigmatic assumptions and alignments but not in ignorance of them (i.e., making such assumptions and alignments clear will certainly be an expectation of ‘quality’ research). I call these criteria ‘meta-criteria’ to clearly signal that they are intended to operate above and beyond any local paradigm-specific quality criteria. I drew upon a range of sources (e.g., Healy & Perry 2000; Krathwohl 1985; Neuman 2006; Onwuegbuzie and Teddlie 2003; Sale & Brazil 2004; Thomas 2006) to develop the meta-criteria and was guided by the complexity science implications set out in Cooksey (2001). The meta-criteria are summarised in Table 1; each associated with a set of trigger questions, prompts and meanings. At the base of the framework/network of meta-criteria is the assumption that research is fundamentally a ‘storytelling’ activity and that research quality is reflected in the impacts of that story on the consuming audience or readership. The research act itself is all about conceiving, constructing and displaying all of the necessary components of that story. Table 1 therefore provides a useful tool for querying the research process from conception to presentation and beyond (i.e., longer-term impacts). As shown in Table 1, the central meta-criterion, Convincingness, is an emergent meta-criterion, heavily dependent upon the foundations, design, execution, analyses and closure of the research and influenced by and, in most cases attached to, a specific presentation of the research story (proposal, report, conference paper, published article). Convincingness is central in the framework simply because judgments about research quality, coherence and value fundamentally rest upon whether or not particular consumers of the research story are convinced by the totality of the effort. Telling a convincing research story then becomes a matter of attending to whatever issues add weight and value to the story while at the same time satisfactorily addressing any concerns or emergent issues may detract from the story. 7 UNIVERSITY OF NEW ENGLAND Table 1. Elaboration of trigger questions, prompts and meanings associated with each of the 12 inter-related meta-criteria in the context of the central meta-criterion: Convincingness. Convincingness of the Research The Central Meta-Criterion Is the research story convincing with respect to the arguments being made? Are the arguments and conclusions in the research story convincing to the consuming audience/readership, given the assumptions held, constraints experienced, research questions, objectives and aims established, roles adopted, trade-offs made, obstacles confronted, methods chosen, data gathered, analyses employed, unanticipated outcomes and setbacks experienced, results presented and conclusions/implications drawn? Juxtapositioning with Other Research What have others done & where does this research fit in? Does the research story converge on or diverge from other stories in literature in the same or similar domain(s)/discipline(s) and is this case made clear in the story’s arguments? Does the researcher focus, in a balanced way, on content and methods issues? Does the researcher, where appropriate to paradigm expectations, make effective use of other research to set the stage and problem context for the research? Does the story effectively build upon and link to the literature and what has been done before and/or are divergences from or gaps observed within the literature clearly argued for? Researcher Positioning Where does the researcher fit within the research context? Is the position of the researcher in the context of the research clearly evident? Is a critical stance or attitude being espoused or enacted? Is a specific value position or set of assumptions being reflected? Are the researcher’s expectations and motivations clear? Are there any opportunities/constraints associated with the researcher’s positioning? Does the researcher enjoy any specific or special role(s) within the research context and have the implications of those roles been spelled out? Positioning of Participants & Other Data Sources Where does everyone or everything else fit within the research context? Is the position of the participants or other data sources in the context of the research clearly evident? What value is attached to the voice of the participant being sought, reflected, heard and/or understood? What assumptions about the participants and/or other data sources are being reflected? How is the relationship between the researcher and participants and/or other data sources conceptualised and/or managed (including ethical constraints and expectations)? How are relevant participants and/or other data sources identified, selected and brought into the research context? If a data source is non-human (e.g., documents, web sites), what is their status and context of production? 8 2007 POSTGRADUATE CONFERENCE Contextual Sensitivity How well does the researcher use knowledge about context to add richness and/or qualification to their research process and the resulting findings? Is the research conducted in a manner that reflects appropriate sensitivity to the substance and dynamics of the context in which it is conducted? Are there contextual issues that should qualify, constrain or amplify what has been learned and how it has been learned? Internal Coherence Does the research, as a whole, hang together as a coherent process to permit the conclusions the researcher seeks or claims? Does the internal organisation of the research exhibit logic and methodological/procedural coherence? Does the research hang together as a systematic and/or systemic approach to a problem or issue? Are epistemology and methodology consistent? Does the research satisfy or adequately address the normal criteria for judging research quality held within its own paradigmatic stance and epistemological assumptions? If causality in conclusions is sought, has the case been made within the context of the intent, design and execution of the research? Has necessary and sufficient detail for evaluating the study, its conduct and conclusions been presented? Has necessary and sufficient methodological detail for potentially reproducing the study been presented? Are methodological choices and implementations recorded/displayed in a sufficiently transparent way? Table 1 (concluded). Elaboration of trigger questions, prompts and meanings associated with each of the 12 inter-related meta-criteria in the context of the central meta-criterion: Convincingness. Analytical Integrity Do the analyses of data lead to or support appropriate, defensible & clear conclusions, given the quality of the data to hand? Do the analytical strategies employed make logical as well as practical sense and do they provide defensible, perhaps even innovative, pathways to conclusions and implications? Has the approach to and execution of the analyses been appropriately managed and transparently recorded/displayed so that potential biases or counter-claims can be ruled out? Extensional Reasoning Do the research findings have meanings or implications for other contexts? Does the research produce findings, implications or other outcomes that can be arguably and/or reasonably applied in or extended to other contexts (e.g., problems, locations, times, samples, participants, cultures) and has this been clearly argued? Is speculation clearly and appropriately signposted and managed? 9 UNIVERSITY OF NEW ENGLAND Value for Learning What can people take away from the research as important messages? Does the study add value for learning about a phenomenon for others and is it clear who those others might be? Is our understanding advanced or uncertainty reduced? Are signposts/directions for possible change provided? Is innovation evidenced? Are the applicability and implications for theory, method and/or practice clearly signalled? Are guides to future practical and contextual actions clearly set out? Fertilisation of Ideas Can others run with or build on what has been shown by or learned from the research? Does the research suggest/stimulate/facilitate non-trivial follow-on research ideas or is it likely to? Can others easily build on the research? Does the researcher demonstrate explicit awareness of the possibilities for further research which are suggested by their study? Is the research cited by/used by others and/or are the ideas applied by others (these are reflections of research impact and they may be delayed until research can be or has been presented in publicly presentable form)? Does the research provide guidance to other researchers as to things to pursue and/or things to avoid in future research? Handling of Unexpected Outcomes How well has the researcher dealt with surprises and unanticipated findings? Are unexpected/surprising/counter-intuitive outcomes in the research handled in a logical, consistent, defensible and unapologetic manner? Are these unexpected outcomes linked to possible new directions and ideas for future research? Are these unexpected outcomes linked to possible defects in my theorising, conceptualisation, implementation and/or analytical approach? Acknowledgement of Limitations What constrains the learning value and applicability of the research? Does the researcher explicitly acknowledge the limitations that accompany their study and indicate how they might be overcome in further research? Does the researcher demonstrate appropriate care in conditioning their conclusions on these acknowledged limitations? Has the research explicitly addressed the existence of weaknesses in the research with respect to one or more of these meta-criteria? Presentational Character Is the presentation suitable for the intended audience(s) (i.e., is the researcher hitting the mark)? Does the researcher demonstrate clear awareness of audience expectations, assumptions and prior knowledge with respect to the title, format, structure content and level of detail in their chosen mode for presenting their research story? Does the chosen mode of presentation employ text, narrative, illustrations and/or other devices in a well-balanced manner designed to make the research story clear and easy to understand, without undue redundancy or excessive/unnecessary detail? Are there clear logical threads running through the presentation from start to finish? Is the balance in emphasis on various meta-criteria appropriate given anticipated audience expectations? Are analytical and data summary/display choices made to best effect for given purposes and audiences? 10 2007 POSTGRADUATE CONFERENCE The 12 supporting meta-criteria are inter-related and at least partially inter-dependent upon each other in producing the most convincing research story for given purposes. Since one should not expect the meta-criteria to be of equal importance or relevance in all research investigations, an important aspect of judgments of Convincingness is the achievement of an appropriate weighting or balance amongst the meta-criteria, given the contexts and intentions of the research. Juxtapositioning with Other Research captures the central role played by the literature in any research investigation. A convincing story will show where the study is situated with respect to previous research, of all relevant types. This situatedness may be explored prior to the design and execution of the study or as a consequence of it (as in the case of some forms of grounded theory) or both, depending upon specific paradigm and methodological assumptions being made. In many cases, what will add to Convincingness will be the clear identification of an emergent niche for the research, a novel twist or a gap to be filled. Researcher Positioning highlights the importance of knowing where and how the researcher is situated in the research context. A research story that clearly conveys the researcher’s context, assumptions and values held, constraints encountered and opportunities available makes for a more credible, real and therefore convincing account. From the researcher’s perspective, this meta-criterion forces a critical reflection on one’s own role in the research context; a role that may have to be negotiated with ‘gatekeepers’ controlling access to research contexts and, where relevant, with significant others, such as higher degree supervisors, research team members and/or institutional partners. Positioning of Participants & Other Data Sources highlights the importance of knowing where and how the participants are situated in the research context. A research story that clearly conveys the participants’ contexts, expectations, and concerns makes for a more credible, real and therefore convincing account. From the researcher’s perspective, this meta-criterion forces a critical reflection on the power dynamics that may exist between researcher and participants in the research context and the implications they might have for the quality and value of the research itself. This meta-criterion reinforces the idea that participants also have expectations about the research and this should be acknowledged (thus giving a voice to the participants). As a consequence, this meta-criterion has important linkages with the Contextual Sensitivity meta-criterion, to be described below. It should be noted that non-human data sources and records (such as films, newspaper articles, company documents, research articles for meta-analysis) are intended to be encompassed by this meta-criterion. A convincing research story would clearly spell out the contexts in which these data sources have emerged and provide indications of their credibility and contextual importance. One final point to make is that this meta-criterion also encompasses sampling issues associated with the selection and/or recruitment of organisations, participants and other data sources for inclusion in the research. Contextual Sensitivity addresses the degree to which the research, its findings and conclusions are appropriately sensitised to local contextual conditions. Such sensitivity may help more richly characterise or qualify the design, execution, data and conclusions associated with decisions made by the researcher as well as by potential gatekeepers in the research context. Some of these contextual factors may have been out of the control of the researcher and a convincing research story should not shy away from spelling out the perceived potential impact of such factors. 11 UNIVERSITY OF NEW ENGLAND Internal Coherence encompasses a broad focus on how well the research works as an integrated project designed to address specific research hunches, ideas, questions, predictions, anticipations or hypotheses. An important aspect of this meta-criterion is whether or not the research is designed and executed in a manner congruent with the specific paradigmatic assumptions and expectations espoused by the researcher. Another important aspect is how well the design and execution of the research, from start to finish, functions to inform and support the inferences, conclusions and implications that the researcher is wanting to draw or claim. A convincing study, against this meta-criterion, should display clear arguments showing that the conclusions presented have the strongest warrant from a range of possible alternative accounts, given the quality of its positioning, design and execution. Judgments about the Internal Coherence meta-criterion would be heavily dependent upon the Presentational Character (situated in the details in the story told about the positioning, design and execution of the study) and Analytical Integrity (congruence between analytical choices and the positioning, design and execution features of the study) meta-criteria as well as upon the Contextual Sensitivity meta-criterion (influences in the research context that may have added to or detracted from the intended positioning, design and execution of the study). Analytical Integrity focuses on the quality and execution of the researcher’s strategic choices of analytical approaches as pathways for making sense of the data that are gathered. Research that is convincing should show how and why analytical choices were made, argue for their appropriateness/suitability to the task at hand and sensitise what is learned against the quality and character of the available data. This meta-criterion will often interact with the Presentational Character and Extensional Reasoning meta-criteria. The former because analytical choices (particularly choices in how best to display outcomes) should be sensitive to audience expectations and anticipated competencies; the latter because analytical choices may enhance or inhibit capacity to engage in strong extensional reasoning (most obviously relevant, of course, with statistical analyses and permissible generalisations from them, but also relevant to many strategies of qualitative analysis). Extensional Reasoning focuses on the intent and reasoning behind the researcher’s extensions of the findings from the localised research context to other contexts of potential interest, including other participants, organisations, observations, measurements, times, places, cultures. In other words, this meta-criterion establishes the defensible reach of the research findings, given the contexts, constraints, design and execution of the investigation (suggesting an important inter-dependence with the Value for Learning meta-criterion). It is also important to note that Extensional Reasoning implicates the role and character of explanations, speculations, forecasts and ‘backcasts’ that may be employed to help make sense of the research outcomes and account for why things happened the way they did (this last also has implications for the Handling of Unexpected Outcomes and Acknowledgement of Limitations meta-criteria). Value for Learning reflects the idea that convincing research will often have important implications for applications of the findings. Such implications need to be clearly targeted to other researchers, managers or practitioners in terms of highlighting where the value for learning is situated and what the nature of that learning is. Value for learning may be situated in a theoretical context, a methodological context, a practical context, a policy context, some other more specialised context or some combination of these. Value for learning may also carry important signals for development, improvement and change in specific contexts and, in action learning situations, may provoke further considerations of the positioning of both the researcher and 12 2007 POSTGRADUATE CONFERENCE participants (thus potentially influencing/informing these two meta-criteria) as well as the institution itself. Fertilisation of Ideas captures the notion that convincing research should stimulate interest in follow-up investigations by other researchers as well as in future investigations by the same researcher. Convincing research not only illuminates directions for further study, it also guides others away from blind alleys. One aspect of this meta-criterion that is subject to potential delay is whether or not the research story is read (or heard) and cited/used by others, an idea that goes to the heart of what is currently being called research ‘impact’. Handling of Unexpected Outcomes is intended to reflect the idea that convincing research is able to deal sensitively and defensibly (without being defensive) with unintended or unanticipated outcomes from the research. Often, the reasoning that a researcher displays in accounting for, rather than explaining away, such outcomes builds important bridges to other meta-criteria such as Fertilisation of Ideas (e.g., identifying future areas for clarifying research) and Acknowledgement of Limitations (e.g., recognising where limitations in the design and execution of the research or insensitivity to certain contextual matters may have given rise to the unexpected outcomes). Acknowledgement of Limitations signals that convincing research openly sets out and critically reflects on the limitations and difficulties encountered during the design and conduct of the research. Often, such limitations arise from trade-offs made by the researcher in order to render the research investigation feasible within constraints. They may also arise from unanticipated obstacles and problems with procedures or data quality that emerged during the research process, which could not be circumvented. A convincing story should not shrink from critically reflecting upon and transparently and unapologetically accounting for limitations. Meeting this meta-criterion in a strong manner can enhance Convincingness by making the case for the Fertilisation of Ideas meta-criterion stronger as well as providing for clearer and appropriately bounded arguments for the Extensional Reasoning and Value for Learning meta-criteria. Presentational Character focuses on the issue of who has to be convinced by the research and how they should be convinced. This reflects the central role played by the audience for the research story. Nothing will detract more from the Convincingness of a research investigation than a story about that research that targets the wrong audience or that is pitched at the wrong level. Convincingness is strongly influenced by the strategic choices a researcher makes when writing or delivering the story, over and above what the strengths of the design and execution of the study might convey. A strong research project, embedded in a poorly crafted story, will suffer by these presentational problems – in short, problems with Presentational Character may override strong positions on many of the other meta-criteria. Even minor conceptual or mechanical difficulties (choosing the wrong format, the wrong representation for a theory or conceptual framework, wrong tables or statistics to report; grammatical errors; lack of proof-reading) may negatively influence Convincingness, even where the audience has been correctly targeted (e.g., readership/reviewers of a specific journal, examiners for a PhD thesis, colleagues, practitioners). 13 UNIVERSITY OF NEW ENGLAND A Mindmap of the Network of Meta-Criteria Figure 1 provides a second tool, in the form of a network mindmap, to assist researchers’ in the planning and evaluation of their own research as well as the research of others. As well, the mindmap displays the potential interconnections and dynamic interplay between the meta-criteria. In pursuit of Convincingness, the 12 meta-criteria can be conceptually grouped into three logical higher-order criteria clusters: Juxtapositioning with Other Research, Contextual Sensitivity, Researcher Positioning and Positioning of Participants & Other Data Sources all reflect broad issues associated with contextualisation of the research. Contextualisation meta-criteria thus establish the boundary conditions, opportunities, constraints and assumptions surrounding the research itself, the researcher and the data sources as well as situating the current research within the context of previous research. Internal Coherence, Analytical Integrity and Extensional Reasoning all reflect broad issues associated with the realisation of the research. Realisation meta-criteria therefore focus on the necessary steps, methodologies and configurations of procedures and technologies, consistent with paradigmatic assumptions, choices and expectations, employed to assemble, analyse and establish meanings, interpretations and extensions for the information gathered during the research process. Handling of Unexpected Outcomes, Value for Learning, Presentational Character, Fertilisation of Ideas and Acknowledgement of Limitations all reflect broad issues associated with explication of the research. Explication meta-criteria thus focus on how the story of the research is configured and presented and what that story might mean for other researchers, for theory, for methods and for practitioners. The higher-order contextualisation, realisation and explication meta-criteria domains interact and trade-off with each other to build a convincing case for the research and its findings, conclusions and implications. The mindmap visually signals the central need to reflect on the convincingness of the research as well as on the appropriateness of the balance in emphasis and strengths/weaknesses across the meta-criteria. 14 2007 POSTGRADUATE CONFERENCE Figure 1: Mindmap of the network of meta-criteria and its implications 15 UNIVERSITY OF NEW ENGLAND Uses of the Meta-Criteria Network The network mindmap (Figure 1) as well as the trigger questions and prompting notes (Table 1) can be used for several purposes (see Figure 2). Researchers, supervisors and higher degree research students (e.g., masters, professional doctorate, PhD) could use the mindmap as a planning guide to provide an annotated record of assumptions held, choices made, design and procedural logic employed, contextual qualifications and related matters and to help spot potential gaps in Convincingness that emerge during the conduct of the research. As a planning tool, the mindmap can assist in building the case for the quality, coherence and value of the research from its initial stages to finished presentation. It could prove useful for anticipating and recording key aspects of the research journey. Thus, in building toward Convincingness in research planning, the Contextualisation and Realisation criteria clusters are foregrounded in anticipation of how Explication might best unfold. Reviewers/research consumers could employ the mindmap as a guide for judging the quality of a specific research story (consultancy report, conference paper, manuscript for a journal). Use of the meta-criteria for judging research quality could help reviewers avoid (or at least recognise) paradigm confusion, blindness or bias in their evaluations while ensuring that the criteria for judgment remain transparent and accountable. Thus, in gauging Convincingness in research evaluation, the Contextualisation and Realisation criteria clusters become the background against which the Explication is contextualised and judged. Furthermore, meta-criteria could also facilitate higher-order ‘learning’ by helping reviewers maintain an awareness of the importance of diversity in paradigms, choices and approaches to social and behavioural research and of the need to employ criteria sensitive to that diversity – a type of learning Flood and Romm (1996) called ‘triple loop learning’. Figure 2. Implications of the meta-criteria for research planning and evaluation. References Armstrong, J.S. 1997, ‘Peer review for journals: Evidence on quality control, fairness, and innovation’, Science and Engineering Ethics, vol. 3, pp. 63-84. Beer, F.A. 1993, ‘Validities: A political science perspective’, Social Epistemology, vol. 7, no. 1, pp. 85-105. Brewer, J. & Hunter, A. 2006, Foundations of multimethod research: Synthesizing styles, Sage Publications, Thousand Oaks, CA. 16 2007 POSTGRADUATE CONFERENCE Brannen, J. 1992, ‘Combining qualitative and quantitative approaches: An overview’, in Mixing methods: Qualitative and quantitative research, ed J. Brannen, Avebury, Aldershot, UK, pp. 3-37. Campbell, D.T. & Stanley, J.C. 1966, Experimental and Quasi-experimental Designs for Research, Houghton Mifflin, Boston, MA. Cooksey, R.W. 2001, ‘What is complexity science? A contextually grounded tapestry of systemic dynamism, paradigm diversity, theoretical eclecticism and organizational learning’, Emergence, vol. 3, no. 1, 77-103. Cresswell, J.W. & Plano Clark, V.L. 2007, Designing and Conducting Mixed Methods Research, Sage Publications, Thousand Oaks, CA. Crotty, M. 1998, The Foundations of Social Research: Meaning and Perspective in the Research Process, Allen & Unwin, St. Leonards, NSW. Flood, R.L. & Romm, N.R.A. 1996, Diversity Management: Triple Loop Learning, John Wiley & Sons, Chicester, UK. Golafshani, N. 2003, ‘Understanding reliability and validity in qualitative research’, The Qualitative Report, vol. 8, no. 4, pp. 597-607. Greene, J.C., Kreider, H. & Mayer, E. 2005, ‘Combining qualitative and quantitative methods in social inquiry’, in Research Methods in the Social Sciences, eds B. Somekh & C. Lewin, Sage Publications, London, pp. 274-281. Guba, E.G. & Lincoln, Y.S. 2005, ‘Paradigmatic controversies, contradictions, and emerging confluences’, in The Sage Handbook of Qualitative Research, eds N.K. Denzin & Y.S. Lincoln, 3rd edn, Sage Publications, Thousand Oaks, CA, pp. 191-215. Healy, M. & Perry, C. 2000, ‘Comprehensive criteria to judge validity and reliability of qualitative research within the realism paradigm’, Qualitative Marketing Research: An International Journal, vol. 3, no. 3, pp. 118-126. Hojat, M., Gonnella, J.S. & Caelleigh, A.S. 2003, ‘Impartial judgment by the “gatekeepers’ of science: Fallibility and accountability in the peer review process’, Advances in Health Sciences Education, vol. 8, pp. 75-96. Krathwohl, D.R. 1985, Social and Behavioral Science Research: A New Framework for Conceptualizing, Implementing, and Evaluating Research Studies, Jossey-Bass, San Francisco. Krathwohl, D.R. 1998, Methods of Educational and Social Science Research: An Integrated Approach, 2nd ed., Addison Wesley Longman, New York. Kuhn, T.S. 1970, The Structure of Scientific Revolutions, The University of Chicago Press, Chicago. Johnson, R.B. & Onwuegbuzie, A. 2004, ‘Mixed methods research: A research paradigm whose time has come’, Educational Researcher, vol. 33, no. 7, pp. 14-26. LeCompte, M. & Goetz, J. 1982, ‘Problems of reliability and validity in ethnographic research’, Review of Educational Research, vol. 52, no. 1, pp. 31-60. Mingers, J. & Gill, A. 1997, Multimethodology: The Theory and Practice of Combining Management Science Methodologies, John Wiley & Sons, Chichester, UK. Neuman, W.L. 2006, Social Research Methods: Qualitative and Quantitative Approaches, 6th edn, Pearson Education, Boston, MA. 17 UNIVERSITY OF NEW ENGLAND Onwuegbuzie, A. & Teddlie, C. 2003, Handbook of Mixed Methods in Social & Behavioral Research, Sage Publications, Thousand Oaks, CA Sale, J.E. & Brazil, K. 2004, ‘A strategy to identify critical appraisal criteria for primary mixed-method studies’, Quality & Quantity, vol. 38, pp. 351-365. Seale, C. 1999, The Quality of Qualitative Research, Sage Publications, London. Thomas, A.B. 2006, Research Concepts for Management Studies, Routledge, Milton Park, UK.