2006 POSTGRADUATE CONFERENCE Using mixed methods in an essentially qualitative on design and process in an Th a sed to ascertain the views of teachers implementing the NSW Sustainable Schools Program. An assumption of the researcher was that teachers would interpret the rogram according to their own beliefs and according to the context within which they worked. Of particular study have plications for the usefulness of the recommendations that arise regarding ways teachers in NSW schools eachers from around NSW. This paper discusses the nswer the research questions that drove the study. It also reflects on the study: reflections education for sustainability context Julie Kennelly, Dr Neil Taylor, Kathy Jenkins Abstract is p per reports on the methodological approach u P interest was the extent to which teachers’ views were in keeping with underlying philosophical assumptions of the Sustainability Program. The research involved the use of mixed methods within an essentially qualitative approach. In-depth interviews with eight teachers provided the main source of data. This was supported by survey data for the purpose of identifying views amongst a wider sample. The details and limitations of the study design and its practical application are discussed. The methods employed in the im can be supported in environmental education. Introduction In 2003 the Sustainable Schools Program (SSP) was set up in 198 volunteer Kindergarten to Year 12 schools in NSW, in an endeavour to facilitate the implementation of the Environmental Education Policy for Schools (NSW Department of Education 2001). The study reported in this paper involved interviews with SSP coordinating teachers, one in each of eight SSP schools in regional NSW. These 8 teachers were all of those with whom the researcher was working in the role of SSP regional program facilitator. Additionally the study included a survey of a further 26 SSP coordinating t data gathering methods chosen to a usefulness and limitations of the research design. The research questions and their implications Tilbury et al. (2005) argued that education for sustainability is underpinned by socially critical theory. It can be argued that the NSW Policy on Environmental Education (hereafter called the Policy) with its focus on sustainability and on the active role of students in addressing environmental issues is also based on a socially critical approach. Socially critical environmental education seeks to provide students with the capacity 184 UNIVERSITY OF NEW ENGLAND to question and improve environment related practices. Huckle (1991) included the following characteristics in a description of socially critical environmental education: • Learning is active and experiential ocesses. This underlying approach is significant because it implies that in implementing the Policy, teachers will teach in a socially critical way. It would appear also that this approach represents a change from current practice in many schools (Tilbury et al. 2005). A fundamental question of the research was: In what ways do the teachers in the study adopt a recognisably socially critical approach? This is a question about how teachers view the world, about what they consider is important learning for their students and about how that should occur. Because this is a question about how people construct and view their social world a constructivist paradigm (Tashakkori & Teddlie 1998) was adopted. The ontological assumption of the qualitative methods of enquiry chosen is that multiple realities exist for those in the research situation (Creswell 1998). In answering questions about the socially constructed reality of the participants, a qualitative approach allows: • Capturing of an understanding of the participants’ ‘everyday’ or ‘normal’ experience • A holistic view of the context within which people work including insights into the complexity of the e of Th er research questions ritical approach to • Students are encouraged to think critically • Values education addresses the sources of social beliefs and values, how they are transmitted and the interests they support • Students engage in collaborative and democratic change pr work situation and the pressures that influence it • Procurement of rich descriptive data facilitating a deeper understanding of the participant’s view • Flexibility, where in the absence of rigidity and control in research design, the dynamic natur social processes over time could be better captured (Miles & Huberman 1994). ese characteristics of qualitative research suited the kind of investigation that was required to answ about teachers’ experience of actively involving students in a c education for sustainability. 185 2006 POSTGRADUATE CONFERENCE Overa Sandelowski (2003 p.23 can be informed b the present study a constructivist paradigm informed the r qualitativ ntitative tool could be linked, in a mann s • to • to broaden t There r (1994). I e researcher conducts a study using a single dominant approach, with one small omponent drawn from an alternative approach. This appears to introduce an inconsistency in choice of approach on e one hand and survey data, commonly associated with a quantitative approach on the other. quantitative tool, was used to derive umerical data. That survey however comprised open questions and rankings of teacher constructed views ut the importance of curriculum, community support and their role in the school. As a specific example, respondents were asked to align themselves along a cation. In this sense the underlying assumption of socially constructed reality was not challe t the level Teddlie and Tashakkori (2003) acknowledged at least six different positions on the issue of how paradigms digm should serve as the foundation for mixed methods research’ (Teddlie & ashakkori 2003 p.17), and within that position some advocate that pragmatism serves as the basis of ll research design 6) argued that ‘a program of research might comprise … a mixed methods study [but] y only one paradigm’ or world view. In esearch. However, while the nature of the question to be answered influenced a choice in favour of e sources, it was perceived that data from an additional qua er uggested by Miles and Huberman (1994). This was: enable corroboration through triangulation he sample used in the investigation. fo e the model used for the study was based on a mixed methods approach described by Creswell n this model th c methods: the use of interview data and observer notes commonly associated with a qualitative th Sandelowski (2003 p.326) argued that ‘methods do not by themselves signal much about the nature of inquiry’. In the present study, the survey, commonly viewed as a n derived from two initial interviews, views for instance abo continuum between two opposing views derived from earlier interviews about the role of the teacher in environmental edu nged. The study design in this way included mixed methods at the level of data gathering but not a of paradigm or overarching world view. are used in the development of mixed methods research. Amongst these is the position ‘where researchers believe that a single para T method selection. In the present study the survey tool was administered to a wider population than the interview sample. The wider sample could not only add breadth to the interview data but also allow the possibility of different viewpoints to arise through use of some open questions. Using the survey was a pragmatic decision where ‘specific decisions regarding the use of mixed methods or qualitative methods or 186 UNIVERSITY OF NEW ENGLAND quantitative methods depend on the research question as it is currently posed’ (Teddlie & Tashakkori 2003 p.20). Bryman (1988) also discussed an approach which combines methods in order to triangulate data. In this instan esent study , as the su f the resea ce, the findings from one type of study can be used as a check against another type. In the pr , interpretation of the evidence from interview data, was checked against evidence in survey data rvey was also administered to the interview sample. The aim here was to enhance the validity o rcher’s interpretation of the interview data. Figure 1: The mixed methods research strategy selected is represented here using notation suggested by Tashakkori and Teddlie (1998). In this notation a plus indicates concurrent data collection and an arrow indicates sequential data collection. Capitalisation indicates emphasis on either quantitative or qualitative data and analysis. In this case, prioritisation was given to a qualitative approach. Note that the eight interviews were treated as a total data set for the purposes of qualitative analysis. The survey was derived s of data occurred after analysis. The same survey as was delivered to 26 SSP coordinating teachers was later delivered to interviewees for the purpose of from the first two interviews. Integration of these two type triangulating with interview data. A researcher’s diary was also kept. In the light of these possibilities the methods used to gain information were as follows: 187 2006 POSTGRADUATE CONFERENCE • Hour long interviews with eight SSP coordinating teachers were undertaken. All transcripts were returned to interviewees for member checking purposes. • A survey was written on the basis of themes arising in the first two interviews, researcher experience and a review of the literature. throughout NSW. The nature of this sample is described below. • Researcher visits to the schools of the eight interviewees occurred on average twice per term over related events in the schools were kept in a researcher’s diary. The diary could be used for adding to contextual eight interviewees for the purpose of triangulation of the interview data ary means of collecting data: … un un It was th seemed An ter p.83). Q some re questions were standardised so that comparisons could be made from one context to another (Kvale 1996). • The survey was presented to a different and separate population of coordinating teachers from six school terms. Précis of conversations held during these visits and records of data and for confirming interpretation of interviews. • The survey was sent to the interpretation. In brief, data was gathered using an interview, a survey and recordings in a researcher’s diary. Information from the interviews could then be compared with survey information derived from a wider sample. Interpretation of information from the interviews could also be checked against information from the survey completed by interviewees for the purpose of triangulation (Figure 1). Interviews and the interview sample With consideration for the following remarks by Kvale (1996 p.105) interview was chosen as a prim it should not be forgotten that interviews are particularly suited for studying people’s derstandings of the meanings in their lived world, describing their experiences and self- derstanding, and clarifying and elaborating their own perspective. e teacher’s own experience and self understanding that was sought therefore an in-depth interview most appropriate. in view schedule was prepared in advance in a manner suggested by Maykut and Morehouse (1994 uestions were mostly open ended and included those requiring description, opinion, perceptions and quiring interviewee reflection on situations prevailing at the time. The wording and sequencing of 188 UNIVERSITY OF NEW ENGLAND The que as a Su cilitator and on a reading of the literature pertaining to environmental education. Basing the interview schedule on an understanding of the situation is recommended by Kvale (19 p. A s for loping a conceptual and theoretical understanding of the phenomena to be investigated, to establish the base to which new kn po As sugg in a school ngaging with the NSW Policy on Environmental Education but not in the SSP, in order to determine: • if all of the questions were clearly expressed ns gave the interviewee the best opportunity to express views After this pilot, the interview guide was reviewed and amended. This study concentrated on teachers within private and public schools who were motivated to work in environmental education, to integrate it into their teaching and to take on the extra duties of arranging whole sues-based approach. As is often the case in qualitative research the sample was small and this relates to their context and studied in depth’ (Miles & Huberman 1994 p.27). The sample was chosen in a purposive way and conforms with the notion of cri io in this case engagement with the SSP achers actually engaging with the NSW Policy on Environmental Education a would be able to comment on their view of how this could be done. My role as SSP facilitator brought me into contact with such teachers and afforded the op rtu e survey samples Af a n Event was held to celebrate the ac ve lusters in NSW. These schools were selected to attend on the basis of their commitment to goals of the SSP, including the formation of a School Environment stions were written on the basis of researcher experience which included teaching in schools, acting stainable Schools Program fa 96 95): ignificant part of any interview project should take place before the tape recorder is turned on the first actual interview. This involves deve owledge will be added and integrated…Knowledge of a phenomenon is required to be able to se significant questions. ested by Kvale (1996), the initial questions were trialled with a colleague who worked e • which questio • which did the interviewee regard as the important issues • what was irrelevant? school involvement in an is the characteristic sampling of people ‘nested in ter n sampling where all participants meet a particular criterion, (Miles & Huberman 1994). Only those te nd with the process of developing a SEMP po nity to conduct interviews. The survey and th ter period of about one year of SSP operation, a Recognitio hie ments of two SSP schools from each of ten regional c 189 2006 POSTGRADUATE CONFERENCE Committee and the preparation of a School Environment Management Plan (SEMP). The researcher was present at the Recognition Event and was assigned the role of assisting participating students and teachers y a small group of students nd invited to attend the Recognition Event. This second sample therefore was distinguished by the following: coordinating teachers in this sample were from across NSW rson (in this case the researcher) can gain an the world (Cohen et al. 2001). However key limitations of sampling, is relevant to contexts outside the study situation. Each piece of research is in a series of activities. The survey sample was drawn from those coordinating teachers accompanied b a • the • they represented the ‘most successful’ schools in their small cluster of eight to ten. The Recognition Event, announced at short notice, presented an opportunity for data to be gathered from a wider group of people than the small number in the initial sample. For this reason it was an opportunity or convenience sample (Punch 1998). Limitations of qualitative research While a qualitative approach promised to provide rich data about teachers’ views and the context of their work, one premise upon which it rests is that another pe understanding of how an individual perceives qualitative research are that one person can never fully enter the consciousness of another, and that the researcher enters the field with preconceived ideas which build in an unknown degree of bias with regard to what is perceived and how it is interpreted (Cohen et al. 2001). Steps therefore need to be taken in qualitative research to ensure that the research is valid in its procedures, conclusions and claims for applicability. Reid and Gough (2000) noted that consideration of the following criteria, credibility, transferability, dependability and confirmability, is appropriate for evaluating qualitative research: • Credible research provides an authentic representation of the subjects’ experience. Credibility must be carefully considered in a situation where the researcher is the ‘instrument’ and it is assumed that there are multiple interpretations of reality. Credibility can be improved with purposive declaration of researcher subjectivity, prolonged engagement of the researcher, and triangulation. All of these processes were used in the present study. Triangulation is the comparison of data about a phenomenon from different sources, and in this case triangulation by type of data, was chosen (Miles & Huberman 1994). • Transferable research bound by its own context and its own time. Consequently it is incumbent upon the researcher to sample purposefully and to provide sufficient description so that others can know and understand 190 UNIVERSITY OF NEW ENGLAND the findings and context well enough to be able to decide if the findings are transferable or generalisable (Punch 1998). • Dependable research is associated with the attempt to minimise the idiosyncrasies of interpretation. researcher. Confirmability is improved by open reporting of procedures including documentation of procedures used in data analysis (Miles & Huberman 1994). archer declared a role of assistant to the teachers in the sample and reporter to the SSP o improve confirmability: Dependability can be improved where the researcher is participant, where the researcher’s role is explicitly described, and where triangulation is used. • Confirmable research is where the influence of biases, motivations, interests and perspectives of the researcher are minimised and declared. It is assumed that such biases can influence the interpretations of the It was therefore important to design a research process that took each of these criteria into account. To improve credibility: • three different kinds of data were gathered (triangulation) • the samples were purposive • the researcher worked in the field for 18 months • the rese organisers. To improve transferability: • the purposeful sample was described in detail • the work context of each interview participant was described To build in dependability of interpretation: • analysis of the data was described in detail • the researcher’s role was described in detail • several data sources were used and compared (triangulation). T • procedures used in data analysis and reporting were described • researcher interests and perspectives were declared. 191 2006 POSTGRADUATE CONFERENCE Limitations of the procedures used According to Gough (2002 p.22) ‘authors of educational research reports have a moral obligation to explain how their work might improve educational practices’. For the present study that meant: to what extent does this research assist those endeavouring to advance education for sustainability? Sample to be e coordinating teacher responsible for the task. The selection of SSP schools therefore made the sample in eliefs in line with a socially critical approach but felt that this approach had no place in the context of his of these two teachers are enlightening. They suggest in the first instance that there must d many other reasons why a policy of this nature could be mismatched with particular ke valuation by the reader possible. In the light of this question it would appear that a primary limitation of the present study rests with the size and the skewed nature of the sample. The aim of the research was to explore the views of teachers attempting to implement a policy that is socially critical in its approach. In order to find out how teachers felt about this, the study was forced to focus on teachers who were in a position of trying to implement it. These were the teachers in the SSP, because their schools had signed an agreement to implement particular key aspects of the Policy and because the individuals interviewed had either volunteered or been chosen th terms of the population of all schools in NSW, very skewed. Do other schools have aspirations regarding education for sustainability, what are their needs, what are their achievements, what do other teachers believe? These questions remained unanswered. In addition to being skewed in nature, the sample was small. Commonalities were identified in the views of eight interviewees, but two were readily identified as having views not in alignment with socially critical elements of the Policy procedures. One of the two believed that it was hypocritical to imagine that students could hold democratic influence within the authoritarian structures of schools. The other held strong personal b school. The views be many other views an schools. A limitation posed by the present sample is that these wider views and reasons remain unknown and therefore recommendations flowing from this study may not be those best suited to many schools. However, as Gough (2002) went on to say, ‘readers of educational research reports must also establish for themselves whether or not such reports inform their own work’. The onus then is on the reader to establish relevance, but responsibility remains with the researcher to provide sufficient contextual detail to ma e Survey The survey was used for two purposes. One purpose was to offer it to interviewees after all the interviews had been completed. It could then be used to check researcher interpretation of interview transcripts. It was issued by post at the end of the school year to the eight interviewees, but only six were returned. This 192 UNIVERSITY OF NEW ENGLAND precluded checking of interview interpretation for two of the interviewees. In order to improve response rates interviewees could have been asked to complete the survey immediately on completion of the interview or the survey could have been presented during school visits subsequent to the interviews. espite the fact that the interview schedule was trialled then modified before use with the study sample, shortcomings in its construction became apparent. Some questions suggested assumptions that could prejudice the openness of interviewee response. These are commonly referred to as leading questions (Cohen et al. 2001). For example, the question ‘What do you think helps students gain this feeling of ownership or involvement?’ implies that the students have such a feeling. If it were not ascertained in advance that teachers believed students experienced a feeling of involvement, the question had to be asked hypothetically or omitted. In this way the interview schedule was used as a guide rather than a format to be strictly followed. Advantages of the procedures used Whilst long engagement of the researcher in the field is thought to improve the credibility of a study it is possible also that such involvement could add to potential bias in researcher interpretation of the situation (Le Compte & Goetz 1982). In this study the researcher worked with the participants over a period of 18 months, helping them to implement a whole school approach to sustainability. This ‘help’ occasionally meant providing advice but mostly meant working with students alongside the teachers or finding resources that they could use. This position provided an understanding of the successes and difficulties of their work but most important of all it allowed a lengthy time for reflection between school visits and after interviews had been conducted. It appeared to the researcher that in this instance engagement in the field improved credibility and dependability rather than detracting from it. As argued by Le Compte and Goetz (1982) however, sufficient rich detail had to be reported to allow readers to judge the interpretation of the situation for themselves. Interview appeared to be a useful means of understanding teacher views. The interviews were conducted in the schools with teachers, while the students played tennis or potted seedlings or played during lunch break. Interviews were sometimes interrupted and then resumed at a later time. This situation appeared to be an The survey was also completed by teachers at the Recognition Event. The limitations of survey as an instrument became apparent during analysis of the data. The survey included both ranking tasks and open ended questions. Although these tasks provided useful information, they also raised many questions that could not be clarified or otherwise pursued. This was an obvious and somewhat frustrating limitation of the survey tool which otherwise gave the advantage of providing views from a wider sample of teachers on the themes identified in interviews. Interview D 193 2006 POSTGRADUATE CONFERENCE advantage as it allowed the researcher and the interviewee to reflect on what was being said, and it allowed erstanding of interview schedule so that lines of questioning that were of interest could be followed, and so that dability of interpretation. The survey had been written on the basis of themes terview: Did his words really mean that his views did not align with the Policy view? Did her discussion of d to hose that occupied the minds of the broader Teachers were asked for example, would the school’s environment programs discontinue if ey (as coordinators) were to leave. Their responses indicated that in about half of the schools the programs g. In this case it searcher and to colleagues, guiding further work as illustrated below. um to teachers and of integrating the aims of the Policy defining the priorities of the school and can act as gatekeepers to the ideas and assistance that become available to and the researcher to be a part of the everyday context of the school. This helped establish an und e way that the normal school day could accommodate the ideals and suggestions of the Policy that was th being considered. The flexibility of qualitative research was also an advantage. It allowed deviation from the repared p those leading questions which had inadvertently been included could be eliminated or altered. Information gained from the surveys sent to interviewees after all interviews were complete was particularly useful as a check on the depen arising in the first two interviews and was therefore in many ways closely aligned to the interview questions. The surveys returned from interviewees were valuable for clarifying interpretation of what had been said in in how well she was supported in the school agree with her survey ranking of support from the principal? Information gained from the same survey completed by teachers at the Recognition Event also appeare be useful in improving transferability. This information was compared with the interview findings and showed that the issues and views explicated by interviewees were t sample of teachers. The survey held a number of open ended questions and the answers to these proved to be informative. th would continue, but it was their explanations as to why this should happen that were revealin appeared that the issues of colleague and principal support were foremost in people’s responses. Discussion As mentioned earlier, there is a case for evaluating educational research in terms of whether or not it dvances practice or in some ways helps teachers in schools. It was also mentioned that there is onus on thea readers to evaluate the usefulness of the findings for their own situation. In the present instance the research findings and the recommendations that were derived from those findings have been useful at least to the re he study clearly showed the importance of curriculT through curriculum related activity. This finding is useful because it suggests and explains in detail what emphasis needs to be applied by those who continue to work with teachers implementing a whole school pproach to sustainability. Another finding of the study was that principals play a key role in a 194 UNIVERSITY OF NEW ENGLAND are promoted within the school community. This is a useful finding because it improves understanding of the role of the principal in cultural change in the school and how to best assist those who are interested in romoting an education for sustainability agenda. p The study also provoked further research questions. As an example, it became evident in the findings that there were particular barriers confronting teachers who wished to implement whole school environmental management in their schools. Amongst the barriers that teachers discussed was the problem of engaging their colleagues and helping them to understand the nature of education for sustainability and the processes involved. There is a clear need for development of professional learning experiences for both practising and pre-service teachers. Given that nation wide there are few teacher training institutions that include education for sustainability in their courses (Tilbury et al. 2005), there is a need for research into what should be included in professional learning experiences and how best to deliver them. 195 2006 POSTGRADUATE CONFERENCE REFERENCES Bryman, A. 1988, Quantity and Quality in Social Research, Unwin Hyman, London. ohen, L., Manion, L. & Morrison, K. 2001, Research Methods in Education, 5th edn, Routledge almer. 1994, Research Design Qualitative and Quantitative Approaches Sage, Thousand uckle, J. 1991, 'Education for sustainability Assessing pathways to the future', Australian Journal of , ucation Policy for Schools, NSW epartment of Education and Training Curriculum Support Directorate. ethodology Combining Qualitative and Quantitative pproaches, Sage, Thousand Oaks, California. housand Oaks, California, pp.3-50. C F Creswell, J. Oaks, California. Creswell, J. 1998, Qualitative Inquiry and Research Design Choosing mong Five Traditions, Sage, Thousand Oaks, California. Gough, N. 2002, 'Ignorance in Environmental Education Research', Australian Journal of Environmental Education, vol.18, pp.19-27. H Environmental Education, vol.7, pp.43-62. Kvale, S. 1996, Interviews: An Introduction to Qualitative Research Interviewing. Sage, London. Le Compte, M. & Goetz, J. 1982, 'Problems of reliability and validity in ethnographic research' Review of Educational Research, vol.52, no.1, pp.31-60. Maykut, M., & Morehouse, R. (1994) Beginning Qualitative Research: A Philosophic and Practical Guide: The Farmer Press Miles, M., & Huberman, M. 1994, Qualitative Data Analysis, 2nd edn, Sage, London. NSW Department of Education 2001, Environmental Ed D Punch, K. 1998, Introduction to Social Research, Sage, London. Reid, A. & Gough, S. 2000, 'Guidelines for reporting and evaluating qualitative research: What are the alternatives', Environmental Education Research, vol.6, no.1, pp.59-91. Sandelowski, M. 2003, 'Tables or tableaux: The challenges of writing and reading mixed methods studies', in Handbook of Mixed Methods in Social and Behavioral Research, eds A.Tashakkori & C.Teddlie, Sage, Thousand Oaks, California, pp.321-350. Tashakkori, A. & Teddlie, C. 1998, Mixed M A Teddlie, C. & Tashakkori, A. 2003, 'Major issues and controversies in the use of mixed methods in the social and behavioral sciences', in Handbook of Mixed Methods in Social and Behavioral Research, eds A.Tashakkori & C.Teddlie, Sage, T 196 UNIVERSITY OF NEW ENGLAND Tilbury, D., Colema Contribution to Sus n, V. & Garlick, D. 2005, A National Review of Environmental Education and its tainability in Australia School Education, Australian Government Department of Environment and Heritage and Australian Institute in Education for Sustainability ARIES, Canberra. 197