In Search of Evidence: Measuring Community Engagement - A Pilot Study

Abstract

Recent discussions about the inclusion of the community engagement agenda as part of the National Protocols for Higher Education Approval Processes and calls for Third Stream funding have highlighted the need for some measures of university-community engagement. Such measures of engagement should offer tangible evidence of the role that universities play in the educational, social and economic wellbeing of local communities and the nation more broadly. In Search of Evidence builds on the self-audit of Victorian universities undertaken for the Department of Education and Training, Victoria in 2005, reported in Beyond Rhetoric: University-Community Engagement in Victoria. One of the recommendations made was for the development of appropriate indicators for engagement in Research and Teaching Quality frameworks. The pilot project therefore investigates the practicality of using currently collected and published information to report against a set of indictors to determine the type and extent of community engagement activity in Victorian universities. This will involve the examination of national and international literature in order to develop a preliminary set of indicators, which will then be tested against the data available at Victorian universities. A major condition for such an indicator set is that it does not generate yet another demand on universities to generate data, but that it can operate within the framework of the current data collected and reported by universities. The intention therefore of this report is not to suggest a new suite of indicators but to select from those that have been suggested in the literature, a set which could be applied to evaluate the nature of community-university engagement in Victoria. Or, to be more precise, to actually assess whether such an approach might work. To test this, a two-stage approach is used. First, on the basis of a comprehensive international literature review an attempt is made to draw up a set of indicators relevant to the concept of community engagement. Second, using information available in the public domain, i.e. published reports and data contained on university websites, we investigate the extent to which the indicator set can be populated to constitute a reasonable understanding of the community engagement activities within and between universities in Victoria. The international literature search resulted in six sets of indicators and protocols that could prove useful in the development of a set of indicators for community engagement. These indicator sets were evaluated for their usefulness to the objective of this project based on the following criteria: Relevance, Reliability, Transparency, Availability, Auditability and Cost. This assessment resulted in the selection of the so-called Russel Group Indicators, which had the highest overall score (according to those criteria) and also provided the most comprehensive coverage of what in the broadest sense may be captured under the heading of community engagement. We opted to continue the project with that set, while adding in a number of other relevant measures. A trial run consisting of an analysis of existing Victorian university reports and other available forms of data was undertaken to see to what extent the suggested set of indicators could indeed be viably used to determine the extent of community engagement. The outcomes of this trial run were disappointing. Our research suggests that at present there is insufficient information reported to even superficially understand the level of community engagement being undertaken by a university. We believe that this is not the result of the indicator set selected, but is the consequence of what universities report – or not report for that matter – in the area of community engagement. Real reporting of achievements against stated intention only takes place to a very limited extent.