Academics' perceptions of ChatGPT-generated written outputs: A practical application of Turing’s Imitation Game

Title
Academics' perceptions of ChatGPT-generated written outputs: A practical application of Turing’s Imitation Game
Publication Date
2023-12-22
Author(s)
Matthews, Joshua
( author )
OrcID: https://orcid.org/0000-0002-2260-2331
Email: jmatth28@une.edu.au
UNE Id une-id:jmatth28
Volpe, Catherine Rita
( author )
OrcID: https://orcid.org/0000-0002-7500-9937
Email: cjohns86@une.edu.au
UNE Id une-id:cjohns86
Type of document
Journal Article
Language
en
Entity Type
Publication
Publisher
ASCILITE
Place of publication
Australia
DOI
10.14742/ajet.8896
UNE publication id
une:1959.11/57095
Abstract

Artificial intelligence (AI) technology, such as Chat Generative Pre-trained Transformer (ChatGPT), is evolving quickly and having a significant impact on the higher education sector. Although the impact of ChatGPT on academic integrity processes is a key concern, little is known about whether academics can reliably recognise texts that have been generated by AI. This qualitative study applies Turing's Imitation Game to investigate 16 education academics' perceptions of two pairs of texts written by either ChatGPT or a human. Pairs of texts, written in response to the same task, were used as the stimulus for interviews that probed academics' perceptions of text authorship and the textual features that were important in their decision-making. Results indicated academics were only able to identify AI-generated texts half of the time, highlighting the sophistication of contemporary generative AI technology. Academics perceived the following categories as important for their decision-making: voice, word usage, structure, task achievement and flow. All five categories of decision-making were variously used to rationalise both accurate and inaccurate decisions about text authorship. The implications of these results are discussed with a particular focus on what strategies can be applied to support academics more effectively as they manage the ongoing challenge of AI in higher education.

Link
Citation
Australasian Journal of Educational Technology, 39(5), p. 82-100
ISSN
1449-5554
1449-3098
Start page
82
End page
100
Rights
Attribution-NonCommercial-NoDerivatives 4.0 International

Files:

NameSizeformatDescriptionLink
closedpublished/AcademicsMatthewsVolpe2023JournalArticle.pdf 516.416 KB application/pdf Published version View document