Logo image
Back
Analyse, evaluate, create: Using ChatGPT to develop student's critical analysis skills for 'what good looks like'
Conference proceeding   Open access

Analyse, evaluate, create: Using ChatGPT to develop student's critical analysis skills for 'what good looks like'

Zachary Quince, Hannah Seligmann and Andrew Maxwell
Proceedings of the 35th Annual Conference of the Australasian Association for Engineering Education (AAEE 2024), pp.75-82
35th Annual Conference of the Australasian Association for Engineering Education (AAEE 2024), 35th (Christchurch, New Zealand, 08/12/2024–11/12/2024)
12/2024
Appears in  Recent Faculty of Science and Engineering Publications
url
Analyse, evaluate, create: Using ChatGPTView
Published (Version of record) Open Free to Read

Related links

Metrics

13 Record Views

Abstract

critical analysis GenAI & innovative assessment Engineering education Higher education
CONTEXT: A student’s ability to critically analyse text is crucial for engineering students both during, and beyond their studies. Building on this ability to analyse, is then the ability to evaluate and create, completing Krathwohl’s revision of Bloom’s taxonomy (Krathwohl, 2002). Current students are suffering from a lack of professional skills when graduating (Crosthwaite, 2021; Crosthwaite et al.,2018), demonstrating the inability to utilise the higher levels of Bloom’s taxonomy even in the most basic examples. Furthermore, with the prevalence of generative artificial intelligence(GenAI), it is more important than ever to develop these critical evaluation skills early in their studies. PURPOSE: This paper describes an innovative assessment design that utilises ChatGPT to generate an essay text that students are required to critique. The idea stems from students’ requests for an essay exemplar to assist them in completing this particular assessment in past years. When investigating the click-through rate of previously provided examples, it showed that these were not typically opened by students. As such, embedding the exemplar essay as part of the assessment yields multiple benefits including exposure to critical analysis, evaluation, and providing feedback. APPROACH: The design of the assessment is constituted of two parts. The first was to read, mark and critique an AI generated essay for topical, formatting, and grammatical errors; the second took this to a higher level by writing their own essay. The premise is that the students would develop their own written essays at a comparable, or higher level, to the exemplar essay ChatGPT had generated. This forms the research question if students are required to read and mark an essay, do they then write to that level? This assessment item was included in a second-level course at UniSQ in2023 that focused on technology and society. The cohort comprised of 64 students most of which in their first or second year of their studies. OUTCOMES: This study showed that the premise of critically analysing an essay that is embedded as part of an assessment allowed for the development of the higher levels of Bloom's taxonomy. Students on average, received a mark that was higher than the mark the GenAI-generated essay received. In three out of the four sections that were marked, the students' essays received a higher grade than the ChatGPT-generated essay by more than 5%. Tangential outcomes were also observed showing the development of students’ knowledge of ChatGPT (GenAI) and it’s place in both industry and study. During this process, it was noted that the ChatGPT-generated essay passed both the academics’ and students' marking process and that both the students and academics marked to the same overall level. CONCLUSIONS Overall, the project demonstrated students were able to develop critical analysis skills while evaluating and creating to a comparable or higher level than what they had previously evaluated. Through the assessment the students were exposed to GenAI and shown its positives and negatives. The results showed that embedding an analysis style assessment into the curriculum can help develop critical skills in a more efficient and applicable manner than directly teaching it. Copyright © Quince, Seligmann, Maxwell 2024KEYWORDSCritical analysis, GenAI & innovative assessment

Details

Logo image