Conference proceeding
Analyse, evaluate, create: Using ChatGPT to develop student's critical analysis skills for 'what good looks like'
Proceedings of the 35th Annual Conference of the Australasian Association for Engineering Education (AAEE 2024), pp.75-82
35th Annual Conference of the Australasian Association for Engineering Education (AAEE 2024), 35th (Christchurch, New Zealand, 08/12/2024–11/12/2024)
12/2024
Metrics
1 Record Views
Abstract
CONTEXT: A student’s ability to critically analyse text is crucial for engineering students both during, and beyond their studies. Building on this ability to analyse, is then the ability to evaluate and create, completing Krathwohl’s revision of Bloom’s taxonomy (Krathwohl, 2002). Current students are suffering from a lack of professional skills when graduating (Crosthwaite, 2021; Crosthwaite et al.,2018), demonstrating the inability to utilise the higher levels of Bloom’s taxonomy even in the most basic examples. Furthermore, with the prevalence of generative artificial intelligence(GenAI), it is more important than ever to develop these critical evaluation skills early in their studies.
PURPOSE: This paper describes an innovative assessment design that utilises ChatGPT to generate an essay text that students are required to critique. The idea stems from students’ requests for an essay exemplar to assist them in completing this particular assessment in past years. When investigating the click-through rate of previously provided examples, it showed that these were not typically opened by students. As such, embedding the exemplar essay as part of the assessment yields multiple benefits including exposure to critical analysis, evaluation, and providing feedback.
APPROACH: The design of the assessment is constituted of two parts. The first was to read, mark and critique an AI generated essay for topical, formatting, and grammatical errors; the second took this to a higher level by writing their own essay. The premise is that the students would develop their own written essays at a comparable, or higher level, to the exemplar essay ChatGPT had generated. This forms the research question if students are required to read and mark an essay, do they then write to that level? This assessment item was included in a second-level course at UniSQ in2023 that focused on technology and society. The cohort comprised of 64 students most of which in their first or second year of their studies.
OUTCOMES: This study showed that the premise of critically analysing an essay that is embedded as part of an assessment allowed for the development of the higher levels of Bloom's taxonomy. Students on average, received a mark that was higher than the mark the GenAI-generated essay received. In three out of the four sections that were marked, the students' essays received a higher grade than the ChatGPT-generated essay by more than 5%. Tangential outcomes were also observed showing the development of students’ knowledge of ChatGPT (GenAI) and it’s place in both industry and study. During this process, it was noted that the ChatGPT-generated essay passed both the academics’ and students' marking process and that both the students and academics marked to the same overall level.
CONCLUSIONS Overall, the project demonstrated students were able to develop critical analysis skills while evaluating and creating to a comparable or higher level than what they had previously evaluated. Through the assessment the students were exposed to GenAI and shown its positives and negatives. The results showed that embedding an analysis style assessment into the curriculum can help develop critical skills in a more efficient and applicable manner than directly teaching it. Copyright © Quince, Seligmann, Maxwell 2024KEYWORDSCritical analysis, GenAI & innovative assessment
Details
- Title
- Analyse, evaluate, create: Using ChatGPT to develop student's critical analysis skills for 'what good looks like'
- Creators
- Zachary Quince - University of Southern QueenslandHannah Seligmann - University of Southern QueenslandAndrew Maxwell - University of Southern Queensland
- Contributors
- Enda Crossin (Author)Matt Pour (Author)
- Publication Details
- Proceedings of the 35th Annual Conference of the Australasian Association for Engineering Education (AAEE 2024), pp.75-82
- Conference
- 35th Annual Conference of the Australasian Association for Engineering Education (AAEE 2024), 35th (Christchurch, New Zealand, 08/12/2024–11/12/2024)
- Publisher
- Engineers Australia; Christchurch, New Zealand
- Number of pages
- 9
- Identifiers
- 991013287148502368
- Copyright
- © Quince, Seligmann, Maxwell, 2024. The authors assign to the Australasian Association for Engineering Education(AAEE) and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to AAEE to publish this document in full on the World Wide Web (prime sites and mirrors), on Memory Sticks, and in printed form within the AAEE 2024 proceedings. Any other usage is prohibited without the express permission of the authors.
- Academic Unit
- Centre for Teaching and Learning; Faculty of Science and Engineering
- Language
- English
- Resource Type
- Conference proceeding