To Evaluate the Efficiency of ChatGPT in Medical Education: An Analysis of MCQ-Based Learning and Assessment

Authors

  • Shirjeel Husain Department of Internal Medicine, Liaquat College of Medicine and Dentistry
  • Zaid Ansari Department of Internal Medicine, Liaquat College of Medicine and Dentistry Darul Sehat Hospital
  • Azhar Hussain Department of Internal Medicine, Liaquat College of Medicine and Dentistry & Darul Sehat Hospital, Karachi.
  • Sahar Zubair Directorate of Medical Education, College of Physicians & Surgeons Pakistan
  • Tahera Ayoob Department of Oral Surgery, Liaquat College of Medicine and Dentistry & Qamar Dental Hospital, Karachi
  • Rimsha Mujahid Department of Internal Medicine, Liaquat College of Medicine and Dentistry & Darul Sehat Hospital, Karachi.

DOI:

https://doi.org/10.58397/ashkmdc.v28i04.759

Keywords:

Medical education

Abstract

Objective: This study aimed to evaluate the potential of ChatGPT to help students for their assess ments via MCQ at different level of cognition by using different subjects of Internal medicine.

Methods: This cross-sectional study was conducted in the Department of Internal Medicine in col laboration with post graduate medical education department from June 2023 to August 2023. An MCQ bank was established from three books of MCQ’s on subject of Internal Medicine. Total 1428 MCQ’s were followed for scrutiny and 307 MCQ’s were selected for the assigned task. The selected MCQ’s were manually entered one by one in a fresh Chat GPT session. The response was noted against the replies given in respective MCQ’s book and marked as correct, not correct or partially correct. MCQ’s were categorized as per chapters in Internal medicine and as per cognition level of MCQ’s i.e. C1, C2 and C3. Data was analyzed on SPSS version 21.00.

Results: Chat GPT replied with 199 correct replies while 98 were wrong and 10 were partially correct. Chat GPT scored 64% overall in all categories. At level of cognition, it solved C2 MCQ’s by 80 % but scored 69% and 54% in C1 and C3 categories respectively. Chat GPT replied with 80% accuracy for C2 level MCQ’s while results were low for C3 category at around 54%. C1 also had low percentage of correct answers standing close to 69.8%. Almost all subjects showed healthy responses around the mean except for endocrinology and hematology where responses are below 60% and 40% respectively.

Conclusion: This study findings suggest that ChatGPT is a useful tool for students and medical educationist with its current framework but a subtle approach should be inclined towards its role in future.

Downloads

Published

2023-12-02

Issue

Vol. 28 No. 04 (2023): Annals of Abbasi Shaheed Hospital and Karachi Medical and Dental College

Section

Original Article

License

Copyright (c) 2023 ANNALS OF ABBASI SHAHEED HOSPITAL AND KARACHI MEDICAL & DENTAL COLLEGE

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Annals of Abbasi Shaheed Hospital and Karachi Medical and Dental College acquires copyright ownership of the content. The articles are distributed under a Creative Commons (CC) Attribution-Non-Commercial 4.0 License (http://creativecommons.org/licenses/by-nc/4.0/). This license permit uses, distribution and reproduction in any medium; provided the original work is properly cited and initial publication in this journal.