Performance of Multimodal GPT-4V on USMLE with Image: Potential for Imaging Diagnostic Support with Explanations

Abstract

Importance Using artificial intelligence (AI) to help clinical diagnoses has been an active research topic for more than six decades. Few research however has the scale and accuracy that can be turned into clinical practice. The tide may be turned today with the power of large language models (LLMs). In this application, we evaluated the accuracy of medical license exam using the newly released Generative Pre-trained Transformer 4 with vision (GPT-4V), a large multimodal model trained to analyze image inputs with the text instructions from the user. This study is the first to evaluate GPTs for interpreting medical images. Objective This study aimed to evaluate the performance of GPT-4V on medical licensing examination questions with images, as well as to analyze interpretability. Design, Setting, and Participants We used 3 sets of multiple-choice questions with images to evaluate GPT-4V performance. The first set was the United States Medical Licensing Examination (USMLE) from the National Board of Medical Examiners (NBME) sample questions in step1, step2CK, and step3. The second set was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the user base. The third set was the Diagnostic Radiology Qualifying Core Exam (DRQCE) from the American Board of Radiology. The study (including data analysis) was conducted from September to October 2023. Main Outcomes and Measures The choice accuracy of GPT-4V was compared to two other large language models, GPT-4 and ChatGPT. The GPT-4V explanation was evaluated across 4 qualitative metrics: image misunderstanding, text hallucination, reasoning error, and non-medical error. Results Of the 3 exams with images, NBME, AMBOSS, and DRQCE, GPT-4V achieved accuracies of 86.2%, 62.0%, and 73.1%, respectively. GPT-4V outperformed ChatGPT and GPT-4 by 131.8% and 64.5% on average across various data sets. The model demonstrated a decreasing trend in performance as question difficulty increased in the AMBOSS dataset. GPT-4V achieves an accuracy of 90.7% in the full USMLE exam, outperforming the passing threshold of about 60% accuracy. Among the incorrect answers, 75.9% of responses included misinterpretation of the image. However, 39.0% of them could be easily solved with a short hint. Conclusion In this cross-sectional study, GPT-4V achieved a high accuracy of USMLE that was in the 70th - 80th percentile with AMBOSS users preparing for the exam. The results suggest the potential of GPT-4V for clinical decision support. However, GPT-4V generated explanation revealed several issues. It needs to improve explanation quality for potential use in clinical decision support.

Hidden mysteries behind genome, epigenome, and exposome of lupus erythematosus

“…The best performance under the open-book setting is achieved by human physicians (95%, CI: 91-99%), though not significantly different from GPT-4V. Our findings, therefore, align with the previous ones, which show the superior performance of GPT-4V in the closed-book setting 12,13 .…”

Deep Learning Classification Methods Applied to Tabular Cybersecurity Benchmarks

“…Wu et al examine GPT-4V's potential in multimodal medical diagnosis, demonstrating substantial promise yet revealing limitations in high-stakes domains [6]. Echoing this view, Yang et al assessed the performance of Multimodal GPT-4V in medical licensing exams, particularly in imaging diagnostics, offering a glimpse into future support systems for medical professionals [7]. The concept of "Socratic models" by Zeng et al brings forth the idea of zero-shot multimodal reasoning, allowing models to compose answers from disparate sources without explicit training [8].…”

“…The creative maker spaces have become vibrant hubs of 21 st -century innovation, merging the traditional tactile experience with digital fabrication and design. However, integrating new artificial intelligence (AI) tools and, in particular, the current generation of multimodal large language models (LLMs) [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17] into these environments has the potential to enhance human creativity and innovation [18][19][20][21][22][23][24][25][26][27][28][29][30][31][32]. In recent years, the intersection of AI and multimodal (MM) learning has spawned a generation of models that integrate and interpret information across various forms of data, including text, images, and speech.…”

Impact of Multimodal Prompt Elements on Diagnostic Performance of GPT-4(V) in Challenging Brain MRI Cases

“…Recent studies have further explored the diagnostic application of multimodal LLMs (also called 'vision-language models') that are able to ingest not only text but also image data as input (12)(13)(14)(15)(16)(17)(18)(19)(20). However, several studies demonstrated low performance of Generative Pretrained Transformer 4 Vision (GPT-4V) by OpenAI in differential diagnosis based on various types of radiological images (12,16,18,20,21).…”