INTRODUCTION
Generative artificial intelligence (AI) based large language models (LLMs) have expanded rapidly in both capability and adoption since the public release of ChatGPT in late 2022 (Mesko 2023). In medicine, these models have demonstrated strong performance on complex reasoning tasks and standardized examinations, suggesting a potential role in clinical care and medical education (Singhal et al. 2023).
Several studies have evaluated LLM performance on high-stakes examinations, including the United States Medical Licensing Examination (USMLE), demonstrating passing-level performance without specific fine-tuning (Brin et al. 2023; Bicknell et al. 2024; Kung et al. 2023). More recently, benchmarking efforts have expanded to specialty board and in-service examinations, including orthopaedic surgery, to compare LLM performance with residence physicians and practicing clinicians (Zhang et al. 2024). The Orthopaedic In-Training Examination (OITE), administered annually by the American Academy of Orthopaedic Surgeons (AAOS), serves as the primary standardized assessment for orthopaedic residents in the United States and a predictor of performance on the American Board of Orthopaedic Surgey (ABOS) Part 1 certifying examination (Swanson et al. 2013). Accordingly, multiple groups have used OITE questions to evaluate LLMs, reporting that contemporary general-purpose models such as GPT-4 and Google Bard or Gemini achieve scores comparable to junior or mid-level orthopaedic residents, though raising questions about how these tools compare with more senior trainees and how such performance should be interpreted in the context of orthopaedic education (Mo et al. 2025).
Despite this growing literature, important methodological limitations persist. Many existing studies focus on a single model or do not fully characterize question types, subspecialty domains, or prompting techniques. Recent commentaries have emphasized the need for more rigorous comparative evaluations with transparent reporting of model versions, prompting strategies, and question selection (Kunze et al. 2025).
Specialized, clinically oriented AI tools have emerged to address limitations of general-purpose models, particularly regarding source transparency and hallucinations. OpenEvidence (OpenEvidence, Inc.) is one such platform; it is a clinician-facing LLM that grounds responses in cited, peer-reviewed medical sources, which may improve transparency and reduce hallucinations but does not eliminate them entirely (Hurt et al. 2025). Although OpenEvidence is increasingly used by physicians and trainees for point-of-care questions, it has not previously been evaluated on standardized orthopaedic examinations such as the OITE as part of a broader comparative analysis of LLMs.
Given the rapid evolution of LLMs, prior benchmark studies can quickly become outdated as new model versions are released (Yang et al. 2025). Rather than identifying a single ‘best’ model, comparing performance across different model architectures may help orthopaedic trainees and educators decide which tools appropriate for evidence-based studying and rotation-level clinical reasoning, and identify where supervision is most needed given known limitations (Xu et al. 2025). Therefore, the objective of this study was to evaluate and compare the performance of three large language models ChatGPT-5 (OpenAI), Gemini 2.5 Pro (Google DeepMind), and OpenEvidence (OpenEvidence, Inc.) on performance in the 2014 OITE across cognitive taxonomies and orthopaedic subspecialities.
METHODS
Study Design
We conducted a cross-sectional comparative study to evaluate the performance of three large language models (LLMs), ChatGPT-5 (OpenAI), Gemini 2.5 Pro (Google DeepMind), and OpenEvidence (OpenEvidence, Inc.) on a standardized orthopaedic exam between October and November 2025. The study utilized the 2014 Orthopaedic In-Training Examination (OITE) as the question source. As this study involved publicly available, de-identified educational materials and no human subjects, it was exempt from Institutional Review Board (IRB) oversight.
Question Source and Classification
The complete 2014 OITE question set was obtained from publicly available archives. Questions were manually screened for inclusion. Items requiring video playback or those with corrupted data were excluded. A total of 265 multiple-choice questions were included in the final dataset. Each question was categorized by orthopaedic subspecialty according to the American Academy of Orthopaedic Surgeons (AAOS) domain structure: Trauma, Spine, Sports Medicine, Pediatrics, Adult Reconstruction (Hip/Knee), Hand/Upper Extremity, Foot and Ankle, Shoulder and Elbow, Oncology, and Basic Science. Additionally, questions were classified by cognitive taxonomy based on established medical education frameworks defined by Buckwalter et al: Taxonomy 1 (T1): Recall and recognition of facts, Taxonomy 2 (T2): Interpretation of data, images, or diagnostic studies, and Taxonomy 3 (T3): Complex clinical reasoning, problem-solving, and management decisions (Buckwalter et al. 1981).
Classifications were done by two reviewers S.P and S.H and disagreements were resolved by a third review, A.K. Questions were further stratified into text-only (n=111) and image-based (n=154) subsets to accommodate the different input capabilities of the AI models. Image-based questions included radiographs, MRI/CT scans, or clinical photographs essential for determining the correct answer.
AI Models and Input Protocol
Three models representing different AI architectures were prompted: ChatGPT-5 (Auto), Gemini 2.5 Pro, and OpenEvidence. As OpenEvidence currently supports text-only input, it was evaluated solely on the text-only question subset (n=111). Consequently, direct three-way comparison across all models is limited to this shared subset, while ChatGPT-5 and Gemini 2.5 Pro were additionally evaluated across the full 265-question dataset including image and video-based items.
A standardized prompt structure was used for all interactions to ensure consistency and to minimize variability (Figure 1). The prompt instructed each model to act as an orthopaedic surgery resident, to select the single best answer choice, and to provide a step-by-step explanation for its reasoning. For multimodal models (ChatGPT-5 and Gemini 2.5 Pro), image files were uploaded alongside the text stem for all relevant questions. The chat context was reset after every question to prevent information leakage or memory bias between items.
Evaluation and Scoring
Model outputs were recorded and scored against the official OITE answer key. A response was graded as “correct” only if the model explicitly selected the keyed option. Primary outcomes included overall accuracy (percentage of correct answers) for each model, and secondary outcomes included performance stratified by cognitive taxonomy (T1–T3), question type (text-only vs. image-based), and subspecialty domain.
Statistical Analysis
Statistical Analysis was performed using Stata (StataCorp LLC, College Station, TX). Descriptive statistics were calculated to determine accuracy rates across all categories, with 95% confidence intervals computed for overall performance. Comparisons of accuracy between models were conducted using McNemar’s test for paired proportions and Cochran’s Q test for comparisons across all three models on the text-only subset. Statistical significance was defined as p < 0.05. All graphical visualizations were generated using GraphPad Prism (GraphPad Software, San Diego, CA).
RESULTS
A total of 265 questions from the 2014 OITE were analyzed. Both multimodal models demonstrated performance exceeding the national mean score for PGY-5 residents (74%). Gemini 2.5 Pro achieved the highest overall accuracy at 80.0% (212/265), followed by ChatGPT-5 at 78.1% (207/265). The difference in overall accuracy between the two multimodal models was not statistically significant (p = 0.59) (Table 1).
When stratified by taxonomy, no statistically significant performance differences were observed between the models across any domain. For factual recall (T1) questions, ChatGPT-5 achieved slightly higher accuracy (88.1%) compared to Gemini 2.5 Pro (84.7%; p = 0.59) and OpenEvidence (86.4%; p = 0.87). In contrast, Gemini 2.5 Pro showed numerically higher accuracy than ChatGPT-5 on questions requiring interpretation of images or data (T2) (76.3% vs. 68.4%; p = 0.44), though this data was not statistically significant. Similarly, for complex reasoning (T3) items, Gemini 2.5 Pro demonstrated a slight increase over ChatGPT-5 (79.2% vs. 76.8%; p = 0.60).
On the subset of 111 text-only questions where all three models could be directly compared, Gemini 2.5 Pro and OpenEvidence achieved identical accuracy (84.7%, 94/111), outperforming ChatGPT-5 (81.1%, 90/111). These differences were not statistically significant (p = 0.71) (Table 2). Notably, OpenEvidence performed comparably to the multimodal models despite lacking visual input capabilities, excelling particularly in text-based reasoning (T3) questions (82.7%).
Performance varied by orthopaedic subspecialty but showed no statistically significant differences between models (p > 0.05 for all pairwise comparisons). Both models achieved their highest accuracy in Basic Science (>93%). The largest numerical discrepancy was observed in Oncology, where Gemini 2.5 Pro achieved higher accuracy than ChatGPT-5 by 22.7 percentage points (77.3% vs. 54.5%), though this did not reach statistical significance. Numerical differences were also observed in Foot and Ankle and Hip and Knee, but similarly should be interpreted with caution given the small question counts per domain and lack of statistical significance (Figure 2).
On text only questions, all three models demonstrated highest accuracy in Adult Spine, with OpenEvidence performing in the same range to the multimodal models (Figure 2). Open Evidence achieved a perfect score on the Foot and Ankle section (100%) compared to ChatGPT-5 and Gemini 2.5 Pro (82.5%).
All three models answered 72.1% (80/111) of questions correctly (Figure 4). Only 5.4% (6/111) of questions were answered incorrectly by all three models.
DISCUSSION
This study investigated overall accuracy between ChatGPT-5, Gemini 2.5 Pro, and OpenEvidence across a standardized orthopaedic examination, as well as variation across cognitive taxonomies (factual recall, interpretation, clinical reasoning) and orthopaedic subspecialties. We additionally evaluated if these LLMs achieved performance at or above the PGY-5 resident benchmarks. In addressing these questions, we found that both ChatGPT-5 and Gemini 2.5 Pro exceeded contemporary PGY-5 national benchmarks from 2024 (73%) for orthopaedic residents in ACGME accredited programs, achieving overall accuracies of 78.1% and 80.0%, respectively, with no statistically significant difference between models (p = 0.59) (Table 1) (Orthopaedic In-Training Examination (OITE) Technical Report 2024 2024).
Because OpenEvidence does not support image interpretation, it was evaluated only on the 111 text-only questions. In this shared subset, where all three models could be directly compared, Gemini 2.5 Pro and OpenEvidence each achieved 84.7% accuracy, while ChatGPT-5 Plus slightly trailed both at 81.1%. The absolute differences were small (a 3.6 percentage point difference between ChatGPT-5 and the other two models), and they did not reach statistical significance (p = 0.71), indicating that no model consistently outperformed the others on purely text-based items (Table 2). Across the text-only questions, all three models agreed on the correct answer for 72.1% of items, and only 5.4% were missed by all models, demonstrating substantial overlap in their strengths (Figure 4). While this overlap likely reflects convergence on well-established clinical knowledge, it cannot be excluded that shared exposure to publicly available exam content during model training contributed to agreement on specific items, a possibility that underscores the need for contamination-resistant evaluation designs in future work. OpenEvidence also contributed the highest number of unique correct responses (7 total) (Figure 4). These results indicate that while overall accuracy was similar across models, differences in performance on specific subspecialties within orthopaedic surgery and cognitive tasks reflect variation in how each model handles distinct types of clinical knowledge.
This study is the first to directly compare the performance of three contemporary frontier LLMs (ChatGPT 5 Plus, Gemini 2.5 Pro, and Open Evidence) on the OITE, and the first to evaluate a retrieval-augmented system (OpenEvidence) on this exam format. Prior work has examined older model generations or focused on general medical licensing tests, but no previous study has assessed these newer architectures, head-to-head, on a specialty-specific orthopaedic assessment.
Our findings align with a quickly developing body of work showing that modern LLMs can achieve high performance on structured medical knowledge assessments. Early research demonstrated that GPT-3.5 could attain passing-level performance on licensing examinations such as the United States Medical Licensing Examination (USMLE) without specialty-specific training (Kung et al. 2023). Subsequent evaluations of GPT-4 and GPT-4 Omni documented substantially higher accuracy, frequently exceeding 88-90% correct responses, and, most notably, outperforming earlier LLMs on tasks requiring both factual recall and clinical reasoning (Bicknell et al. 2024; Chen et al. 2024).
Parallel advances have been reported for non-OpenAI systems. In specialty board examinations, Gemini 2.5 Pro has achieved ≥ 85-90% accuracy on national pediatric surgery and obstetrics-gynecology exams, well above local passing thresholds, which were approximately 60% for each assessment (Wielochowska et al. 2025). More recent comparative evaluations of ChatGPT-5 and Gemini 2.5 on internal medicine subspecialty board examinations demonstrate higher overall accuracy for Gemini, while also showing that relative performance varies across subspecialties rather than remaining consistent across domains (Sheikhalishahi et al. 2025). This data suggests that general-purpose LLMs from different model families have converged to a similarly high performance level on structured, multiple-choice assessments, with variability concentrated more in subspecialty-specific reasoning style than in raw score.
Despite these high accuracies reported across prior studies, LLM performance remains imperfect, particularly for questions that require nuanced clinical judgment, integration of multiple pieces of information, or differentiation between closely related management options. These limitations likely reflect the probabilistic nature of current models and their reliance on pattern recognition rather than true causal clinical reasoning. Although ongoing model development may continue to improve performance, consistently perfect accuracy on complex clinical assessments is unlikely in the near term. This highlights the role of LLMs as adjuncts rather than substitutes in educational and clinical decision making.
Retrieval-augmented systems such as OpenEvidence represent a related but distinct trajectory in LLM design. Recent work has shown that these LLMs, augmented with targeted retrievals and agentic workflows, can provide more evidence-grounded and therefore clinically actionable responses to real-world clinical questions than base models can alone (Low et al. 2025). In medical education settings, OpenEvidence and similar platforms have been reported to assist students integrate guideline-based reasoning and efficiently locate relevant literature during clinical rotations (Patel et al. 2025). OpenEvidence’s strong performance in our text-only OITE questions is consistent with these findings and suggests that retrieval-based reasoning can perform as well as or better than general multimodal models on exam-style clinical questions, particularly when questions rely on standards of care (Table 2). A qualitative review of model-generated explanations for incorrect responses revealed potentially meaningful differences in error character across models. ChatGPT-5 incorrect responses most commonly reflected confident factual errors, in which the clinical reasoning framework applied was appropriate, but the specific conclusion was incorrect. In several cases, the model appeared to apply more contemporary clinical guidelines that diverge from the 2014 examination key, which may itself reflect the model’s training on more recent literature. Gemini 2.5 Pro’s errors more frequently reflected misinterpretation of the question stem or application of overly aggressive clinical judgement despite demonstrating sound underlying knowledge. In multiple instances, it explicitly acknowledged alternative correct approaches within its own explanation. OpenEvidence’s incorrect responses showed a qualitatively different pattern: explanations were either absent or consisted of brief, citation-grounded statements that described correct clinical principles yet arrived at the wrong answer choice, suggesting errors more consistent with answer-selection limitations than reasoning or knowledge failures. This pattern is consistent with OpenEvidence contributing the highest number of unique correct responses among the three models (n=7, Figure 4), suggesting that when it diverged from the other models it did so on the basis of evidence-grounded reasoning rather than pattern-matched responses. While this analysis was qualitative and exploratory rather than formal, these differences in error character may have practical implications for clinical use, as the nature and transparency of model errors may be as important as aggregate accuracy when clinicians rely on these tools for point-of-care decisions.
Subspecialty analysis further highlighted meaningful differences among models. Within the text-only subset, OpenEvidence achieved 100% accuracy on Foot and Ankle questions, whereas ChatGPT-5 Plus scored 77.3% followed by Gemini 2.5 Pro scored at 68.2% here (Table 3, Figure 3). While these differences did not reach statistical significance, this pattern may warrant further investigation into whether retrieval-based systems offer advantages in subspecialties with well-established diagnostic and treatment pathways.
Differences also emerged across cognitive taxonomies. ChatGPT-5 Plus performed best on Taxonomy 1 (recall-oriented) items, aligning with its broad exposure to factual medical content and its strength in direct knowledge retrieval. Gemini 2.5 Pro fared relatively better on Taxonomy 3 (clinical reasoning) questions, consistent with reports than newer Gemini models often excel on complex and integrative problem solving tasks (Wielochowska et al. 2025; Sheikhalishahi et al. 2025; Boczkowski et al. 2025). OpenEvidence’s Taxonomy 3 performance was comparable to that of Gemini 2.5 Pro despite its lack of multimodal input, suggesting that explicit evidence retrieval can support reasoning on clinical scenarios involving multiple steps (Table 1 and 2). Regarding image-based performance, Gemini-2.5 Pro demonstrated numerically higher accuracy than ChatGPT-5 on T2 interpretation questions, which include radiographs, MRI, CT, and clinical photograph (76.3% vs 68.4%; p = 0.44). While this difference did not reach statistical significance, it suggests potential variation in visual reasoning capability between model architectures. Given that image-based questions comprise the majority of OITE items, the ability of multimodal LLMs to accurately process medical imaging represents and important and underexplored dimension of their utility for orthopaedic education (Figure 2).
Our findings and those of existing work suggest that contemporary LLMs can match or exceed published PGY-5 benchmarks on structured orthopaedic assessments. However, examination accuracy reflects a narrow dimension of medical knowledge and should not be equated with the clinical reasoning or real-world competency of senior trainees. Furthermore, these results should not be interpreted as evidence that LLMs can replace traditional study methods or expert instruction, as model outputs remain imperfect and require verification against established resources (Kim et al. 2025; Bedi et al. 2025). Rather, their strong performance on standardized examinations suggests that trainees will increasingly encounter accurate automated responses to factual and structures clinical questions. From an educational and assessment perspective, these findings also have implications for exam development. As LLMs continue to perform well on lower-order knowledge and recall-based items (Schubert et al. 2023), greater emphasis on higher-order reasoning, synthesis of clinical information, and judgement-based decision making may better reflect the skills in which orthopaedic surgeons provide the greatest added value and are least replicated by automated systems.
From a practical standpoint, these findings offer several considerations for orthopaedic residents and training programs navigating the increasing availability of AI tools. The comparable performance of OpenEvidence and general-purpose models on text-based questions, combined with its qualitatively distinct error profile, suggests that retrieval-grounded platforms may offer a more transparent and verifiable starting point for evidence-based self-study – particularly for subspecialties with well-established clinical guidelines where sources responses can be thoroughly corss-checked against primary literature. ChatGPT-5 and Gemini 2.5 Pro may offer broader utility across question types including image and video-based items, but their tendency toward confident errors warrants active verification of model outputs rather than passive acceptance. For program directors, the pattern of subspecialty and taxonomy-level variation observed across models suggests that LLM performance is uneven in ways that are not captured by aggregate accuracy alone, and that residents should be coached to use these tools critically rather than as authoritative sources. These findings do not support replacing structured study resources or expert instruction with LLMs, but they do suggest that understanding how different model architectures fail – not just how often – may be important as raw accuracy in guiding appropriate clinical and educational use.
Although this study showed high accuracy in these models, these results should still be interpreted in the context of limitations. The study we conducted was limited to a single OITE exam year (2014), which meaningfully limits generalizability. Orthopaedic surgery has evolved considerably since 2014 across clinical guidelines, implant technology, surgical techniques, and evidence-based management across subspecialties. Questions anchored to 2014 standards of care may no longer reflect current accepted practice. As a result, a model answering correctly by 2014 standards could theoretically provide outdated guidance by contemporary standards, and performance on this exam may not predict accuracy on more recent OITE administrations. Additionally, prior exposure of LLMs to the 2014 OITE during model training cannot be excluded. As a result, the accuracy estimates reported here may reflect, at least in part, familiarity with previously encountered content rather than generalizable clinical reasoning ability. OpenEvidence also does not allow for image interpretation, which only allows us to compare all three models using text only questions. Consequently, we could not directly compare all three platforms on the full exam, including image-based items. Subspecialty analyses were further limited by small question counts per domain, ranging from 16 questions in Hand to 49 in Trauma, which reduces statistical power and limits the reliability of domain-level comparisons. Findings from smaller subspecialties such as Oncology (n=22) and Hand (n=16) in particular should be interpreted with caution. In addition, resident benchmarks were taken from prior literature, and newer model versions such as ChatGPT 5.2 and Gemini 3.0 were not used in our study. We also used the paid versions of these models, so performance may differ for users who utilize the free versions, which may rely on older or smaller models. These factors may influence external validity and limit conclusions made about performance on newer exams, updated model releases, or current clinical standards.
This study should allow future research to build on these findings to address our limitations and improve generalizability. Repeating this analysis across multiple OITE exam years would strengthen generalizability and allow assessment of whether model performance holds across evolving clinical content. Incorporating more recent examinations would also reduce the risk of data contamination given that older publicly available exams are more likely to have been indexed in model training datasets. Similarly, evaluating updated model releases would clarify if there are any differences in reasoning and accuracy across versions and over time. Comparing the free and paid versions of these models would provide insight on the performance and accuracy for users who may not have access to the paid versions. Future work should also compare models in a controlled setting that allows access to external sources versus when only relying on the model’s internal database. To support fair comparison, this design should use the same directions across platforms and include sources used in the answers. Platforms that allow for multimodal interpretation should be used to evaluate image-based questions. This study should be conducted by either limiting comparisons to models capable of interpreting images or by reporting text only and image-based results separately. All these steps would allow for a more updated and clinically relevant depiction of the accuracy of these models and their usefulness to orthopaedic residents.
CONCLUSION
In this study, ChatGPT-5, Gemini 2.5 Pro, and OpenEvidence demonstrated high accuracy on the 2014 OITE, with performance at or above published senior resident benchmarks. While overall accuracy did not differ significantly between models, variation across subspecialties and cognitive taxonomies suggests meaningful differences in how these systems retrieve and apply orthopaedic knowledge. These findings support the use of LLMs as adjuncts in orthopaedic education and assessment, while reinforcing the continued significance of higher-order reasoning and clinical judgment that remain less reliably mastered by automated systems.