INTRODUCTION
ChatGPT (Chat Generative Pre-trained Transformer) is an publicly available artificial intelligence (AI) language model (Kunze et al. 2022). It employs deep-learning techniques and has been trained on a diverse corpus of text, enabling it to generate human-like responses across a wide range of conversational contexts. Since its debut, ChatGPT has undergone significant advancements, evolving from earlier versions with relatively fewer parameters to more sophisticated iterations. Recently, a new feature was added, namely the GPT-4V(vision) system card, enabling it to interpret image-based queries and generate corresponding responses (Zhu et al. 2024). This innovative feature holds promise as it may aid in diagnosing and managing common orthopaedic fractures by integrating patient data, radiological findings, and medical history to generate informed diagnostic suggestions and treatment strategies (Saad et al. 2025). Recent studies, such as Hiredesai et al (Hiredesai et al. 2024)., demonstrated variable agreement in identifying common upper extremity bony pathologies (e.g. distal radius fracture, metacarpal fracture, carpometacarpal osteoarthritis, etc. ) with agreement ranging from 0% to 55%. Hiredesai’s study, among others, demonstrate the potential for ChatGPT; however, additional studies are needed to validate its consistency and reliability across a broader spectrum of clinical scenarios.
Osteolysis is defined as the progressive loss of bone resulting from an inflammatory response, typically initiated by wear particles and other biological mediators that stimulate osteoclast activity (Algan et al. 1996; Blaine et al. 1996; Teitelbaum 2000). In the context of periprosthetic osteolysis, this inflammatory cascade leads to the resorption of bone adjacent to a prosthetic implant, potentially resulting in implant loosening and eventual failure (Goodman and Gallo 2019). Radiographically, periprosthetic osteolysis is characterized by the appearance of radiolucent lines or cavitary defects at the interface of bone with either cement or the implant itself. These findings are integral for early diagnosis and ongoing monitoring of implant integrity as osteolysis may lead to the positional shift of the implant observed on successive radiographs without infection (Hornung et al. 2022; Maloney and Smith 1996). To date, only a single study has evaluated radiolucent lesions in field of dentomaxillofacial radiology with ChatGPT, albeit with reportedly low accuracy (Silva et al. 2024). The application of ChatGPT in orthopaedic radiology, particularly in detecting periprosthetic radiolucency, may provide a valuable tool for enhancing diagnostic accuracy and assisting surgeons in decision-making. As the number of shoulder arthroplasty procedures continues to rise in the US, the importance of recognizing osteolysis post-arthroplasty grows (Kunze et al. 2022; Farley et al. 2021).
The purpose of this study is to investigate the agreement between ChatGPT and a fellowship-trained orthopaedic surgeon in detecting periprosthetic radiolucency on radiographs. Specifically, we hypothesized that although ChatGPT would successfully detect osteolysis on postoperative radiographs, its overall agreement with a fellowship-trained shoulder and elbow surgeon would be inferior.
METHODS
Patient Selection
A total of 130 consecutive cases of failed anatomic total shoulder arthroplasty (aTSA) from the years 2010 to 2024 were retrospectively reviewed from a prospectively maintained implant retrieval database to determine eligibility for inclusion. Clinical data, including patient age, implant duration (time in situ), laterality, and sex, were abstracted from the electronic medical records. Radiographic categorization of stem length was performed, classifying implants as short, standard, or long. Inclusion criteria comprised adult patients with a documented history of anatomic total shoulder arthroplasty and the availability of at least one pre-revision radiograph. Both primary and revision aTSA cases were considered eligible.
Exclusion criteria included the absence of pre-revision radiographic imaging and cases involving non-anatomic arthroplasty types, including stemless total shoulder arthroplasties, reverse total shoulder arthroplasties (rTSAs), and hemiarthroplasties. Of the 130 cases reviewed, 73 met the inclusion criteria and were selected for analysis.
Lastly, each patient was assigned a PT-ID, and the patient information was removed. A blinded researcher was then provided with the radiographs to put into ChatGPT, unaware of the scoring by the orthopaedic surgeon. This ensured that the assessment remained unbiased, allowing for a more standardized comparison of ChatGPT’s agreement with the surgeon. This retrospective study received institutional review board (IRB) exemption. All radiographs and patient data were deidentified prior to analysis, and no protected health information was entered into ChatGPT during the study.
Comparison with Orthopaedic Surgeon Assessment
To benchmark agreement with the surgeon, a board-certified, fellowship-trained orthopaedic surgeon specializing in shoulder and elbow surgery independently evaluated the same set of radiographs. These assessments served as the comparator for agreement analysis and not as an independent diagnostic gold standard.
Radiographic Analysis by ChatGPT
Periprosthetic radiolucency was evaluated using a large language model (ChatGPT) based on a standardized input prompt. The model was prompted to apply the Gruen and Modified Gruen scoring systems corresponding to long and short stem implants, respectively (Gruen et al. 1979). It was provided with detailed zone definitions, annotated reference radiographs, and representative examples. We instructed ChatGPT to analyze periprosthetic osteolysis around the implant in the provided radiographs using these scoring systems. Instructions were consistent for all short stems but utilized the Modified Gruen Zone Scoring system.
To support standardized AI interpretation, we provided ChatGPT with detailed definitions, including the term ‘osteolysis’, first defined by Jacobs et al (Jacobs et al. 2001). in 2001 as the bone loss or deterioration observed around an implant, impacting implant stability and longevity. Additionally, we supplied example radiographs demonstrating both the presence and absence of osteolysis in all zones. These examples were provided as contextual reference examples to guide ChatGPT with both scoring systems and what to look for in each type of radiograph to standardize interpretation across different stem types.
Gruen zone scoring, traditionally used to assess femoral stem loosening in total hip arthroplasty, has been adapted to evaluate the humeral stem in anatomic total shoulder arthroplasty (aTSA). For standard and long humeral stems, the scoring evaluates radiolucencies in 8 distinct zones (Figures 1, 2, 3), using a binary system where 1 indicates observed osteolysis and 0 indicates no osteolysis, with a maximum possible score of 8 if osteolysis is present in all zones (Sperling et al. 2000). Agreement categories were defined based on concordance between ChatGPT and the surgeon as follows: “Agree +” indicates both identified osteolysis in a given zone; “1+” indicates ChatGPT identified osteolysis while the surgeon did not; “1–” indicates the surgeon identified osteolysis while ChatGPT did not; and “Agree –” indicates that neither identified osteolysis.
For short humeral stems, a Modified Gruen Zone scoring system is utilized, which specifically adapts the original framework to accommodate the shorter stem length (Figure 4, 5) (Schnetzke et al. 2016). This system also employs a binary scoring mechanism across 5 designated zones (L1, L2, US, M1, M2), with each zone scored as either 1 (observed osteolysis) or 0 (no osteolysis), culminating in a maximum score of 5.
The orthopaedic surgeon graded this short stem osteolysis as a 5, meaning osteolysis in all zones; ChatGPT gave it a score of 4. We applied this same methodology for all radiographs.
Statistical Analysis
Descriptive statistics were computed, including means and standard deviations (SD) for continuous variables. Group comparisons were performed using two-sample independent t-tests (Student’s t-test) and non-parametric Mann-Whitney U tests, as appropriate. Inter-rater agreement between the AI model and the human evaluator was determined using Cohen’s kappa statistic. Agreement with the surgeon was quantified using standard metrics: agreement, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and false positive rate (FPR). These metrics reflect agreement relative to the surgeon evaluator rather than true diagnostic accuracy. A p-value of less than 0.05 was considered indicative of statistical significance. All statistical analyses were conducted using RStudio (RStudio, PBC, Boston, MA). Subgroup analyses were performed to evaluate the impact of stem type on the detection of periprosthetic osteolysis by ChatGPT.
RESULTS
Baseline Characteristics
73 patients fit the inclusion criteria, with the mean age of the cohort 66.2 (48-72) years. The average time in situ was 88.1 (12-302) months. The majority of the patients were men (n=43,58.9%), and right-sided arthroplasty was more common (n=37). The cohort consisted of 39 patients with long humeral stems, 24 with short stems, and 10 with standard stems.
Comparison with Fellowship-Trained Orthopaedic Surgeon
Inter-rater reliability, as assessed by Cohen’s kappa (κ), demonstrated a wide range of agreement across scoring models (Table 1). Kappa values ranged from -0.067 to 1.000. The highest agreement was observed in Modified Gruen M2 (κ = 1.000) and Modified Gruen Loose (κ = 0.812), followed by Gruen 7 and Gruen 8 (κ = 0.754), Gruen 1 (κ = 0.734), Gruen 6 (κ = 0.686), and Gruen Loose (κ = 0.670). Moderate agreement was observed in Gruen 3 (κ = 0.456), Modified Gruen L2 (κ = 0.437), and Modified Gruen US (κ = 0.441). Fair agreement was seen in Gruen 5 (κ = 0.333), while slight to poor agreement was observed in Gruen 4 (κ = 0.183), Modified Gruen M1 (κ = 0.161), and Modified Gruen L1 (κ = −0.067). The composite Overall Loose category demonstrated substantial agreement (κ = 0.718).
Agreement Metrics Relative to the Surgeon
Agreement metrics relative to the surgeon revealed notable variation across models (Table 2). Agreement ranged from 50% in Modified Gruen L1 to 100% in Modified Gruen M2, with most zones exceeding 85%. Sensitivity demonstrated the greatest variability, ranging from 33% in Modified Gruen L1 to 100% in multiple zones, including Gruen 7, Gruen 8, Modified Gruen M2, and Modified Gruen Loose. Specificity remained consistently high, ranging from 55% to 100%. The false positive rate (FPR) was generally low, with the highest values observed in Modified Gruen M1 (0.45), while most zones demonstrated low false positive rates.
Agreement metrics differed across individual zones. The highest false positive rates were observed in Modified Gruen M1 (0.45), Modified Gruen L1 (0.40), and Gruen Loose (0.25). Lower specificity was observed in Modified Gruen M1 (55%) and Modified Gruen L1 (60%) compared with other zones. No significant difference in agreement with the surgeon or inter-rater agreement was observed based on stem type (p > 0.05). Subgroup analysis revealed that ChatGPT’s agreement with the surgeon remained consistent across long, short, and standard stem types.
DISCUSSION
This study demonstrates that ChatGPT was able to detect osteolysis in some cases, but its agreement varied across zones, with sensitivity ranging from 33% to 100%. When compared to a fellowship-trained orthopaedic surgeon, ChatGPT demonstrated substantial or greater agreement in 5/8 (62.5%) of Gruen zones and 1/5 (20%) of Modified Gruen zones. The highest agreement was observed in Modified Gruen M2 (κ = 1.000), followed by Modified Gruen Loose (κ = 0.812), Gruen 7 and 8 (κ = 0.754), Gruen 1 (κ = 0.734), Gruen 6 (κ = 0.686), and Gruen Loose (κ = 0.670). Moderate agreement was observed in Gruen 3 (κ = 0.456) and Modified Gruen L2 (κ = 0.437). Fair agreement was observed in Modified Gruen US (κ = 0.395) and Gruen 5 (κ = 0.333), while slight to poor agreement was observed in Gruen 4 (κ = 0.183), Modified Gruen M1 (κ = 0.161), and Modified Gruen L1 (κ = −0.067). These findings indicate that ChatGPT’s consistency varies significantly by anatomical zone.
Agreement ranged from 50% to 100%, with most zones exceeding 85%. Sensitivity demonstrated the greatest variability, ranging from 33% to 100%, while specificity remained consistently high (55% to 100%). False positive rates were generally low but were higher in certain zones, including Modified Gruen M1 and Gruen 7. While these agreement-based metrics suggest that ChatGPT may be a useful adjunct to surgeon interpretation, the variability in sensitivity and agreement across zones indicates it is not yet a reliable standalone tool. Importantly, these findings should be interpreted within the context of the study cohort, which consisted exclusively of failed aTSAs from an implant retrieval database. This selected population likely has a higher prevalence of osteolysis and implant loosening than a routine postoperative surveillance cohort, which may influence agreement-based metrics and limit extrapolation of these findings to well-functioning shoulder arthroplasties.
Both scoring systems revealed high overall agreement (50% to 100%) and specificity (55% to 100%), but large variability in sensitivity (33% to 100%). Still, the broader composite categories demonstrated substantial agreement, with Gruen Loose (κ = 0.670), Modified Gruen Loose (κ = 0.812), and Overall Loose (κ = 0.718), supporting ChatGPT’s potential role as a supplemental reviewer in detecting periprosthetic osteolysis. However, further validation is necessary to improve agreement and reliability across all zones.
The distribution of stem lengths and types was fairly balanced, with long humeral stems being the most common. While it is possible that stem type could be associated with the radiographic presentation of osteolysis, our analysis revealed no significant effect on ChatGPT’s agreement across different stem types. The Modified Gruen system, used specifically for short stems, also showed no significant differences in agreement with the surgeon, suggesting that ChatGPT’s agreement was not notably affected by the stem length or type.
A major obstacle in osteolysis research is the lack of standardized definitions and grading systems, making comparisons between studies challenging. Hornung et al (Hornung et al. 2022). identified significant heterogeneity in radiographic assessment methods for glenoid osteolysis, with most relying on subjective, categorical grading and few using quantitative measurements. Our findings similarly highlight the importance of consistent, objective criteria to assess and define osteolysis.
To date, much of the orthopaedic literature involving ChatGPT and other AI tools has centered on patient education (Eng et al. 2025; Lack et al. 2024). The current study is among the first to examine ChatGPT’s utility in radiographic evaluation of orthopaedic pathology. A systematic review by Keshavarz et al (Keshavarz et al. 2024). found that 37/44 studies (84.1%) reported high agreement for ChatGPT in radiology, yet none supported unsupervised use. Of the 24 studies that assessed model agreement, median agreement was 70.5% and agreement 83.6%. Our results align with this trend, with comparable agreement relative to the surgeon (50% to 100%) but slightly lower agreement (62.5% and 20% for Gruen and Modified Gruen, respectively).
One included study from the Keshavarz review assessed ChatGPT’s ability to generate fracture reports using structured templates and found high agreement with expert reports (Bosbach et al. 2024). However, criticisms included lack of medical precision and verbosity, limitations that may extend to its application in identifying osteolysis, a less-defined radiographic finding.
Current evidence suggests that while ChatGPT performs reasonably well when given structured inputs like classification systems, it is more consistent with distinct radiographic pathologies (e.g., fractures) than more subjective findings such as osteolysis (Saad et al. 2025; Bosbach et al. 2024). While it is not yet capable of replacing human expertise, ChatGPT may serve as a useful secondary reviewer, similar to how radiology reports are used to guide interpretation by orthopaedic surgeons. Discrepancies between radiologists and surgeons are common, and ChatGPT may offer additional context in borderline or uncertain cases. Ultimately, it remains the surgeon’s responsibility to determine clinical relevance.
Interestingly, studies in neuroradiology have shown that the standard (text-only) ChatGPT 4.0 outperformed the vision-enabled 4.0V model in image-based tasks (Hiredesai et al. 2024; Horiuchi et al. 2024). This may reflect the complexity of medical image interpretation and differences in training data. It underscores the need for standardized inputs, especially in uncontrolled clinical environments.
Prompt engineering and pretraining are crucial to maximize AI utility (Meskó 2023). Minor changes in input wording can significantly alter outputs. Tools like RadLex, developed by the RSNA, offer standardized terminology that may improve AI performance by minimizing ambiguity and aligning output with radiologic conventions (Keshavarz et al. 2024). Until such standardization is widespread, variability will continue to hinder consistent clinical application.
Lastly, the false negative rate in our study illustrates a major concern: AI may overlook pathology, leading to missed diagnoses and poor outcomes. While AI offers speed and scalability, it must be supervised, especially when variability in sensitivity ranges from 33% to 100%. Notably, ChatGPT did not undergo iterative learning during this study; all analyses were performed using a fixed prompt with standardized definitions and reference examples, and agreement did not change over time. While representative contextual examples of osteolytic and non-osteolytic radiographs were provided during prompt development, false positives and false negatives were still observed, highlighting limitations in pattern recognition despite structured input.
A key limitation of ChatGPT identified in our study was variability in interpreting glenohumeral osteolysis following aTSA. While some reads were consistent with expert interpretation, others were poor in quality. This inconsistency limits its immediate clinical utility, particularly for general orthopaedic surgeons less specialized in shoulder arthroplasty. However, agreement has improved with successive ChatGPT iterations, suggesting future potential as a decision-support tool (Temperley et al. 2024). Additionally, this cohort consisted exclusively of failed anatomic total shoulder arthroplasties undergoing pre-revision evaluation, which may overrepresent the prevalence of osteolysis and limit the generalizability of PPV, NPV, and overall agreement metrics to routine surveillance or well-functioning arthroplasty populations. The single-center, retrospective design may further restrict external applicability. Furthermore, the use of a single fellowship-trained orthopaedic surgeon as both the comparator and reference standard limits the robustness of the findings and precludes assessment of interobserver variability. In the absence of a true independent gold standard, the reported sensitivity, specificity, PPV, and NPV should be interpreted as measures of agreement with the surgeon rather than true diagnostic agreement. Future studies incorporating multiple surgeons and standardized reference criteria are needed to strengthen the validity and generalizability of these findings.
CONCLUSION
This study found that ChatGPT demonstrated variable agreement with a fellowship-trained orthopaedic surgeon in identifying periprosthetic osteolysis, with lower agreement observed in more complex scenarios. The cohort consisted exclusively of failed anatomic total shoulder arthroplasties from an implant retrieval database, which may overrepresent the prevalence of osteolysis and limit the generalizability of PPV, NPV, and overall agreement metrics. These results highlight the need for ongoing improvements in AI technologies and careful integration into clinical workflows to ensure they enhance rather than replace the expertise of healthcare professionals.