Introduction
References play a central role in scientific literature, serving multiple critical purposes. Primarily, they prevent plagiarism by crediting original authors. Citations provide evidence for claims, enhancing the validity of research findings. Additionally, they create a structured network of information, allowing readers to trace methodologies and findings with precision. When studies are frequently cited, they have the power to shape their field by establishing foundational knowledge and driving further research. The importance of citations is indisputable.
The pace of scientific discovery has dramatically accelerated in recent decades, especially considering rapid advancements in generative artificial intelligence (AI), such as ChatGPT (OpenAI 2025). Large language models, often trained on scientific literature, depend on accurate citations to generate dependable responses. The importance of references within scientific literature is once again underscored by the role they play in AI. Yet, existing publications demonstrate some inaccuracies within the published literature, ranging from as low as 13.6% (Gazendam et al. 2021) to as high as 84.1% (Montenegro et al. 2021). Multiple studies examining orthopaedic literature have attempted to quantify inaccuracies within subspecialties including sports medicine (Gazendam et al. 2021), spine (Montenegro et al. 2021), pediatrics (Davids et al. 2010), and foot and ankle (Luo et al. 2013). Error classification methodology varied, with some literature documenting merely the prevalence of errors, while others used more granular methods to additionally differentiate between the severity of each inaccuracy. For example, in two separate analyses of reference accuracy, both Mogull (Mogull 2017) and Montenegro et al (Montenegro et al. 2021). initially determined the prevalence of errors and subsequently classified them as either major or minor. However, to further convolute matters, the criteria for major and minor errors differed between the two papers. Citation fidelity remains crucial, but there is a demonstrated need for a framework to classify the severity of citation inaccuracies.
The primary aim of our study is to assess the prevalence of citation inaccuracies in orthopedic surgery literature. The secondary aim of our study is to propose a pragmatic and reproducible classification system for defining and categorizing citation inaccuracies. We hypothesize the rate of citation inaccuracies in orthopedic surgery literature to be similar to prior studies, and thus we should be cautious about the appropriateness of AI for citation collection and bibliography generation.
Methods
We performed a literature search using the PubMed database to assess for prior reports on citation accuracy as well as citation accuracy definitions. Search terms included a combination of “citation accuracy,” “citation accuracies,” “citation inaccuracy,” “citation inaccuracies,” “definition,” and “orthopaedic surgery,” with and without quotations. No timeline restriction was applied. Citation error definitions were combined to create a comprehensive and novel citation accuracy classification, adopted predominantly from the works of Choi, et al. and Mogull (Table 1) (Mogull 2017; Choi et al. 2021). Our goal was to assess the level of accuracy of the information cited, rather than errors in article location (i.e. journal name, volume, edition, page numbers, etc.)
The first issue of 2024 from the American edition of The Journal of Bone and Joint Surgery (JBJS, Volume 106, Issue 1) was selected for review. All 229 citations from all seven scientific articles from this issue were included in our study with no exclusions (Larson, Marks, Gonzalez Sepulveda, et al. 2024; Anastasio, Kim, Wixted, et al. 2024; Skolasky et al. 2024; Richardson et al. 2024; Harris et al. 2024; MacKechnie, Shearer, Verhofstad, et al. 2024; Singh, Jolissaint, Kohler, et al. 2024). The full-text articles for each citation were obtained electronically via institutional access or interlibrary loan methods and assessed for citation accuracy via manual review. If a secondary source was cited, the original source was searched for citation accuracy. For example, if paper A cited a statement from paper B when paper B was quoting paper C, then paper C was also searched for the information stated in paper A.
Each citation was given an alpha-numeric rating based on our novel classification system (Table 1). The classification scale ranges from 1 to 5 and citations are further delineated as accurate, minor errors, or major errors. Class 1 includes citations that are accurate and support the primary author’s claims and/or their statistics. Class 2 errors include secondary citations and are subcategorized into 2A and 2B. A class 2A error signifies accuracy from the primary source but does not earn a class 1 designation due to citation of a secondary article. Class 2B errors include an oversimplification, overgeneralization, or trivial inaccuracy from either the primary or secondary source. Class 3 errors also include inaccurate statistics or population applications, subclassified into minor or major errors. A class 3A error is equivalent to a class 2B error, but from a primary source. A class 3B error involves statistical misrepresentations, inaccurate population applications, and contradictions. If direct or implied statements were absent or not supported, then a class 4 error was designated. This also included citations that were numbered or placed incorrectly in the manuscript, as such errors would prevent AI from accurately identifying the correct source. Lastly, a class 5 error includes statements that are contradictory to or refute the cited paper’s conclusion. This is a stronger contradiction than may be observed in a class 3B error, one only based on a simple statement. Minor errors include error classes 2B and 3A that contain an oversimplification, overgeneralization, or trivial inaccuracies. Major errors include error classes 3B, 4, and 5, and fail to substantiate claims, are contradictory, or are absent entirely. Citations were not classified by typographical errors of the reference section as all cited publications were able to be obtained.
Error rates and 95% confidence intervals were calculated using Microsoft Excel (Redmond, VA). Significance was determined as p < 0.05. A 2-proportion Z-test was used to assess for association of degree, professorship, and training status of the first, senior, and corresponding authors with citation error rates. Finally, the citation accuracy classification system was entered into ChatGPT (OpenAI 2025) and OpenEvidence (“OpenEvidence” 2025) to test AI’s application of the rubric for these same seven articles.
Results
All full-text articles from the 229 citations referenced by the seven scientific articles in the first issue of JBJS in 2024 were reviewed for accuracy (Larson, Marks, Gonzalez Sepulveda, et al. 2024; Anastasio, Kim, Wixted, et al. 2024; Skolasky et al. 2024; Richardson et al. 2024; Harris et al. 2024; MacKechnie, Shearer, Verhofstad, et al. 2024; Singh, Jolissaint, Kohler, et al. 2024). First authors included an attending physician, research professor, director of global programs, three residents, and a medical student. Five of the first authors had a doctorate degree. Senior authors included five attending professors, an associate professor, and an assistant professor, all of orthopaedic surgery. Corresponding authors included four professors of orthopaedic surgery, a research professor, an assistant professor of orthopaedic surgery, and a resident. The median number of named authors was 7, not including additional study groups (range, four to fifteen). The mean number of citations was 32.7 (range, 24 to 48).
Improper citations were noted in 27.5% of all references (63 of 229, 95% confidence interval (CI): 21.7%-33.3%; see Table 2, Figure 1). This included minor errors, major errors, and secondary citations that were accurate. Twenty (8.7%) of the references cited a secondary article, of which 3 were inaccurate (95% CI: 5.1%-12.4%). The total citation error rate was 20.1% (46 of 229, 95% CI: 14.9%-25.3%). A minor error was found in 25 references (10.9%, 95% CI: 6.9%-15.0%) and a major error in 21 references (9.2%, 95% CI: 5.4%-12.9%).
Citations were given a class 1 rating for accuracy 72.5% of the time (166 of 229 references, 95% CI: 66.7%-78.3%) with an additional seventeen accurate citations that were from secondary sources for a total accuracy rate of 79.9% (183 of 229, 95% CI 68.7%-80.0%). The rate of a class 2B error was 1.3% (3 of 229, 95% CI: -0.2%-2.8%). A class 3A or 3B error was observed in 9.6% (22 of 229, 95% CI: 5.8%-13.4%) and 0.4% (1 of 229, 95% CI: -0.4%-1.3%) respectively. We observed seventeen (7.4%, 95% CI: 4.0%-10.8%) class 4 errors and three (1.3%, 95% CI: -0.2%-2.8%) class 5 errors.
There was no statistical significance reached to show an association between degree or training level of the first author and any error type or prevalence. Additionally, there was no association between senior author professorship and citation inaccuracy prevalence. All senior authors were attending orthopaedic surgeons, therefore, no calculation for association of training type was indicated. Lastly, there was also no association between professorship or training level of the corresponding author and error prevalence (Table 3).
ChatGPT self-reported the ability to evaluate the accuracy of citations by “direct comparison of the quoted text to the original text,” and via “context and nuance checks (misrepresented conclusions, ignored limitations, exaggerative language)” (OpenAI 2025). It could also apply our provided rubric for citation review. However, it could not directly verify this information against the original 229 source citations and instead provided accuracy ratings based on a “logical fact-check using well-established knowledge and standard authoritative sources.” (OpenAI 2025) Additionally, although ChatGPT requested the full-text PDFs of each reference for a more in-depth citation review, the platform did not allow file uploads, which ultimately made a thorough source verification impossible, therefore, no comparison of human versus AI review using Chat GPT was possible. The OpenEvidence platform was unable to apply the classification system entirely.
Discussion
We found a total citation error rate of 20.1%, with 9.2% categorized as major errors. This aligns with previous studies on the prevalence of citation inaccuracies. As such, we advise caution when relying solely on AI for citation collection and bibliography generation, given the inherent human errors present in scientific literature. Additionally, we identified multiple definitions of citation inaccuracy, contributing to inconsistent representations of citation error prevalence. To address this, we propose a novel classification system for citation accuracy that can be easily applied across all scientific disciplines (Table 1).
Four authors participated in the citation accuracy review. Before the formal evaluation, each author independently assessed the same ten citations to establish inter-observer reliability. Given the subjective nature of the classification system, we encountered varying opinions regarding appropriate error designations. To address this, we agreed to classify errors by selecting the more severe designation when two ratings were deemed applicable. Ultimately, we concluded that the most significant contributions of our study lie in the introduction of a novel classification system and the identification of errors, rather than solely quantifying the exact incidence of a specific error type. While we recognize that the subjectivity inherent in our classification may limit reproducibility with exact precision, it highlights the need for ongoing dialogue and refinement in the assessment of citation accuracy.
In scientific literature, authors hold primary responsibility for ensuring the accuracy of citations, a task that demands both diligence and integrity (Cowell 1998). Once an inaccuracy is published, it may be inadvertently propagated, particularly when subsequent authors lack full-text access due to licensing restrictions or cost barriers. As artificial intelligence continues to permeate academic workflows, it is important to recognize its potential limitations in this context. AI, such as ChatGPT, may generate inaccurate reference lists due to pre-existing human errors. The validity of the articles it compiles can also be questionable, since AI may not be able to verify the accuracy on its own. Recent studies have shown that between 20.6% and 98.1% of articles compiled by various ChatGPT models were fabricated (Chen and Chen 2023), and up to 70% of citations generated by AI-only articles may be inaccurate (Kacena, Plotkin, and Fehrenbacher 2024). These findings underscore the importance of authors to maintain “citation integrity” (Howard and Englesbe 2022) and to implement appropriate oversight to prevent AI tools from compounding existing challenges in reference accuracy.
This study further illustrates both the promise and the current limitations of applying AI to citation accuracy assessment. While ChatGPT was able to interpret and apply our structured classification system, it lacked the ability to cross-verify citations against the original 229 source documents. As a result, accuracy ratings were derived from general knowledge and logical inference rather than direct evidence from the primary literature. Although the platform indicated that full-text access would enable a more robust audit, the inability to upload source files remains a significant constraint. OpenEvidence can access full-text scientific literature, however, it was unable to apply our classification system. This highlights an important gap between emerging AI tools and the practical requirements of rigorous citation verification. Future integration of AI tools with institutional full-text databases or linked research repositories may help bridge this gap, enhancing the feasibility and rigor of AI-assisted citation verification within the literature.
Conclusion
In this study, we present a novel citation accuracy classification system that has the potential to be applied across scientific disciplines. Our findings confirm that citation inaccuracies remain prevalent in orthopaedic surgery literature, with nearly 10% of references containing major errors. While artificial intelligence, such as ChatGPT, shows promise for assisting with citation screening, limitations remain, including the inability to directly verify original sources. Recent evidence shows that AI-generated citations may themselves be inaccurate or fabricated, underscoring the critical need for oversight. Authors hold primary responsibility for maintaining citation integrity to prevent the propagation of inaccuracies in future research. Moving forward, combining human diligence with thoughtfully integrated AI tools may help strengthen the accuracy and reliability of scholarly references.