Introduction
Social media platforms are a vital pillar for the query and exchange of information. In 2021, 7 out of 10 Americans used some form of social media for social connection, information, or entertainment. Approximately 84% of Americans aged 18-29 reported use of at least one social media site (Auxier and Anderson 2021). As much as 90% of physicians have social media accounts, and 80% of the patient population uses the internet for searching medical information (Varady et al. 2019). Still, a study of pediatric orthopedists found that physicians may be underutilizing social media presence (Lander et al. 2017; Narain et al. 2021). Likewise, when searching for radiation oncologists by names using Google (Alphabet Inc., Mountain View, CA USA) search engine, one study found that patients are predominantly exposed to material curated by third parties, in contrast to material generated and controlled by physicians themselves (Prabhu et al. 2017).
In particular, the video media platform TikTok (ByteDance Ltd., Beijing, China; formerly known as musical.ly) has become increasingly relevant. In 2022, TikTok was the most downloaded application worldwide, boasting over 1 billion active users with an average use of 52 minutes per day (Paul, n.d.; Comp, Dyer, and Gottlieb 2020a). Among US adults, 10% reported using TikTok as a regular source of news, tripling the amount previously reported in 2020; 67% of US teens reported using TikTok in some way (Vogels, Gelles-Watnick, and Massarat 2022). Hashtags using the terms “medicine” and “doctor” on the TikTok platform received 1.4 billion and 6.7 billion views respectively (Comp, Dyer, and Gottlieb 2020a).
Physicians and patients alike use social media platforms for researching and discussing medical topics. A study of foot and ankle orthopedic surgeons determined that physicians who generated content on a YouTube channel had significantly higher ratings when compared to physicians without a YouTube channel (Garofolo-Gonzalez et al. 2021). Another investigation into the role of social media in pediatric orthopedists found that none of the investigated physicians were active users of the TikTok platform (Chiang et al. 2022).
With the vast supply of information available on TikTok, there is a rising concern for the spread of misinformation. A known limitation of social media platforms is the lack of scientific oversight, which is an avenue for dissemination of false information (Gottlieb and Dyer 2020). This gives influencers an inflated impact on information spread given their large follower base, regardless of quality of information (Eghtesadi and Florea 2020). TikTok has been reported on for its contribution to the spread of misinformation, both within the field of medicine as well as other fields (Paul, n.d.). Contributing factors to the spread of misinformation are multifold, including financial incentives from third parties, peer endorsement, and general limitations of knowledge. Of note, approximately 27% of physician content creators were discovered to have undisclosed financial conflicts of interest according to a study of EM physicians (Niforatos, Lin, Narang, et al. 2019). In the case of social media influencers, peer endorsement alone is enough to gain viewership, meaning that information spread may be subject to biases or is just fully incorrect (De Veirman, Hudders, and Nelson 2019).
Both Facebook and Instagram (Meta Platforms, Inc., Menlo Park, California, USA) have been evaluated for their current roles in medical education, as well as their potential use as tools in the future. While there are limitations to the use of such platforms in this context, their large audience and wide accessibility make them compelling avenues (Pander et al. 2014; Shafer et al. 2018). Further, the use of social media platforms as educational tools in medicine is compelling due to the format of content delivery. TikTok limits video time to 60 seconds for primary posts, which sets it apart from other video media content platforms previously analyzed including YouTube (Comp, Dyer, and Gottlieb 2020a). In recent studies assessing adult learning, the average learner’s attention span tapered off after about 15-20 minutes, at which time working memory and interference contribute to a drop in effective learning (Cooper and Richards 2017).
With the steady increase in use of the TikTok platform as a source for information, there is interest in understanding how available content is interacted with. In addition, the characteristics of material itself may shed light on underlying video qualities that determine popularity and impact. The purpose of this study was to explore popular orthopedic TikTok videos, classify based on content type, and assess for the presence of bias.
Methods
Social media platforms including TikTok use hashtags(#) to increase content interaction and direct searches for specific topics (O’Brien 2022). Hashtags can also be used to categorize content. By typing in specific hashtags, users can view videos within areas of interest. The resulting content is sorted according to TikTok’s proprietary algorithm, which considers view count, like count and comments. Using a new account created on October 30, 2022, the TikTik platform was investigated using the search criteria “#orthopedicsurgery.” TikTok makes use of a personalized recommendation algorithm which considers user history, search history, likes, and other personal information when curating content (Zhang and Liu 2021b). As such, a new account was created on a cleared browser on a shared university computer using a new email address in order to control for specific viewer impact on search results. The inclusion criteria were limited to videos published in English by creators using the hashtag “#orthopedicsurgery.” The top 100 results meeting the inclusion criteria were assessed for the study.
The resulting videos were assessed qualitatively and sorted into one or more of the following categories: entertainment, educational, procedural, or lifestyle. This was done according to the methodology set by a similar investigation into neurosurgerical content using 3 reviewers (McBriar et al. 2022). Three reviewers (H.T., K.T., O.A.) viewed content together, assessed the content criteria individually, and discussed each video and came to a consensus about the perceived goal of each video as intended by the creator. Any discrepancies were reviewed with the senior author for definitive decision. Entertainment was defined as any video that made use of popular trends, sounds, and hashtags to achieve the purpose of humor or commentary meant to entertain the audience. Educational videos were classified as videos that used teaching to communicate information about a topic to the audience, including specific orthopedic subspecialties. The procedural videos were identified as videos that described an orthopedic procedure in detail or showed an operative video. Lifestyle videos included “day-in-the-life” or informative videos about the daily life of orthopedic surgeons. In addition to qualitative measures, the date posted, the number of views, number of likes, number of hashtags, hashtags used, and number of comments were recorded for each video.
Features of the content creators were also characterized. The account type was categorized into: orthopedic surgeon, non-orthopedic surgeon healthcare provider, non-physician healthcare provider / medical student, or non-healthcare provider. The creators’ account name along with total number of likes and followers were also recorded.
Alternative characteristics of the video were also recorded including use of overlying audio, duet feature, and stitch feature. The TikTok platform allows for the overlay of popular audioclips onto content to connect with other content that uses the same sound. This overlying audio can include narration, music, or other sounds. The duet feature allows 2 audiovisual clips to be displayed side by side, allowing for the attachment with other content on the same screen. The stitch feature allows creators to attach their audiovisual clips to other content. Videos that made use of multiple tools, or satisfied the criteria for more than one category were recorded as having “more than one” feature, and the multiple characteristics were reported. Such videos were recorded within each of the appropriate categories; for example, a video could be considered both educational and entertainment, while making use of overlying audio and duet.
Initially, video rank, defined as the order of appearance under the search criteria “#orthopedicsurgery”, was considered as a representation of video popularity and engagement. It was, however, weakly correlated with number of likes. Due to the relatively low correlation coefficient (0.26) between video rank, as originally defined, and video likes, a new metric was assigned according to the total user engagement rate with the video (adapted from a model used for Instagram (Yew et al. 2018):
\[\begin{aligned} &\quad\quad\quad\quad\quad\quad TikTok\ Engagement\ Score \\&= \frac{(Number\ of\ Likes\ + \ Number\ of\ Comments)}{Number\ of\ Followers\ x\ 1,000}\end{aligned}\]
Additional measures that were considered for analysis included video total like count and video total view count. The TikTok engagement score, along with the rank, like count, and view count were utilized in analysis.
The DISCERN criteria is a tool widely described and accepted for the assessment of quality of written information about the treatment of a health problem. While TikTok is not a platform that curates content in the written form, several studies have used this metric to assess video quality (McBriar et al. 2022; Ward et al. 2019; Song et al. 2021). The DISCERN metric is divided into a sequence of 15 questions, each score on a scale from 1 to 5 points (1 corresponding to no and 5 corresponding to yes). A high DISCERN score indicates that a consumer can reliably learn from high-quality information about a disease process or treatment. Lower scores align with less reliability, greater bias, and less useful information. While the intended information is not a treatment modality, the tool has been adapted to assess the specific topic being discussed. As such, the treatment specific questions were most relevant in videos that described a procedure or treatment course. Three independent raters [H.T., K.T., and O.A] reviewed the audiovisual content and used the DISCERN metric to rate the overall quality of the videos. To assess for the reliability between the three independent raters, a 2-way mixed-effects model was selected according to the dataset and the number of raters (3) (Koo and Li 2016). Total DISCERN scores were used for each video to classify the content quality with scores of 15-26 as “very poor” quality, 27-38 “poor” quality, 39-50 “fair” quality, 51-62 “good” quality, and 63-75 “excellent” quality (a total maximum score was equal to 75 points, and a minimum total score was equal to 15 points). The DISCERN score classification system was adapted from the DISCERN tool handbook. Following the methodology set by Shrout et al, the intraclass correlation coefficient for reliability was calculated via the two-way mixed effects consistency model used to assess inter-reviewer reliability for the given number of reviewers (3).
One of the goals of this paper is to evaluate whether specific creator characteristics correlate with higher engagement. The creator type was correlated with video rank, engagement score, view count and like count using multivariate linear regression. Additionally, multivariate linear regression was used to identify whether video aims or video characteristics were correlated with these variables (rank, engagement score, view count and like count).
Data was recorded using Microsoft Excel, and statistical analysis was performed using MATLAB R2022a (Mathworks Inc., Natick, MA, USA). The data and script used for statistical analysis are available upon request.
Results
Analysis of Videos
The top 100 TikTok videos using “#orthopedicsurgery” that satisfied the inclusion criteria were reviewed. The videos were posted between August 23, 2020 and August 26, 2022. The search was conducted on October 30, 2022. The data set includes 100 videos, totaling 61 minutes of content with over 184.5 million views, 1.24 million likes, 148,181 comments, and 1,110 hashtags. Of these 100 videos reviewed, 37 were entertainment, 42 were educational, 17 were procedural, and 19 were lifestyle. The mean duration of video length was 37.06 ± 36.34 seconds. There were 77 videos that made use of overlying audio, 4 using the duet feature, and 9 stitch videos. The most commonly used hashtags in addition to #orthopedicsurgery included: #orthopedics, #surgery, #fyp (for you page, a commonly used phrase describing the application launch page), and #medical. The mean number of hashtags used was 11.1 ± 9.19 . The cumulative video characteristics are described in Table 1 and Supplementary Table 1. A breakdown of characteristics of the 6 accounts that posted the most content with “#orthopedicsurgery” are detailed in Table 2.
Investigation of Factors The Correlate to Higher Video Engagement
Video rank was a measure established by the reviewers indicating the order in which the video appeared under the search criteria “#orthopedicsurgery”. Additional metrics included TikTok engagement score, video view count, and video like count were additionally recorded. Separate linear regressions were performed to assess for correlation between video category and video engagement. A generalized linear regression model was generated for each of the cases described in Table 3. There were not enough videos classified as Non-physician HCP / medical students (n=4) to generate models for this group. A linear regression correlated to rank demonstrated that videos of the procedural category were of higher rank (B=-18.40 P=0.013) and that video hashtags correlated with lower rank (B=1.023 P=0.001). In a linear regression model predicting engagement score, the orthopedic surgeon and non-orthopedic surgeon HCP groups were correlated with lower engagement scores overall (B=-0.043,P<0.001 and B=-0.049,P<0.001). The video goal and video characteristics did not appear to correlate with engagement score. In a linear regression model for video views, there did not appear to be a correlation between account type, video goal, or video characteristics and video views. The linear regression model for video likes demonstrated that non-orthopedic surgeon HCPs were correlated with lower like-count (B=-290,495 and P=0.013). Lifestyle as video goal was a predictor for higher like count (B=156,411, P=0.036). Other video characteristics did not appear to be correlated with like-count.
Comparison of Content Validity
Initially, 101 videos were analyzed. 1 video was excluded due to the content not being published in English. The remaining 100 videos were assessed using the DISCERN criteria. The intraclass correlation coefficient for consistency in DISCERN is 0.9727 (95% confidence interval 0.9619-0.9808) indicating excellent integrated reliability (Koo and Li 2016). A Kruskal-Wallis H test was employed to evaluate whether there was any difference in DISCERN scores across reviewers (P=0.3135). Additionally, the Kruskal-Wallis H test was used to identify any differences between creator types and DISCERN scores (P=7.89968743406009e-08). The mean DISCERN score for videos generated by orthopedic surgeons was 33.48 ± 6.71. The mean DISCERN score for non-orthopedic surgeon physicians was 26.74 ± 3.29. The mean DISCERN score for videos generated by non-physician healthcare providers or medical students was 26.33 ± 1.66. The lowest mean DISCERN score was recorded for non-healthcare provider creators with 25.62 ± 5.12. The mean rank for videos curated by orthopedic surgeons was 47.046 (n=65), the mean rank for videos created by non-physician healthcare providers/medical students was 37.25 (n=4), the mean rank for videos posted by non-orthopedic surgeon physicians was 59.04 (n=24), and the mean rank for videos created by non-healthcare providers was 60.86 (n=7).
Discussion
The use of social media as a tool for communicating healthcare knowledge to patients and providers has been proven across multiple fields. Specifically, the available content in surgical fields such as neuro, plastic, and aesthetic surgery have been previously reported (McBriar et al. 2022; Das and Drolet 2021; Om et al. 2021). There is, however, a paucity in literature describing the utilization of platforms by orthopedic surgeons and providers. To date, no studies, to the authors’ knowledge, have investigated the orthopedic surgery material currently available on TikTok.
The present study demonstrated an abundance of content generated by physicians and nonphysicians discussing topics related to orthopedics. The videos analyzed yielded 188,015,043 total views, indicating that content tagged with #orthopedicsurgery was seen by a far-reaching audience. There were 12,500,170 total likes, demonstrating high user interaction. Leveraging a platform of this scale can be revolutionary in sharing information amongst professionals and distributing knowledge to patients and families. The style of content generated varies widely, from informational procedural videos to entertaining anecdotes or patient testimonies. With such a breadth of available content, institutions may consider encouraging physicians and patients to normalize the use of platforms such as TikTok to access a wide audience. A seemingly important component of video engagement was associated with account popularity.
A total of 44 of the 100 screened videos were generated by the 6 top accounts summarized in Table 2. These accounts were curated by orthopedic surgeons as well as non-surgeon healthcare professionals. It is of note, however, that orthopedic surgeons and non-surgeon HCPs were associated with lower engagement scores. Additionally, non-surgeon HCP’s were associated with lower like-count. The implication of this finding implies that higher quality content generated by healthcare professionals received lower engagement by generic viewers. As such, videos generated by non-HCP users using #orthopedicsurgery received more engagement. According to the measure of content validity, orthopedic surgeon type accounts generated the highest validity content, while non healthcare provider accounts demonstrated the lowest validity content, which raises concern for the quality of information most widely consumed. There is a well-studied theory of “social proof” in which individuals are susceptible to cognitive bias where they conform to beliefs of larger groups. This has been applied to social media in the context of likes and engagements where it was found that a user is more likely to believe content with higher numbers of likes and comments to be more valid compared to content with less interaction. In the context of healthcare, it is important for clinicians to identify such bandwagon effects and work to combat them with clinical reasoning and patient education (O’Connor and Clark 2019).
Studies have demonstrated the benefits of strong social media presence by providers and organizations, as 57% of patients reported social media influencing provider choice and 81% associating strong social media presence as an indicator of cutting-edge medical technology (Ventola 2014). The rise in social media engagement in healthcare also indicates a need for organizational guidelines when members generate and distribute content. Adapting this knowledge to the field of orthopedic surgery creates an opportunity for patients to view and interact with content surrounding common diagnoses and procedures outside the setting of the clinic or hospital.
Additionally, patients can familiarize themselves with physician specific advances in technology and areas of expertise to better guide provider selection. An estimated 41% of TikTok users are between the ages of 16 and 24 (Comp, Dyer, and Gottlieb 2020b). This poses an exciting opportunity for orthopedic surgeons and departments to connect with younger audiences. Analysis of engagement with younger audiences have covered relevant healthcare topics such as COVID-19 information distribution, as well as characterization of e-cigarette content (Li et al. 2021; Xie et al. 2023). Similarly, orthopedic surgeons can consider similar strategies in discussing relevant topics for adolescent and young adult consumers including scoliosis (Truumees et al. 2021). Such strategies can act as primary and secondary prevention for chronic conditions in young audiences in a format that is widely accepted and used.
This study was limited by a number of factors. To control for differences in individually recommended content on the TikTok platform, the videos were viewed and analyzed by 3 reviewers on the same device concurrently using a cleared browser, new user account, and email. While these measures were effective in minimizing individual user bias, it was not representative of the standard use case of the platform. The TikTok video recommendation algorithm considers user interests, videos, and interaction history when promoting videos to specific users (Zhang and Liu 2021a). Therefore, the use of a user naïve search discounted the impact of specific user metrics on the type of content generated within a given search. Similarly, the search was conducted on one day on a single device in order to access a cross section of the content available including “#orthopedics”. As a result, the findings of this study do not consider changes in content over time or variability in content that is recommended across different days. Further, the search results yielded only 4 videos classified as “Non-physician HCP/medical student”. This limited the ability to extrapolate insights on the videos generated by these users.
Conclusions
The TikTok platform is used by millions of consumers globally. Ultimately the results of this study indicate that there is an opportunity for healthcare providers and students to curate high quality orthopedic surgery content on the TikTok platform. By leveraging features such as hashtags and popular sounds, accurate orthopedic content may be distributed in a new less-conventional way. It is vital that institutions and physicians adapt to this novel venue for information exchange and content creation to connect with the younger generation of users to continue to provide quality care to patients.
Funding
No external funding was obtained for this project.