Introduction
Orthopaedic surgery is one of the most competitive medical residencies to match into. It requires competitive USMLE (United States Medical Licensing Examination) scores, a high class rank, multiple research experiences, superior performance on clinical rotations and strong letters of recommendation (Andriole et al. 2002; Andriole, Whelan, and Schechtman 2003; Rinard et al. 2010). With the rising difficulty of matching into orthopaedic surgery, there is an increasing number of medical students who fail to successfully match on a yearly basis (Schrock et al. 2017). In 2020, Chen et al. reported that the match rate for orthopaedic surgery in 2016 was 75% which ranks the lowest out of all listed specialties, while on average the number of applicants per residency position was 124, ranking orthopaedic surgery the highest out of all other specialties (Chen et al. 2020). Studies have shown that a recent trend amongst orthopaedic surgery residency programs has been to place a greater emphasis on objective criteria like Step I and II scores, American Orthopaedic Association (AOA) status, and research productivity in their evaluation of applicants, when compared to previous years (Andriole et al. 2002; Andriole, Whelan, and Schechtman 2003; Rinard et al. 2010; Chen et al. 2020; Karnes, Mayerson, and Scharschmidt 2014). It is likely that objective criteria such as Step II scores, AOA status, and research productivity will be even more important with the change of Step I to a pass/fail grading system.
Due to the competitiveness and heavy reliance on objective criteria for the evaluation of orthopaedic residency applicants, interested medical students must emphasize building objectively sound but also well-rounded academic portfolios for the residency application process. This is no small feat; therefore, medical schools must be adequately prepared to provide their students with sufficient mentorship, as well as scholarly and clinical opportunities that will help them build competitive academic portfolios.
The primary aim of this study was to evaluate students’ satisfaction with our institution’s provision of resources to assist them in obtaining an orthopaedic surgery residency position. Furthermore, we sought to identify areas of strength and weakness within our department. With this information we intend to institute action plans to emphasize our institution’s current strengths and address weaknesses to improve the experience for future students.
Methods
This is a prospective cross-sectional survey of alumni from our institution, an allopathic medical school affiliated with a tertiary-care university teaching hospital who successfully matched into an orthopaedic surgery residency program between 2006 and 2021. Alumni contact information was acquired from an internally-managed database. Students are given the opportunity to opt out of sharing their match information and thus the data set may not include the full list of matched applicants.
An anonymous survey was designed to capture the perspectives of alumni satisfaction with various aspects of our institution’s orthopaedic education program including: overall quality of orthopaedic education, adequacy of mentorship and guidance throughout the residency application process, orthopaedic faculty and residents’ responsiveness to students seeking opportunities in shadowing/research/mentorship, ability to form strong personal connections with at least one member of the orthopaedic faculty, adequacy of feedback on elective rotations and sub-internships, and the adequacy of the 4th year orthopaedic elective/sub-internship in preparing students for away rotations. Current Robert Wood Johnson Medical School second-, third-, and fourth-year medical students along with program faculty collaborated to create the survey questions. Contributors at varying stages of medical education allowed for a comprehensive view of the most important factors contributing to orthopaedic education and application success. Respondents’ year of graduation was also recorded. The survey was distributed to the cohort between June 20th and July 9th, 2021. Follow-up emails were sent to alumni who had not yet completed the survey (for a total of three times). This study was approved by the New Brunswick Health Sciences Institutional Review Board.
Statistical Analysis
Survey data was collected and analyzed using RedCap (hosted at Rutgers Robert Wood Johnson Medical School). REDCap is a secure, web-based software platform designed to support data capture for research studies (Harris et al. 2019, 2009). Survey responses were organized by graduation year and were compared using the Kruskal-Wallis test to note any differences in survey values across years for response categories. Responses were also categorized by year of graduation (2014 and before or after 2014) and compared with survey values (<8 or >=8) using the Chi-Square test. Data were further analyzed using the Jonckheere–Terpstra test to determine the overall trend of survey responses across graduation years. Data was reported as “mean ± standard deviation.” A p-value < 0.05 was considered statistically significant. Analyses were performed using the IBM Statistical Package for the Social Sciences (SPSS) version 27 (IBM Corp., Armonk, NY).
Results
Of 83 matched applicants in the overall cohort, forty-five (54.2%) responded to the survey. Of all survey respondents, forty-four (97.8%) successfully matched into a PGY-1 orthopaedic surgery residency position on their first attempt. An impressive 50% of students matched into their top choice residency, 70.0% matched into their top three ranked residency program, and ~90.0% matched into their top five ranked residency program. Twelve applicants matched to an orthopaedic residency at our institution (RWJMS). Only two (4.4%) survey respondents dual-applied to another specialty. [Table 1]
When evaluating survey responses using a 1-5 Likert scale, the greatest mean survey response value was received for whether survey respondents’ sub-internship at our home institution adequately prepared them to excel on away rotations (4.38 ± 0.912). Conversely, survey respondents from our institution rated faculty and resident receptiveness to students seeking research opportunities the lowest (3.38 ± 1.007). [Table 2]
When evaluating survey respondents’ overall confidence level, confidence level with residency interviews and overall satisfaction with the residency application process on a scale of 1-10, responses averaged ~8.00. [Table 2] Confidence during residency interviews received the highest survey response values at 8.27 ± 1.250 followed by overall confidence in matching into orthopaedic surgery at 8.00 ± 1.931, followed by satisfaction with the overall residency application process 7.89 ± 1.910. [Table 2]
Survey respondents noted that board exam scores (33.3%), performance on away rotations (28.9%), and letters of recommendation (15.6%) were found to be their greatest strengths while greatest weaknesses were reported as research experience (53.3%), lack of inclusion (17.8%) in the AOA, and board exam scores (15.6%). [Table 2]
There were no statistically significant differences noted when comparing response values by graduation year across any of the survey questions.
A statistically significant difference in comparing graduation year (2014 and before vs after 2014) with response values (< 8 vs ≥ 8) was not found in the “Overall Satisfaction,” “Confidence in Matching,” or “Confidence in Interviewing” questions. [Table 3]
A statistically significant difference was noted in the “Mentorship and Guidance” (p = 0.004) and “Personal Connection” (p = 0.027) questions when trending increases in response values over graduation years. Response values showed a trend towards significance but were not statistically significant in the “Feedback to Improve” (p = 0.078) category. There was no statistical significance noted in any other survey questions when trending response values over graduation years. [Table 4]
Discussion
We found that students from our institution most commonly listed board exam scores, performance on away rotations, and letters of recommendation as their greatest strengths while research experience, lack of inclusion in the AOA, and board exam scores were listed as their greatest weaknesses. This was consistent amongst the 15 years of matched students from our institution. In general, items listed as strengths (performance on clinical/away rotations, and letters of recommendations) are aspects of the academic profile that develop later in a medical student’s career. Items commonly listed as weaknesses (leadership/research/service experiences) are aspects of the academic profile that tend to be encountered earlier in a medical student’s career. This contrast in the themes of strengths and weaknesses may imply that at our institution, there is more guidance available for students in clerkship years compared to early pre-clerkship years.
Comparison of responses by graduation year yielded no statistically significant differences, however a trend towards reaching significance is visible and a clinical significance may already be evident. The “Overall Satisfaction” category showed higher ratings in students that graduated after 2014 when compared to other graduating classes (p = 0.058). While not considered statistically significant, this data indicates that clinical significance may be relevant as faculty have recently taken on larger roles in student guidance. We believe that the sustained involvement of faculty, along with the addition of more faculty, will continue to increase student satisfaction, allowing us to reach a statistically significant conclusion.
Recognizing the impact that orthopaedic faculty can have on their interested students, we highly recommend the adoption of a structured, faculty-guided orthopaedic education and mentorship program at medical schools with these departments. Some of the insight gained from this study has already sparked change at our institution as we have seen an active increase in the emphasis on faculty-student mentorship relationships in the department of orthopaedic surgery at RWJMS. Given what we know about the effect of these relationships, the authors expect to see more satisfaction with mentorship, guidance and personal connections made at RWJMS, which we expect will play a key role in students’ success.
Findings regarding Mentorship/Comments about RWJ Mentorship
Three questions were written to specifically gauge the quality and availability of orthopaedic mentorship: 1) “the orthopaedic faculty and residents at RWJUH were receptive to students seeking mentorship,” 2) “I was able to form a strong personal connection with at least one orthopaedic faculty member throughout my time at RWJMS,” and 3) RWJMS provided me with adequate mentorship and guidance throughout the residency application process." The data show that most students were satisfied with the overall experience at our institution. However, there were a significant number of students who felt this experience was inadequate over the years. The highest percentage of negative responses were associated with identifying if the institution provided adequate mentorship and guidance. In an effort to address these responses, we set specific objectives in the form of an action plan to help enhance and expand the mentorship experience for both medical students and orthopaedic surgeon mentors.
Action Plan: Mentorship
An action plan was developed to improve the orthopaedic medical student mentorship experience. The action plan focuses on two fundamental aspects of mentorship: peer mentorship from senior students and individualized mentorship pairing faculty and, in the future, residents with students. Peer mentoring can provide easier access for mentees, help facilitate opportunities with senior mentors, and fortify the sense of community amongst students interested in orthopaedics (Aiyer et al. 2021). Interested fourth year medical students (M4) collectively serve as mentors for the first-, second-, and third-year medical students. The mentoring program includes informative lectures and group meetings throughout the year.
Additionally, a one-on-one mentorship program will take place through pairing students with an orthopaedic faculty mentor. A PGY3 resident may be assigned to an M3 student as an additional mentor. The goal of this program is to pair students that demonstrate a strong desire to pursue orthopaedic surgery with a faculty mentor early in their M3 year to help guide them through the application process. Once paired, mentees and mentors will meet at regular intervals to discuss goals of the relationship. With the help of the mentor, students can get involved earlier with research and/or clinical experiences that are crucial to developing the academic portfolio.
The goal of this action plan is to develop long-term relationships between faculty, residents, and students. These mentoring opportunities will serve to increase students’ competitiveness by providing personalized guidance during the critical 3rd and 4th years of medical school. The success of this action plan will be measured in a future study by comparing prospective student satisfaction against satisfaction score prior to action plan implementation.
Away Rotations / Boot camp
Preparing students for their away rotations was identified as a strength of the program and performance on away rotations was rated by students as their most substantial strength in 28.9%. Utilizing this information, our institution created an orthopaedic bootcamp to prepare students for their 4th year orthopaedic rotations. This involves a curriculum of in-person training sessions and lectures. The training sessions were focused on the fundamentals of orthopaedic surgery that would be applicable to students on their sub-internship rotations. These included an anatomy lab, a splinting/casting workshop, and a sawbones lab with a primer on orthopaedic equipment.
Findings about Research
Our study revealed that medical students at our institution were unsatisfied with the lack of research opportunities available, and this limitation was an overall weakness at our institution. The majority (40%) of students felt neutral about the orthopaedic faculty and residents at RWJUH being receptive to students seeking research opportunities, with an additional 11.1% and 4.4% of students disagreeing, and strongly disagreeing, respectively. In addition, another series of questions asked students to report the top three strengths and weaknesses of their residency applications. Collectively, only 24.4% of students reported that research was one of the most significant strengths of their application. Conversely, the majority of students, 68.9%, reported that research was one of the most significant weaknesses of their application. Overall, while most students felt neutral about the responsiveness of orthopaedic faculty getting students involved in research, most students felt that the lack of research opportunities at our institution was one of the main weaknesses of their residency applications. After identifying that participation in research was largely inadequate at our institution, we identified a set of objectives that can help us facilitate more opportunities for student involvement in this arena.
Action plan: Research
To improve research involvement for students at our institution, we developed an action plan that focuses on three principal concepts: transparency between faculty and students in relation to ongoing research projects, organization of a list of students interested in research involvement, and the facilitation of departmental, specialty-specific, and project-specific research meetings.
After assessing the current status of ongoing research projects at our institution, we learned that students often sent mass emails to various orthopaedic faculty to inquire about the availability of ongoing research projects. This process was tedious for students as well as cumbersome for faculty. To circumvent this problem, we created an easily accessible, “live,” centralized list of ongoing projects across the department of orthopaedics. Faculty and residents were invited to add any ongoing study, or study proposal, and highlight if medical students were needed to help assist with the project. Students interested in research can access the spreadsheet at any time and learn if new projects are available for their assistance.
Another weakness that we found at our institution relating to research involvement was an uneven balance between student research involvement; some students were involved in numerous projects while others had difficulty joining any projects at all. To overcome this, we developed an interest form for research involvement. All students who were interested in orthopaedic research were encouraged to fill out a brief survey indicating their interest. As more projects became available, the next student(s) on the list would be invited to assist with the project to provide fair opportunities.
It has been established that involvement in research and publications plays an important role in the success of medical student matching into orthopaedic residencies (Chen et al. 2020; Campbell, Gupta, and Avedian 2016). Therefore, we realized the sense of urgency that was necessary to address the weakness at our institution’s orthopaedic department regarding research involvement. The hope with these three objectives and action plans was to ensure that there is both an increased availability of research projects for students to participate in, as well as a more even and consistent way of distributing projects amongst interested students. Similarly, regarding how the success of the mentorship action plan will be assessed, improvement in student access to research will be measured by comparing student satisfaction in future years to scores received prior to action plan implementation. Additionally, we may consider measuring research output from students going forward to compare to that of respondents of the initial study.
Future Directions
Our findings on mentorship, along with the importance of away rotations, boot camp, and research opportunities, have allowed for the direct implementation of solutions to our institution’s flaws. The survey used to conduct this study will be sent out on a yearly basis to obtain prospective data that will allow for the continuous improvement of our program. It is our hope to continue to improve upon this survey by introducing additional questions addressing areas of importance. Specifically, future study will aim to address student thought on inclusion and diversity of RWJMS orthopaedic education. Additionally, with Step I changing to a pass or fail scoring system the initial importance that was placed on this exam will shift towards other areas of the application such as Step II, letters of recommendation, and research experience.
Future questions will also be directed at understanding student perspective of this change and how our institution supports students to achieve these new goals. The survey will be adjusted on a yearly basis to target areas of particular interest. For example, more specific questions will be utilized to assess applicant data, such as number of publications, research experiences, away rotations performed, recommendation letters received, interviews offered/attended.
Limitations
The current study has a few limitations to address. First, the survey cohort had a limited sample size and since it included only successfully matched students, thus an element of selection bias may have been introduced during data analysis. Analysis of unmatched students was not possible due to institutional prohibition on obtaining unmatched student information. Additionally, a portion of the respondents (53.3%) have already completed residency, leading to the possibility of recall bias playing a role in the analysis. Lastly, from our overall cohort, our survey response rate was 45/83 (54.2%), as 38 matched applicants did not complete this survey, which may have introduced an element of selection bias.
Conclusion
An internal assessment of orthopaedic departments at medical institutions is vital in evaluating the strengths and weaknesses of a program’s capability to prepare their students to successfully match in orthopaedic residencies. The RWJMS Orthopaedic Education Satisfaction Survey serves as a model that medical schools across the country can adopt to evaluate the adequacy and availability of clinical, mentorship, and research opportunities. In addition to identifying any internal inadequacies, a given program must be able to respond by setting objectives for improvement and implementing action plans to circumvent any identified deficiencies. With consistent assessment of student satisfaction and subsequent implementation of plans for improvement, institutions across the country can ensure that their students are in the best position to succeed, especially given the current competitive state of successfully matching into an orthopaedic residency.
Funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Declaration of Interest
None