Aims: According to the guidance hypothesis, tutoring during technical skills training can result in tutoring over-reliance, reflected in a negative effect on performance when tutoring is discontinued. In this study, we wanted to explore if similar effects would be found for cognitive load.
Methods: Two cohorts of novice medical students were recruited for distributed virtual simulation training (five practice blocks of three procedures): 16 participants received intermittent simulator-integrated tutoring and 14 participants served as a reference cohort and did not receive simulator-integrated tutoring. Cognitive load during simulation was estimated using secondary task reaction time. Linear mixed models were used to account for repeated measurements.
Results: Overall, the tutored cohort had a significantly higher cognitive load than the reference cohort (mean difference = 7 %, p=0.006). Simulator-integrated tutoring did seem to lower cognitive load when active but also caused the tutored cohort to have a substantially higher cognitive load in subsequent performances where it was turned off (mean difference = 7 %, respectively, p<<0.001).
Conclusions: Concurrent feedback by simulator-integrated tutoring causes tutoring over-reliance and modifies cognitive load. This suggests that tutoring, in addition to degrading motor skills learning also affects the cognitive processes involved.
OBJECTIVE: Handheld otoscopy requires both technical and diagnostic skills, and is often reported to be insufficient after medical training. We aimed to develop and gather validity evidence for an assessment tool for handheld otoscopy using contemporary medical educational standards.
STUDY DESIGN: Educational study.
SETTING: University/teaching hospital.
SUBJECTS AND METHODS: A structured Delphi methodology was used to develop the assessment tool: nine key opinion leaders (otologists) in undergraduate training of otoscopy iteratively achieved consensus on the content. Next, validity evidence was gathered by the video-taped assessment of two handheld otoscopy performances of 15 medical students (novices) and 11 specialists in otorhinolaryngology using two raters. Standard setting (pass/fail criteria) was explored using the contrasting groups and Angoff methods.
RESULTS: The developed Copenhagen Assessment Tool of Handheld Otoscopy Skills (CATHOS) consists 10 items rated using a 5-point Likert scale with descriptive anchors. Validity evidence was collected and structured according to Messick’s framework: for example the CATHOS had excellent discriminative validity (mean difference in performance between novices and experts 20.4 out of 50 points, p<0.001); and high internal consistency (Cronbach’s alpha=0.94). Finally, a pass/fail score was established at 30 points for medical students and 42 points for specialists in ORL.
CONCLUSION: We have developed and gathered validity evidence for an assessment tool of technical skills of handheld otoscopy and set standards of performance. Standardized assessment allows for individualized learning to the level of proficiency and could be implemented in under- and postgraduate handheld otoscopy training curricula, and is also useful in evaluating training interventions.
Purpose: At graduation from medical school, competency in otoscopy is often insufficient. Simulation-based training can be used to improve technical skills, but the suitability of the training model and assessment must be supported by validity evidence. The purpose of this study was to collect content validity evidence for a simulation-based test of handheld otoscopy skills.
Methods: First, a three-round Delphi study was conducted with a panel of nine clinical teachers in otorhinolaryngology (ORL) to determine the content requirements in our educational context. Next, the authenticity of relevant cases in a commercially available technology-enhanced simulator (Earsi, VR Magic, Germany) was evaluated by specialists in ORL. Finally, an integrated course was developed for the simulator based on these results.
Results: The Delphi study resulted in nine essential diagnoses of normal variations and pathologies that all junior doctors should be able to diagnose with a handheld otoscope. Twelve out of 15 tested simulator cases were correctly recognized by at least one ORL specialist. Fifteen cases from the simulator case library matched the essential diagnoses determined by the Delphi study and were integrated into the course.
Conclusion: Content validity evidence for a simulation-based test of handheld otoscopy skills was collected. This informed a simulation-based course that can be used for undergraduate training. The course needs to be further investigated in relation to other aspects of validity and for future self-directed training.
OBJECTIVE: Self-directed training represents a challenge in simulation-based training as low cognitive effort can occur when learners overrate their own level of performance. This study aims to explore the mechanisms underlying the positive effects of a structured self-assessment intervention during simulation-based training of mastoidectomy.
METHODS: A prospective, educational cohort study of a novice training program consisting of directed, self-regulated learning with distributed practice (5×3 procedures) in a virtual reality temporal bone simulator. The intervention consisted of structured self-assessment after each procedure using a rating form supported by small videos. Semi-structured telephone interviews upon completion of training were conducted with 13 out of 15 participants. Interviews were analysed using directed content analysis and triangulated with quantitative data on secondary task reaction time for cognitive load estimation and participants’ self-assessment scores.
RESULTS: Six major themes were identified in the interviews: goal-directed behaviour, use of learning supports for scaffolding of the training, cognitive engagement, motivation from self-assessment, self-assessment bias, and feedback on self-assessment (validation). Participants seemed to self-regulate their learning by forming individual sub-goals and strategies within the overall goal of the procedure. They scaffolded their learning through the available learning supports. Finally, structured self-assessment was reported to increase the participants’ cognitive engagement, which was further supported by a quantitative increase in cognitive load.
CONCLUSIONS: Structured self-assessment in simulation-based surgical training of mastoidectomy seems to promote cognitive engagement and motivation in the learning task and to facilitate self-regulated learning.
OBJECTIVE: Often the assessment of mastoidectomy performance requires time-consuming manual rating. Virtual reality (VR) simulators offer potentially useful automated assessment and feedback but should be supported by validity evidence. We aimed to investigate simulator metrics for automated assessment based on the expert performance approach, comparison with an established assessment tool, and the consequences of standard setting.
METHODS: The performances of 11 experienced otosurgeons and 37 otorhinolaryngology residents. Participants performed three mastoidectomies in the Visible Ear Simulator. Nine residents contributed additional data on repeated practice in the simulator. One hundred and twenty-nine different performance metrics were collected by the simulator and final-product files were saved. These final products were analyzed using a modified Welling Scale by two blinded raters.
RESULTS: Seventeen metrics could discriminate between resident and experienced surgeons’ performances. These metrics mainly expressed various aspects of efficiency: Experts demonstrated more goal-directed behavior and less hesitancy, used less time, and selected large and sharp burrs more often. The combined metrics-based score (MBS) demonstrated significant discriminative ability between experienced surgeons and residents with a mean difference of 16.4% (95% confidence interval [12.6-20.2], P << 0.001). A pass/fail score of 83.6% was established. The MBS correlated poorly with the final-product score but excellently with the final-product score per time.
CONCLUSION: The MBS mainly reflected efficiency components of the mastoidectomy procedure, and although it could have some uses in self-directed training, it fails to measure and encourage safe routines. Supplemental approaches and feedback are therefore required in VR simulation training of mastoidectomy.
This article presents a summary of the current simulation training for otologic skills. There is a wide variety of educational approaches, assessment tools, and simulators in use, including simple low-cost task trainers to complex computer-based virtual reality systems. A systematic approach to otologic skills training using adult learning theory concepts, such as repeated and distributed practice, self-directed learning, and mastery learning, is necessary for these educational interventions to be effective. Future directions include development of measures of performance to assess efficacy of simulation training interventions and, for complex procedures, improvement in fidelity based on educational goals.
OBJECTIVES/HYPOTHESIS: To explore why novices’ performance plateau in directed, self-regulated virtual reality (VR) simulation training and how performance can be improved.
STUDY DESIGN: Prospective study.
METHODS: Data on the performances of 40 novices who had completed repeated, directed, self-regulated VR simulation training of mastoidectomy were included. Data were analyzed to identify key areas of difficulty as well as the procedures terminated without using all the time allowed.
RESULTS: Novices had difficulty in avoiding drilling holes in the outer anatomical boundaries of the mastoidectomy and frequently made injuries to vital structures such as the lateral semicircular canal, the ossicles, and the facial nerve. The simulator-integrated tutor function improved performance on many of these items, but overreliance on tutoring was observed. Novices also demonstrated poor self-assessment skills and often did not make use of the allowed time, lacking knowledge on when to stop or how to excel.
CONCLUSION: Directed, self-regulated VR simulation training of mastoidectomy needs a strong instructional design with specific process goals to support deliberate practice because cognitive effort is needed for novices to improve beyond an initial plateau.
Virtual reality (VR) simulation-based training is increasingly used in surgical technical skills training including in temporal bone surgery. The potential of VR simulation in enabling high-quality surgical training is great and VR simulation allows high-stakes and complex procedures such as mastoidectomy to be trained repeatedly, independent of patients and surgical tutors, outside traditional learning environments such as the OR or the temporal bone lab, and with fewer of the constraints of traditional training. This thesis aims to increase the evidence-base of VR simulation training of mastoidectomy and, by studying the final-product performances of novices, investigates the transfer of skills to the current gold-standard training modality of cadaveric dissection, the effect of different practice conditions and simulator-integrated tutoring on performance and retention of skills, and the role of directed, self-regulated learning. Technical skills in mastoidectomy were transferable from the VR simulation environment to cadaveric dissection with significant improvement in performance after directed, self-regulated training in the VR temporal bone simulator. Distributed practice led to a better learning outcome and more consolidated skills than massed practice and also resulted in a more consistent performance after three months of non-practice. Simulator-integrated tutoring accelerated the initial learning curve but also caused over-reliance on tutoring, which resulted in a drop in performance when the simulator-integrated tutor-function was discontinued. The learning curves were highly individual but often plateaued early and at an inadequate level, which related to issues concerning both the procedure and the VR simulator, over-reliance on the tutor function and poor self-assessment skills. Future simulator-integrated automated assessment could potentially resolve some of these issues and provide trainees with both feedback during the procedure and immediate assessment following each procedure. Standard setting by establishing a proficiency level that can be used for mastery learning with deliberate practice could also further sophisticate directed, self-regulated learning in VR simulation-based training. VR simulation-based training should be embedded in a systematic and competency-based training curriculum for high-quality surgical skills training, ultimately leading to improved safety and patient care.
OBJECTIVES/HYPOTHESIS: To establish the effect of self-directed virtual reality (VR) simulation training on cadaveric dissection training performance in mastoidectomy and the transferability of skills acquired in VR simulation training to the cadaveric dissection training setting.
STUDY DESIGN: Prospective study.
METHODS: Two cohorts of 20 novice otorhinolaryngology residents received either self-directed VR simulation training before cadaveric dissection training or vice versa. Cadaveric and VR simulation performances were assessed using final-product analysis with three blinded expert raters.
RESULTS: The group receiving VR simulation training before cadaveric dissection had a mean final-product score of 14.9 (95 % confidence interval [CI] [12.9-16.9]) compared with 9.8 (95% CI [8.4-11.1]) in the group not receiving VR simulation training before cadaveric dissection. This 52% increase in performance was statistically significantly (P < 0.0001). A single dissection mastoidectomy did not increase VR simulation performance (P = 0.22).
CONCLUSIONS: Two hours of self-directed VR simulation training was effective in increasing cadaveric dissection mastoidectomy performance and suggests that mastoidectomy skills are transferable from VR simulation to the traditional dissection setting. Virtual reality simulation training can therefore be employed to optimize training, and can spare the use of donated material and instructional resources for more advanced training after basic competencies have been acquired in the VR simulation environment.
LEVEL OF EVIDENCE: NA.
BACKGROUND: Cognitive overload can inhibit learning, and cognitive load theory-based instructional design principles can be used to optimize learning situations. This study aims to investigate the effect of implementing cognitive load theory-based design principles in virtual reality simulation training of mastoidectomy.
METHODS: Eighteen novice medical students received 1 h of self-directed virtual reality simulation training of the mastoidectomy procedure randomized for standard instructions (control) or cognitive load theory-based instructions with a worked example followed by a problem completion exercise (intervention). Participants then completed two post-training virtual procedures for assessment and comparison. Cognitive load during the post-training procedures was estimated by reaction time testing on an integrated secondary task. Final-product analysis by two blinded expert raters was used to assess the virtual mastoidectomy performances.
RESULTS: Participants in the intervention group had a significantly increased cognitive load during the post-training procedures compared with the control group (52 vs. 41 %, p = 0.02). This was also reflected in the final-product performance: the intervention group had a significantly lower final-product score than the control group (13.0 vs. 15.4, p < 0.005).
CONCLUSIONS: Initial instruction using worked examples followed by a problem completion exercise did not reduce the cognitive load or improve the performance of the following procedures in novices. Increased cognitive load when part tasks needed to be integrated in the post-training procedures could be a possible explanation for this. Other instructional designs and methods are needed to lower the cognitive load and improve the performance in virtual reality surgical simulation training of novices.
IMPORTANCE: The ultimate goal of surgical training is consolidated skills with a consistently high performance. However, surgical skills are heterogeneously retained and depend on a variety of factors, including the task, cognitive demands, and organization of practice. Virtual reality (VR) simulation is increasingly being used in surgical skills training, including temporal bone surgery, but there is a gap in knowledge on the retention of mastoidectomy skills after VR simulation training.
OBJECTIVES: To determine the retention of mastoidectomy skills after VR simulation training with distributed and massed practice and to investigate participants’ cognitive load during retention procedures.
DESIGN, SETTING, AND PARTICIPANTS: A prospective 3-month follow-up study of a VR simulation trial was conducted from February 6 to September 19, 2014, at an academic teaching hospital among 36 medical students: 19 from a cohort trained with distributed practice and 17 from a cohort trained with massed practice.
INTERVENTIONS: Participants performed 2 virtual mastoidectomies in a VR simulator a mean of 3.2 months (range, 2.4-5.0 months) after completing initial training with 12 repeated procedures. Practice blocks were spaced apart in time (distributed), or all procedures were performed in 1 day (massed).
MAIN OUTCOMES AND MEASURES: Performance of the virtual mastoidectomy as assessed by 2 masked senior otologists using a modified Welling scale, as well as cognitive load as estimated by reaction time to perform a secondary task.
RESULTS: Among 36 participants, mastoidectomy final-product skills were largely retained at 3 months (mean change in score, 0.1 points; P = .89) regardless of practice schedule, but the group trained with massed practice took more time to complete the task. The performance of the massed practice group increased significantly from the first to the second retention procedure (mean change, 1.8 points; P = .001), reflecting that skills were less consolidated. For both groups, increases in reaction times in the secondary task (distributed practice group: mean pretraining relative reaction time, 1.42 [95% CI, 1.37-1.47]; mean end of training relative reaction time, 1.24 [95% CI, 1.16-1.32]; and mean retention relative reaction time, 1.36 [95% CI, 1.30-1.42]; massed practice group: mean pretraining relative reaction time, 1.34 [95% CI, 1.28-1.40]; mean end of training relative reaction time, 1.31 [95% CI, 1.21-1.42]; and mean retention relative reaction time, 1.39 [95% CI, 1.31-1.46]) indicated that cognitive load during the virtual procedures had returned to the pretraining level.
CONCLUSIONS AND RELEVANCE: Mastoidectomy skills acquired under time-distributed practice conditions were retained better than skills acquired under massed practice conditions. Complex psychomotor skills should be regularly reinforced to consolidate both motor and cognitive aspects. Virtual reality simulation training provides the opportunity for such repeated training and should be integrated into training curricula.
OBJECTIVE: To evaluate the short-term stability of postoperative hearing results after tympanoplasty.
STUDY DESIGN: Prospective database study.
SETTING: Tertiary referral center.
PATIENTS: 1,367 cases of tympanoplasty I-IV were registered in the OTOKIR database between February 2004 and November 2013.
INTERVENTION: The authors included the 553 cases attending postoperative follow-ups at both 3 and 12 months.
MAIN OUTCOME MEASURE: Analysis of the changes in pure-tone average of air conduction (AC), air-bone gap, and speech reception threshold and Word Recognition Score between follow-ups were performed.
RESULTS: The overall mean change between follow-ups was 0.7, 0.5, and 0.3 dB for the AC, air-bone gap, and speech reception threshold, respectively. A majority of cases (87.7%) had a change in AC of 10 dB or less, and only 7.6% of the tympanoplasty type I cases had a decrease in AC of more than 10 dB. Of the 1,367 cases registered, 47.5% of cases were lost to follow-up at 12 months.
CONCLUSION: The changes in hearing results after tympanoplasty are minimal during 3 to 12 months after surgery. This suggests that 3-month results are as valid for reporting as 12-month results. In addition, a possible bias that compromises the validity of reported results is introduced at 12 months because half of the cases are lost to follow-up. Including results from 3-month postoperative follow-up when reporting on tympanoplasty could reduce bias in reporting and enable more centers to contribute valid results.
CONCLUSION: Current guidelines recommend reporting short-term results of > 12 months after treatment of conductive hearing loss. This study suggests that short-term hearing results after stapedotomy recorded at the 3-month follow-up are without loss of vital information compared with data from the currently recommended > 12-month follow-up. The use of 3-month data in reporting outcome could reduce the bias inherent to the loss to follow-up at 12 months.
OBJECTIVE: To investigate the stability of short-term postoperative hearing after stapedotomy for otosclerosis.
METHODS: This was a prospective database study; 371 cases with otosclerosis were registered in the database between August 2004 and June 2013. We included the 166 primary cases and 37 revision cases that had attended both follow-ups.
RESULTS: The mean changes in postoperative hearing thresholds between the 3-month and 12-month follow-up in both primary and revision cases were minimal and clinically insignificant. In all, 3-5% of primary cases and 14-16% of revision cases experienced a change of ≥ 10 dB for the worse of one or more parameters between follow-ups. Results were also stable when considering a range of traditional success criteria. Other complications following surgery were infrequent and typically resolved long term.
OBJECTIVE: To present a prospective ear surgery database and investigate the graft take-rate and prognostic factors for graft take-rate in tympanoplasty using the database.
STUDY DESIGN: Prospective database study.
SETTING: Tertiary referral center.
PATIENTS: A total of 1606 cases undergoing tympanoplasty types I to IV were registered in the database in the period from February 2004 to November 2013.
INTERVENTION: A total of 837 cases underwent myringoplasty/tympanoplasty type I.
MAIN OUTCOME MEASURE: Graft take-rate and prognostic factors (age, discharge at time of surgery, tuba function, technique, graft material, and revision surgery) for tympanoplasty type I were studied. A comparison with the graft take-rates for tympanoplasty types II to IV and/or cholesteatoma was made.
RESULTS: A user-friendly ear surgery database with fast data entry and direct import of audiometric data was developed. The graft take-rate was found to be 93.0% at 2 to 6 months and 86.6% at more than 12 months. Except for a discharging ear at the time of surgery, no significant differences using χ² test of association were found when comparing graft take-rates for different prognostic factors or more advanced tympanoplasty with or without cholesteatoma. A long-term graft take-rate overestimation of 6% was found if cases with defaulted follow-up because of early reperforation were not included.
CONCLUSION: A prospective database can be used to study prognostic factors and reduce bias in reporting the graft take-rate. Prospective databases are needed for high-quality longitudinal studies but require a continuous and daily effort of involved surgeons and therefore need to be convenient and fast to use.