Learning supports – Steven Andersen | MD, PhD, DMSc

Automated summative feedback improves performance and retention in simulation training of mastoidectomy: A randomised, controlled trial

stevenarild — Sat, 27 Feb 2021 16:44:18 +0000

Objective: This study aimed to investigate the effects of automated metrics-based summative feedback on performance, retention and cognitive load in distributed virtual reality simulation training of mastoidectomy.

Method: Twenty-four medical students were randomised in two groups and performed 15 mastoidectomies on a distributed virtual reality simulator as practice. The intervention group received additional summative metrics-based feedback; the control group followed standard instructions. Two to three months after training, participants performed a retention test without learning supports.

Results: The intervention group had a better final-product score (mean difference = 1.0 points; p = 0.001) and metrics-based score (mean difference = 12.7; p < 0.001). At retention, the metrics-based score for the intervention group remained superior (mean difference = 6.9 per cent; p = 0.02). Also at the retention, cognitive load was higher in the intervention group (mean difference = 10.0 per cent; p < 0.001).

Conclusion: Summative metrics-based feedback improved performance and lead to a safer and faster performance compared with standard instructions and seems a valuable educational tool in the early acquisition of temporal bone skills.

Reliable assessment of surgical technical skills is dependent on context: an exploration of different variables using Generalizability theory

stevenarild — Tue, 23 Jun 2020 18:40:30 +0000

Purpose: Reliable assessment of surgical skills is vital for competency-based medical training. Several factors influence not only the reliability of judgements but also the number of observations needed for making judgments of competency that are both consistent and reproducible. The aim of this study was to explore the role of various conditions-through the analysis of data from large-scale, simulation-based assessments of surgical technical skills-by examining the effects of those conditions on reliability using Generalizability theory.

Method: Assessment data from large-scale, simulation-based temporal bone surgical training research studies in 2012-2018 were pooled, yielding collectively 3,574 assessments of 1,723 performances. The authors conducted generalizability analyses using an unbalanced random-effects design, and they performed decision studies to explore the effect of the different variables on projections of reliability.

Results: Overall, five observations were needed to achieve a Generalizability coefficient > 0.8. Several variables modified the projections of reliability: increased learner experience necessitated more observations (5 for medical students, 7 for residents, and 8 for experienced surgeons); the more complex cadaveric dissection required fewer observations than virtual reality simulation (2 vs. 5 observations); and increased fidelity simulation graphics reduced the number of observations needed from 7 to 4. The training structure (either massed or distributed practice) and simulator-integrated tutoring had little effect on reliability. Finally, more observations were needed during initial training when the learning curve was steepest (6 observations) compared with the plateau phase (4 observations).

Conclusions: Reliability in surgical skills assessment seems less stable than it is often reported to be. Training context and conditions influence reliability. The findings from this study highlight that medical educators should exercise caution when using a specific simulation-based assessment in other contexts.

Effects on cognitive load of tutoring in virtual reality simulation training

stevenarild — Thu, 02 Apr 2020 11:56:56 +0000

Aims: According to the guidance hypothesis, tutoring during technical skills training can result in tutoring over-reliance, reflected in a negative effect on performance when tutoring is discontinued. In this study, we wanted to explore if similar effects would be found for cognitive load.

Methods: Two cohorts of novice medical students were recruited for distributed virtual simulation training (five practice blocks of three procedures): 16 participants received intermittent simulator-integrated tutoring and 14 participants served as a reference cohort and did not receive simulator-integrated tutoring. Cognitive load during simulation was estimated using secondary task reaction time. Linear mixed models were used to account for repeated measurements.

Results: Overall, the tutored cohort had a significantly higher cognitive load than the reference cohort (mean difference = 7 %, p=0.006). Simulator-integrated tutoring did seem to lower cognitive load when active but also caused the tutored cohort to have a substantially higher cognitive load in subsequent performances where it was turned off (mean difference = 7 %, respectively, p<<0.001).

Conclusions: Concurrent feedback by simulator-integrated tutoring causes tutoring over-reliance and modifies cognitive load. This suggests that tutoring, in addition to degrading motor skills learning also affects the cognitive processes involved.

Understanding the effects of structured self-assessment in directed, self-regulated simulation-based training of mastoidectomy: a mixed methods study

stevenarild — Mon, 23 Dec 2019 08:00:08 +0000

OBJECTIVE: Self-directed training represents a challenge in simulation-based training as low cognitive effort can occur when learners overrate their own level of performance. This study aims to explore the mechanisms underlying the positive effects of a structured self-assessment intervention during simulation-based training of mastoidectomy.

METHODS: A prospective, educational cohort study of a novice training program consisting of directed, self-regulated learning with distributed practice (5×3 procedures) in a virtual reality temporal bone simulator. The intervention consisted of structured self-assessment after each procedure using a rating form supported by small videos. Semi-structured telephone interviews upon completion of training were conducted with 13 out of 15 participants. Interviews were analysed using directed content analysis and triangulated with quantitative data on secondary task reaction time for cognitive load estimation and participants’ self-assessment scores.

RESULTS: Six major themes were identified in the interviews: goal-directed behaviour, use of learning supports for scaffolding of the training, cognitive engagement, motivation from self-assessment, self-assessment bias, and feedback on self-assessment (validation). Participants seemed to self-regulate their learning by forming individual sub-goals and strategies within the overall goal of the procedure. They scaffolded their learning through the available learning supports. Finally, structured self-assessment was reported to increase the participants’ cognitive engagement, which was further supported by a quantitative increase in cognitive load.

CONCLUSIONS: Structured self-assessment in simulation-based surgical training of mastoidectomy seems to promote cognitive engagement and motivation in the learning task and to facilitate self-regulated learning.

The effect of simulator-integrated tutoring for guidance in virtual reality simulation training

stevenarild — Wed, 09 Oct 2019 08:00:31 +0000

INTRODUCTION: Simulation-integrated tutoring in virtual reality (VR) simulation training by green-lighting is a common learning support in simulation-based temporal bone surgical training. However, tutoring overreliance can negatively affect learning. We therefore wanted to investigate the effects of simulator-integrated tutoring on performance and learning.

METHODS: A prospective, educational cohort study of a learning intervention (simulator-integrated tutoring) during repeated and distributed VR simulation training for directed, self-regulated learning of the mastoidectomy procedure. Two cohorts of novices (medical students) were recruited: 16 participants were trained using the intervention program (intermittent simulator-integrated tutoring) and 14 participants constituted a non-tutored reference cohort. Outcomes were final-product performance assessed by two blinded raters, and simulator-recorded metrics.

RESULTS: Simulator-integrated tutoring had a large and positive effect on the final-product performance while turned on (mean difference 3.8 points, p<0.0001). However, this did not translate to a better final-product performance in subsequent non-tutored procedures. The tutored cohort had a better metrics-based score, reflecting higher efficiency of drilling (mean difference 3.6 %, p=0.001). For the individual metrics, simulator-integrated tutoring had mixed effects both during procedures and on the tutored cohort in general (learning effect).

CONCLUSIONS: Simulator-integrated tutoring by green-lighting did not induce a better final-product performance but increased efficiency. The mixed effects on learning could be caused by tutoring overreliance, resulting from a lack of cognitive engagement when the tutor-function is on. Further learning strategies such as feedback should be explored to support novice learning and cognitive engagement.

The effect of structured self-assessment in virtual reality simulation training of mastoidectomy

stevenarild — Sun, 01 Sep 2019 08:00:29 +0000

PURPOSE: Virtual reality (VR) simulation surgical skills training is well established, but self-directed practice is often associated with a learning curve plateau. In this study, we investigate the effects of structured self-assessment as a means to improve performance in mastoidectomy training.

METHODS: The study was a prospective, educational study. Two cohorts of novices (medical students) were recruited for practice of anatomical mastoidectomy in a training program with five distributed training blocks. Fifteen participants performed structured self-assessment after each procedure (intervention cohort). A reference cohort of another 14 participants served as controls. Performances were assessed by two blinded raters using a modified Welling Scale and simulator-recorded metrics.

RESULTS: The self-assessment cohort performed superiorly to the reference cohort (mean difference of final product score 0.87 points, p = 0.001) and substantially reduced the number of repetitions needed. The self-assessment cohort also had more passing performances for the combined metrics-based score reflecting increased efficiency. Finally, the self-assessment cohort made fewer collisions compared with the reference cohort especially with the chorda tympani, the facial nerve, the incus, and the malleus.

CONCLUSIONS: VR simulation training of surgical skills benefits from having learners perform structured self-assessment following each procedure as this increases performance, accelerates the learning curve thereby reducing time needed for training, and induces a safer performance with fewer collisions with critical structures. Structured self-assessment was in itself not sufficient to counter the learning curve plateau and for continued skills development additional supports for deliberate practice are needed.

Decentralized Virtual Reality Mastoidectomy Simulation Training: A Prospective, Mixed-Methods Study

stevenarild — Fri, 26 Jul 2019 08:00:28 +0000

PURPOSE: Virtual reality (VR) training of mastoidectomy is effective in surgical training-particularly if organized as distributed practice. However, centralization of practice facilities is a barrier to implementation of distributed simulation training. Decentralized training could be a potential solution. Here, we aim to assess the feasibility, use, and barriers to decentralized VR mastoidectomy training using a freeware, high-fidelity temporal bone simulator.

METHODS: In a prospective, mixed-methods study, 20 otorhinolaryngology residents were given three months of local access to a VR mastoidectomy simulator. Additionally, trainees were provided a range of learning supports for directed, self-regulated learning. Questionnaire data were collected and focus group interviews conducted. The interviews were analyzed using thematic analysis and compared with quantitative findings.

RESULTS: Participants trained 48.5 h combined and mainly towards the end of the trial. Most participants used between two and four different learning supports. Qualitative analysis revealed five main themes regarding implementation of decentralized simulation training: convenience, time for training, ease of use, evidence for training, and testing.

CONCLUSIONS: Decentralized VR training using a freeware, high-fidelity mastoidectomy simulator is feasible but did not lead to a high training volume or truly distributed practice. Evidence for training was found motivational. Access to training, educational designs, and the role of testing are important for participant motivation and require further evaluation.

Mapping the plateau of novices in virtual reality simulation training of mastoidectomy

stevenarild — Sat, 01 Apr 2017 08:00:23 +0000

OBJECTIVES/HYPOTHESIS: To explore why novices’ performance plateau in directed, self-regulated virtual reality (VR) simulation training and how performance can be improved.

STUDY DESIGN: Prospective study.

METHODS: Data on the performances of 40 novices who had completed repeated, directed, self-regulated VR simulation training of mastoidectomy were included. Data were analyzed to identify key areas of difficulty as well as the procedures terminated without using all the time allowed.

RESULTS: Novices had difficulty in avoiding drilling holes in the outer anatomical boundaries of the mastoidectomy and frequently made injuries to vital structures such as the lateral semicircular canal, the ossicles, and the facial nerve. The simulator-integrated tutor function improved performance on many of these items, but overreliance on tutoring was observed. Novices also demonstrated poor self-assessment skills and often did not make use of the allowed time, lacking knowledge on when to stop or how to excel.

CONCLUSION: Directed, self-regulated VR simulation training of mastoidectomy needs a strong instructional design with specific process goals to support deliberate practice because cognitive effort is needed for novices to improve beyond an initial plateau.

Virtual reality simulation training of mastoidectomy – studies on novice performance

stevenarild — Mon, 01 Aug 2016 08:00:23 +0000

Virtual reality (VR) simulation-based training is increasingly used in surgical technical skills training including in temporal bone surgery. The potential of VR simulation in enabling high-quality surgical training is great and VR simulation allows high-stakes and complex procedures such as mastoidectomy to be trained repeatedly, independent of patients and surgical tutors, outside traditional learning environments such as the OR or the temporal bone lab, and with fewer of the constraints of traditional training. This thesis aims to increase the evidence-base of VR simulation training of mastoidectomy and, by studying the final-product performances of novices, investigates the transfer of skills to the current gold-standard training modality of cadaveric dissection, the effect of different practice conditions and simulator-integrated tutoring on performance and retention of skills, and the role of directed, self-regulated learning. Technical skills in mastoidectomy were transferable from the VR simulation environment to cadaveric dissection with significant improvement in performance after directed, self-regulated training in the VR temporal bone simulator. Distributed practice led to a better learning outcome and more consolidated skills than massed practice and also resulted in a more consistent performance after three months of non-practice. Simulator-integrated tutoring accelerated the initial learning curve but also caused over-reliance on tutoring, which resulted in a drop in performance when the simulator-integrated tutor-function was discontinued. The learning curves were highly individual but often plateaued early and at an inadequate level, which related to issues concerning both the procedure and the VR simulator, over-reliance on the tutor function and poor self-assessment skills. Future simulator-integrated automated assessment could potentially resolve some of these issues and provide trainees with both feedback during the procedure and immediate assessment following each procedure. Standard setting by establishing a proficiency level that can be used for mastery learning with deliberate practice could also further sophisticate directed, self-regulated learning in VR simulation-based training. VR simulation-based training should be embedded in a systematic and competency-based training curriculum for high-quality surgical skills training, ultimately leading to improved safety and patient care.

The effect of implementing cognitive load theory-based design principles in virtual reality simulation training of surgical skills: a randomized controlled trial

stevenarild — Sat, 02 Jul 2016 08:00:23 +0000

BACKGROUND: Cognitive overload can inhibit learning, and cognitive load theory-based instructional design principles can be used to optimize learning situations. This study aims to investigate the effect of implementing cognitive load theory-based design principles in virtual reality simulation training of mastoidectomy.

METHODS: Eighteen novice medical students received 1 h of self-directed virtual reality simulation training of the mastoidectomy procedure randomized for standard instructions (control) or cognitive load theory-based instructions with a worked example followed by a problem completion exercise (intervention). Participants then completed two post-training virtual procedures for assessment and comparison. Cognitive load during the post-training procedures was estimated by reaction time testing on an integrated secondary task. Final-product analysis by two blinded expert raters was used to assess the virtual mastoidectomy performances.

RESULTS: Participants in the intervention group had a significantly increased cognitive load during the post-training procedures compared with the control group (52 vs. 41 %, p = 0.02). This was also reflected in the final-product performance: the intervention group had a significantly lower final-product score than the control group (13.0 vs. 15.4, p < 0.005).

CONCLUSIONS: Initial instruction using worked examples followed by a problem completion exercise did not reduce the cognitive load or improve the performance of the following procedures in novices. Increased cognitive load when part tasks needed to be integrated in the post-training procedures could be a possible explanation for this. Other instructional designs and methods are needed to lower the cognitive load and improve the performance in virtual reality surgical simulation training of novices.