Simulation Wars: A Competition to Increase Participation in Emergency Manuals Simulation Training and a Novel Tool for Rating Simulation Competitions

Background: Emergency Manuals (EMs) are valuable tools to guide healthcare professionals during anesthesia-related emergencies that require prompt diagnosis and effective treatment. Its use has been shown to improve simulated and actual patient outcomes in various operating room (OR) critical scenarios. However, integration of EMs into the standard practice of OR crises has been particularly challenging. Methods: The Simulation Wars was a competition created in China to increase awareness of EMs, the use of EMs among multidisciplinary teams, and to promote health care professional participation in EM simulation training. Each participant completed a post-competi-tion questionnaire detailing their opinions about the event. A novel scoring system was created and tested for use in simulation competitions. Results: Ninety-three percent of participants agreed that this competition could enhance the participation of simulation training in multidisciplinary health professionals and that they would continue participating in their hospital simulation training. The novel scoring system exhibited high internal consistency and good reliability. There is a strong positive correlation between the judges ’ score and the participants ’ score as R = 0.862, P = 0.013. Conclusion: The Simulation Wars can increase participants ’ understanding of how and why to use EMs, and enhance participation of multidisciplinary teams. The scoring system created for the competition has shown good reliability and justifies further development and evaluation. ABSTRACT

mergency Manuals (EMs) are tools to guide health care professionals when they are performing non-routine, critical tasks to reduce errors and increase efficiency. EMs are particularly useful for anesthesiologists who encounter rare clinical emergencies, such as amniotic fluid embolisms or ST-elevation myocardial infarctions, and need to remember exactly how to recognize and treat the crisis in a step-wise fashion. Often times, performing the right steps at the right time can be the difference between life and death in an OR emergency, as it was for a 4month-old boy who was saved from malignant hyperthermia with the guidance of EMs (1). The anesthesiologists in the OR noted that the EMs provided clear, concise instructions about the necessary steps for prompt diagnosis and rapid intervention, ultimately preventing a fatal outcome. Many efforts are being dedicated to the adoption of EMs in operating rooms worldwide because of its life-saving potential. Dr. Goldhaber-Fiebert proposed a 4-step method for successful implementation of EMs into the OR that consist of creating the content and design, familiarizing yourself through training, using the EMs, and integrating into the culture of the OR (2). However, many issues including scheduling conflicts, conflicting financial priorities, and staff training have stymied their applicability and clinicians are continuously searching for more effective methods to implement EMs into the OR.
One method for increasing routine EM use in ORs is through widespread participation in EM simulation training events. EM simulation training has resulted in a 17% reduction in missed critical steps in responding to various OR scenarios (3). Training has translated into improved simulated patient outcomes during simulated malignant hyperthermia and ST-elevation myocardial infarction during cesarean section (4,5). Further, these simulations have resulted in positive clinical outcomes as 45% of anesthesiology residents reported the successful use of EMs during clinical critical events, many agreeing that the EM immersive simulation training and selfreview positively influenced their later EM use (6). EM simulation training can be a very powerful tool for improving EM use, therefore it is imperative to understand how to increase healthcare worker participation in simulation training.
The Simulation Wars was a competition created in China to promote organized EM simulation training and increase training participation. Various Chinese hospitals competed to see which team can most cleverly simulate an OR emergency and clinically treat the patient with guidance from EMs. Each team was graded using a novel scoring system based on various skill elements portrayed such as scenario design, proper use of EM, and team communication.
The primary goal of this study is to understand if a simulation competition can increase EM training participation. The secondary goal of the study is to assess the reliability of the scoring system used in the Simulation Wars.  (7). Within a month of publishing the 2nd Chinese edition of the Stanford OR Emergency Manual, more than 25, 000 copies were downloaded. The two top Chinese anesthesiology societies encouraged anesthesiologists to incorporate the use of EMs in the management of critical events after appropriate training (7). In an effort to stimulate multidisciplinary team training for EM implementation, the Simulation Wars, an EM simulation competition, was launched as a pilot program in Zhongshan City, Guangdong, China. This study was approved by a local hospital authority, Zhongshan Boai Hospital. Consent was obtained from the individual hospital offices in charge of research.

Competition Design
Each Zhongshan City hospital was invited by the Zhongshan City Society of Anesthesiology to participate in the Simulation Wars. The competition's design consisted of preliminary rounds where each participating hospital submitted its September, 2018 Volume 5 Number 5 The Emergency Manual Simulation Wars Jeffrey Huang et al.
self-written and self-directed video demonstrating the application of the Stanford Operating Room Emergency Manuals in an anesthesia-related emergency. Each hospital was instructed to focus their simulations on crisis resource management skills by following the EMs. Each video was scored by the Zhongshan Society of Anesthesiology organizing committee based on a modified version of the Anesthetists' Non-Technical Skills (8). The organizing committee members watched each teams' video submission and provide a score. The committee members were instructed not to discuss their opinions about each video until after grading all videos. The individual team scores from each judge were added together to comprise a team' s preliminary round score. Because there are over 20 hospitals in Zhongshan City, the final round was restricted to the seven hospitals with the highest scores.
The seven hospital teams that received the highest combined scores in the preliminary round would advance to the final round, a half-day event where the finalists performed in a hospital auditorium in front of their peers, competitors, and the judging panel.
The final round consisted of each team performing a reenactment of their individual team submission videos. Each individual judge provided a score from 0 to 40 based on the modified scoring rubric immediately following the team' s live presentation. Each team's final score consisted of a combination of the 3 judges' scores from the final round. Both the online video submission and in-person crisis management performances were scored using the novel scoring system created for the competition. The final round judges' panel consisted of 3 anesthesiologists: Wenqi Huang, the president of Guangdong Society of Anesthesiology; Wuhua Ma, a well-known airway management expert; and Jeffrey Huang, a professor of anesthesiology at the University of Central Florida and the corresponding author of the present article.
Following the event, each participant was asked to complete a questionnaire (Appendix). To maximize total responses, the survey did not contain any open-ended questions. Survey questions included the 5-point Likert scale (strongly disagree to strongly agree).
Each participant retrospectively rated his or her own team's performance one month following the competition using the same scoring rubric as the judges. Although the participants knew their scores already, they were asked to be as objective as possible. While participants watched the other teams' performances, no participant was asked to grade another team besides his or her own.

Scoring System Development
The selection criteria of nontechnical skill items to be included in the new rating tool was based on principles derived from the Anesthetists ' Non-Technical Skills, and was guided by 2 fundamental considerations: 1. to focus specifically on behaviors and skills that can be observed during the EM simulation competition; and 2. to make the rating intuitive and feasible (8). Each element was defined as clearly as possible, taking extra precautions against combining several different skills into large categories (9, 10). Intricate constructs were condensed down to the most essential observable skill or divided into separate groups, resulting in the 10 components of the scoring rubric (Table 1). Particular emphasis was placed on skills that were relevant to sim- ulation training for EM application during a crisis such as scenario design, proper use of EM, effective communication, distribution of workload, patient assessment, leadership role, etc. The judges' panel rated each team' s performance using the scoring rubric in Table 1. A 4point scale was chosen to allow the observable skills to be rated, as follows: 4 good, 3 acceptable, 2 marginal, 1 poor, and 0 not observed.
Pearson correlation coefficients (R) were calculated between the judge mean ratings and the mean self-ratings to determine the interrater reliability. A correlation coef fi cient was considered statistically signi fi cant with a P value less than 0.05. Correlation coefficients greater than 0.75, between 0.75 and 0.4, and less than 0.4 were considered to represent excellent, good, and poor reliability, respectively (14). All tests were 2-tailed with a type I error rate of 0.05. For quantitative variables, mean and standard devia-tion (SD) were used.

Subjects
Participant and hospital demographic data are shown in Table 2. Seven hospital teams participated in the Simulation Wars with a total of 45 healthcare workers. The original intent was to select the top ten video submissions. However, because there were only seven submissions, all seven hospital teams were invited to compete in the final round. Each hospital consisted of 5 to 8 members from a vast number of disciplines in healthcare, age, and working experience. Each team had at least one attending physician, resident physician, and nurse. The participant ages ranged from 25 to 45 years old, with the average age being 32 years old. The average number of years of experience was 8.4, ranging from 3 to 19 years worked. All participants completed a post-competition questionnaire and graded their own team's performance to compare to the judges' grades of their own performance.

Primary Outcome
The primary goal of the study was to understand if a simulation competition can increase EM simulation training participation. Ninetythree percent of participants agreed that this competition could enhance the participation of simulation training in multidisciplinary health professionals and that they would continue participating in their hospital simulations. Ninetyfive percent of participants stated they would support the organization of next year's competition. However, only eighty-two percent of participants stated they would participate in next year' s competition. Complete survey responses are displayed in Figure 1.

Secondary Outcome
The novel scoring system exhibited high internal consistency as Cronbach' s α = 0.757 and good reliability as ICC = 0.757, P = 0.001, and 95% CI (0.25, 0.82). Pearson correlation coefficients were computed for the relationship between averaged judge score and averaged participant scores. Our analysis showed a positive correlation between the judges' score and the partici- Original Article Q1: I believe that the organization of the competition is very reasonable; Q2: I believe that this competition can enhance the participation of simulation training in multidisciplinary healthcare professionals; Q3: I believe that I increase my understanding of why and how to use EMs; Q4: I believe that I understand why simulation training is so important; Q5: I believe that I benefit from observation during the competition; Q6: I will continue participating in my hospital simulations; Q7: I will support the organization of next year' s competition; Q8: I will participate in next year' s competition; Q9: I believe that the scoring system is fair.

DISCUSSION
The Emergency Manual Simulation Wars was created in China to promote simulation training via a team-based EM simulation competition. To our knowledge, this competition is the first of its kind in China. In many ways, the event was a success as evident in the overwhelmingly positive feedback from participants, the involvement of numerous hospitals, and the support of the leadership from the local government health department and the local society of anesthesiology. Ninety-seven percent of the survey participants agreed that the organization of the competition is very reasonable and ninety-five percent agreed that they will support the organization of next year's competition.
The primary aim of this competition was to promote health care professional participation in simulation training. The survey results indicate that we are heading in the right direction towards accomplishing that goal. Ninety-eight percent agreed that the competition enhanced their understanding of the importance of simulation training (Figure 1). Ninety-three percent of the participants agreed that this competition can enhance the participation of multidisciplinary simulation training for EM implementation. However, only eighty-two percent agreed that they would actually participate in next year's competition. Further studies are necessary to compare future simulation training participation among the subjects who were involved with the competition.
The Simulation Wars was a meaningful educational modality as ninety-five percent of participants increased their understanding of how and why to use EMs. Increasing one' s EM knowledge is vital to its integration as physicians perform better in managing simulated cardiac arrests after they were trained to use ACLS cognitive aids (15). Familiarizing the team members to the EM location, content, and application was one of the four key steps towards proper integration into operating rooms (2).
While simulation participation is a dynamic learning approach, watching other teams perform can be just as effective in enhancing OR crisis management skills. Teamwork training via expert demonstration has resulted in similar teamwork behavior scores as actual participation in simulations and was more effective than traditional didactics (15). Demonstration-based methods provide many of the same educational benefits as participation in simulations but one glistening advantage in learning-by-watching is the much lower resources required to manage the event. Reducing the cost, manpower, and physical space makes future simulation events more manageable, affordable, and feasible. The Simulation Wars participants watched each team' s performance during the competition and ninetyeight percent agreed that they benefited from observing the competition. Therefore, future simulation competitions should incorporate an observational component to their events.

Rationale and Future Applications of the Novel Scoring System
A reliable assessment system is necessary to rate each team during the competition. We preferred the Anesthetists' Nontechnical Skills model to guide our scoring system for the assessment of anesthesia simulation training (Table 1). Non-technical skills can be de fi ned as "the cognitive, social, and personal resource skills that complement technical skills, and contribute to safe and efficient task performance" (16). Fundamentally, they assess participants' non-technical skills, and typically include situation awareness, decisionmaking, teamwork, leadership, workload distribution, and the management of stress and fatigue.
Nontechnical skills rating has been shown to require immense training to reach reliability, which is hindered by a large consumption of time and resources, essentially making a largescale assessment training almost unfeasible (17). A realistic scoring system for any simulation competition should be reliable after a limited amount of rater training and produce valid test scores (18). We attempted to legitimize the scoring system by comparing the participants' ratings of their own performance with the judges' ratings of their performance. The high degree of correlation (R = 0.862, P = 0.013) between judge score and participant score is a reflection of the reliability of the scoring system, indicating that the scoring system is encouraging and justifies further development and evaluation.
Reliability refers to the reproducibility of assessment data or scores, over time or occasions (19). In this study, two statistical tests were used to demonstrate the reliability of the scoring system, the Cronbach's alpha and the intraclass correlation coefficient (ICC). The internal consistency of the scoring system was high due to the Cronbach' s alpha greater than 0.7. Our result was consistent with Graham' s study which demonstrated that Cronbach' s alpha for the ANTS was greater than 0.7 as well (20). Therefore, this new scoring system has good internal consistency.
Interrater reliability was assessed via the intraclass correlation coefficient (ICC). Our study indicated that the interrater reliability between the judges and the participants was good with an ICC of 0.757 (P = 0.001). Experts with experience in simulation and crisis resource management can apply the Anaesthetists' Nontechnical Skills (ANTS) scale and produce reliable scores after training (21). Expert level background knowledge and complete extensive training before using the ANTS are recommended (22). However, because of the complexity of ANTS, even experienced clinicians and simulation instructors may have difficulty using the scoring system. As such, ANTS has not seen widespread adoption (20). Because of these limitations, the development of a new, simpler tool to assess the emergency manuals training is needed. Our study showed our modified ANTS scoring system can produce reasonable and reliable scores from both experienced and inexperienced clinicians. This finding can play a role in future simulation competitions as the new scoring system can be used by a wide audience with various training backgrounds. There was little confusion in what was expected for each non-technical skill between participants and judges. Teams that consist of substantial disparities in work experience and training level generate the potential for differing opinions on team performance. However, it appears as though the multidisciplinary teams agreed on the shortcomings and strengths of each presentation as ninety-one percent of participants felt that the scoring system was fair. We recognize that this is a preliminary test for evaluating the reliability of the scoring system, however, this is the first step towards a more sophisticated validation process. This event provides the groundwork for future simulation scoring systems.
The competition required each team to create their own simulation scenario demonstrating the proper application of the Stanford Operating Room Emergency Manuals. Scenario design was a major criterion in the scoring system because a high degree of realism is necessary in order to deter disbelief and maximize the educational benefits (23,24). High fidelity simulations promote the participants to become an engaged and motivated problem solver (25). Conversely, simulations that are not realistic have been noted as significant barriers for simulation-based education (26). Therefore, performance realism and participant engagement were included in the scoring system to enhance educational outcome.
Leadership was included from the ANTS as one component for the scoring rubric because of its pivotal role in successful emergency management. When leaders establish a structure within the team, the team dynamics work more efficiently and resuscitative performance increases (27). Building a team structure can be accomplished by verbal and non-verbal directions. Although the leaders instruct each team member what role to embody during the crisis, they garner trust and respect from the team by being malleable towards the needs of each member and situation. The teams with clear leaders giving directions embody the most control and least confusion. In contrast, as leaders are more "handson", there is a decrease in team structure, dynamics, and resuscitative performance (27).
Another essential component of team building is the consistency in approach and action. EMs play a pivotal role in team building as they provide a consistent flowchart, checklist, or algorithm during times of high stress. The leaders need to utilize their "recognition-primed decision making" where familiar conditions provide the platform to enhance the decision-making process (16). These instinctual decisions are often termed as a "gut reaction" and are best managed by the most experienced member. EMs provide a structure in crisis management to augment the leader's ability to motivate and direct the team. The initial steps for enacting the EM can sometimes be ambiguous. One approach could be for EMs to start by 3 steps: 1. acting immediately; 2. asking "who is the crisis manager"; and 3. assigning tasks (24). To increase efficiency and reduce confusion, the leader should assign specific responsibilities to team members. A recent intraoperative crisis simulation study had a crisis manager direct team members into 3 groups: procedure providers, recorders / readers, and runners (28). The participants believed that the initial checklist steps were easy to use and helped them feel better prepared throughout the simulated emergency. Therefore, proper distribution of workload was included in the Simulation Wars scoring system and is considered a key characteristic of EM success.
Team coordination and communication were also included in the scoring system due to their crucial role in EM success. While communication has been established as fundamental for intraoperative patient safety (29), breakdowns in effective communication have also been associated with perioperative complications and errors during handoffs (30). Effective communication should always be a component of simulation scoring systems for the proper EM application and, most importantly, for patient safety when using EMs in the clinical setting.
Patient assessment was the last scoring item drawn from the ANTS as situation awareness and decision making have been fundamental to many other nontechnical skills performance ratings in the OR (31)(32)(33)(34)(35). Patient assessment is guided by gathering and prioritizing information; recognizing and understanding the crisis at hand; balancing risks with benefits; selecting options; re-evaluating; and identifying outcomes. EM effectiveness can be attributed to its ability to aid healthcare professionals in recognizing OR crises and decide rapidly on the next course of action. Patient assessment is an appropriate conglomeration of the technical and nontechnical skills essential for improving patient outcome.
The validity of ANTS scoring system has already been assessed in previous studies (5). Our modified ANTS rubric varied minimally from the ANTS as outlined in the methods section. However, other methods of verification can be explored. Previous scoring tools have demonstrated validity in improved non-technical skills scores after teamwork training during laparoscopic cholecystectomies (33). While the Simulation Wars was unable to compare team performances pre-competition versus post-competition, the questionnaire provided optimistic responses that participants have a better understanding of how and why to use EMs. Next year' s competition design could include a pre -and post-competition simulation to measure improvement in actual team performance. In addition, it will be of great value to include a longterm assessment of team performance at one year following the competition to understand if long-term improvement in EM use and crisis management occurred (6).
One advantage in testing non-technical skills in the Simulation Wars is the broad nature in its application. The final round scenarios from the inaugural Simulation Wars competition included intraoperative acute myocardial ischemia, anaphylaxis, amniotic fluid embolism, local anesthetic toxicity, and cardiac arrest. Often at the forefront of care for intraoperative patients, these scenarios are pivotal for anesthesiologists and the perioperative surgical team to master. However, the scoring system is based on nontechnical skills, and therefore, can be applied to various scenarios. As it captures skills that are independent of medical specialty training, any observer from a variety of background can use the scale. With slight modifications to our scoring system, it can be applied to a wide range of competitions whose setting is outside of anesthesiology-related crises.

Limitations
Because this study incorporated survey respons-es from the participants, the accuracy and honesty cannot be verified and the survey tool was not validated beyond the initial survey inquiry.
One area for improvement in future studies could be the addition of qualitative data in addition to quantitative survey data. Open-ended questions provide healthcare workers' feedback regarding the setup of the event, the reasons for participants' EM support or skepticism, and how EMs influenced workflow (36). In the process of keeping the questions concise to maximize survey response, valuable qualitative data was not obtained. Qualitative data can provide experiences that are much more expansive and informative than quantitative data and should be considered in future simulation survey studies.
More competitions need to be organized to further test the scoring system. In future studies, one suggestion would be to video record the performance and ask external reviewers to rate the performance. Therefore we may have better data to test reliability. Because participants rated their own performance one month following the competition, recall bias involved in the participants' self-assessment of their performance can be reduced by asking participants to grade their team immediately following their final round performance.
Another area for improvement that can be addressed for future simulation events could be to further diversify the hospital teams beyond physicians and nurses. Seven percent of participants did not feel that the Simulation Wars enhanced the participation of the multidisciplinary team ( Figure 1). This could be explained because one team of 8 consisted of 6 physicians, potentially limiting the diversity of ideas in the team. Further, the overwhelming number of physicians could hinder the participation of those from other educational backgrounds out of fear of uncertainty and judgment in the interprofessional setting (37). Simulation competitions are successful when they consist of a multitude of healthcare professionals including paramedics, respiratory therapists, medical students, and physicians from different specialties such as internal medicine, anesthesiology, and critical care (37). Future events should include varying depths of healthcare specialties and levels of education.
As this was the Simulation Wars' first year in Original Article action, the response from hospitals was lower than expected. The original goal of the competition was to select the top 10 hospital video submissions. Because we only received seven entries, we invited each team to perform. As the competition gains in popularity, we expect more video submissions. This will provide more opportunity to improve the scoring system and competition design to maximize the Simulation Wars' educational impact. Regardless of competition submission number, the first year of the Simulation Wars is still considered a success in terms of EM awareness and trialing the novel scoring system. The Simulation Wars is intended to be a multi-year project that will continuously develop as we understand ways to improve.

CONCLUSIONS
Ultimately, the goal of the Simulation Wars was to increase EM simulation training participation via involvement in a competition. This study provides insight into the positive impact that simula-tion competitions can have on promoting EM simulation training. Most of the participants enjoyed the opportunity to interact with the multidisciplinary team and learned a great amount about the applicability and advantages of EMs. In addition, they increased their understanding and willingness to support and participate in future EM simulation training. The scoring system created has shown to be a reliable source for assessing nontechnical skills and lays the groundwork for future simulation events. Barriers to EM implementation have been identified and are being addressed to increase the influence that simulation competitions can have on establishing the routine use of emergency manuals. We disclose many challenges that a simulation competition can encounter and provide suggestions for future events.
The authors have no other potential conflicts of interest for this work.
The Emergency Manual Simulation Wars Jeffrey Huang et al.