Pandem-Sim: Development and Pilot Testing of a Live Simulation of Infectious Disease Outbreaks

Pandem-Sim is a live, distance-learning simulation where 10th-12th grade students perform as epidemiologists to investigate infectious disease outbreaks. Prior to a mission, teachers review content and introduce students to the sim interfaces. During the mission students analyze data concerning demographics, symptoms and patient histories for a disease outbreak somewhere in the world. Student teams make initial diagnoses and order medical tests. If the tests do not confirm the diagnosis, the process is repeated. Following correct diagnosis, students identify the transmission method, prescribe treatments, recommend procedures to contain the outbreak, and issue a public health alert. During the 90 minute simulation the Chief Epidemiologist (a STEM educator at Wheeling Jesuit University) asks students critical thinking questions. Pilot tests were conducted at four schools (78 students) with data analyzed from pre-post surveys, classroom observations, interviews with Chief Epidemiologists and teachers, and detailed logs of student work. The first pilot test revealed needed revisions of procedures and teacher preparation, which were made before subsequent testing. Differences in students’ pre and post surveys showed a small but statistically significant increase in content knowledge. Additionally, students reported that they enjoyed participating in the experience and it increased confidence in their abilities necessary to pursue a health related career.


INTRODUCTION
Excellent STEM programs are frequently introduced in specific cities or even in single schools, but often they are not available for use across the nation and are sometimes stopped when funding ends.A solution to these problems was initiated twenty-five years ago when the Challenger Learning Center (CLC) at Wheeling Jesuit University (WJU) introduced Operation Montserrat, a live, role-playing simulation (sim) that uses distance learning technologies to bring engaging and realistic learning experiences into classrooms.Since then, often working with developers at WJU's Center for Educational Technologies, 25 additional sims have been created and used in all 50 states and in 22 countries.
Since 2006, NIH's Science Education Partnership Award (SEPA) program has awarded funding to WJU to create two life science sims, CyberSurgeons and Pandem-Sim.In CyberSurgeons (http://www.e-missions.net/cybersurgeons/),high school students portray medical personnel on a research ship traveling up the Amazon River, diagnosing medical maladies presented by crewmembers and Brazilian citizens along the shore.CyberSurgeons demonstrates another benefit of sims.Although CyberSurgeons funding ended in 2012, the CLC continues to share this educational experience and earn revenue by presenting the mission in classrooms across the USA and the world (68 missions worth $30,000 in the 2016-7 school year), continuously generating income to sustain and update the project.
The key to the success of CLC sims is that they immediately engage students in data collection and analysis tasks that actual scientists would do during a crisis.In Operation Montserrat, students respond to emerging information about a volcanic eruption and an incoming hurricane to rescue the island's population.During Space Station Alpha, teams work to protect scientists on board the space station as a solar flare threatens to damage systems, and the station has to survive being cut off from solar power during an eclipse.Although formal studies document the learning achieved by Challenger missions (Chen and Howard, 2010;Calinger and Piecka, 2011), the best evidence that these sims are compelling and educationally valuable is that teachers and school districts return every year, paying between $200 and $550 per mission, to expose new cohorts of students to the learning experiences.Montserrat has earned more than a million dollars in revenue, helping support CLC's continued existence.

PROGRAM DESCRIPTION
Pandem-Sim builds on the 25-year history of the CLC and Center for Educational Technology developers at WJU in creating and delivering innovative, interactive learning experiences via distance learning technologies.A major goal of the Pandem-Sim mission is to increase high school students' knowledge of human infectious diseases, microorganisms, the immune system, and the epidemiological processes used to investigate disease outbreaks and health conditions.
The Pandem-Sim simulation follows the process that epidemiologists at the Center for Disease Control (CDC) actually use to identify and respond to unknown disease outbreaks (CDC, 2011).Figure 1 identifies the main tasks in the CDC process and how they are distributed across the three specialist groups that constitute a team in the simulation.There are two students in each specialist group and the six students comprise a team that has to share data and collaborate to complete the mission.Typically, a mission is run with four or five teams, each investigating a different disease, working simultaneously and under the guidance of a CLC educator acting as a Chief Epidemiologist (CE).
Each mission begins with the CE, visible to students via Skype or Google Hangouts, giving a 3-minute overview of the fictional Pandemic Disease Center (our version of the CDC) which students will be part of during their mission.Students learn that diseases are rapidly spreading into new places because of changing climate and rapid air travel.The interface to the mission screens is quickly reviewed and then the students start their team investigations.First is a video of one of the CLC educators dressed as a doctor describing the scenario at a location somewhere in the world where people are reporting symptoms that have not yet been diagnosed.
Students then receive case reports from 25 patients including symptoms and demographic data.Each specialist group analyzes part of the data to determine which symptoms are most common and/or unique, to tabulate and plot data to look for age and gender clues, to construct an epicurve that shows the daily development of reported cases, and to plot on a map the temporal spread of the outbreak.
Next, students view patient interviews.Patients report symptoms, activities, disease progression, and other clues that help students identify the disease, transmission method, and in some cases, Patient Zero.The videos provide a human perspective of the disease's impact on patients.The videos show WJU students from the countries where the outbreaks are occurring.The videos are close-captioned which allows students to work in pairs and to review the videos without raising the noise level in the classroom.
Based on the scenario, case reports, and patient videos that they have analyzed, students write a case definition that summarizes the basic details of the outbreak.Like a newspaper story it tells the who, what, where, and when of the outbreak that students need for their next task which is to diagnose the disease.They use Epi-Net, which is an interactive database that comprehensively lists symptoms, pathogen characteristics, disease progression, treatment and transmission for many diseases.Epi-Net is a tool for students to apply their knowledge, deductive reasoning and critical thinking skills to identify the disease they are investigating.
Students select a symptom such as fever and Epi-Net lists all of the diseases for which fever is a symptom.They will see that fever characterizes many infectious diseases and it will be more efficient to choose a less common symptom such as bleeding from the eyes.In this case, Epi-Net would identify only a few diseases, all hemorrhagic fevers, to be investigated.Students examine a number of diseases that match the major symptoms, remembering information from the video interviews and their data plots, and then submit their initial diagnosis, followed by ordering one or more tests (such as real-time reverse transcription PCR, ELISA, or viral cultures).Students receive an alert with test results that either confirm or deny their diagnosis.If the diagnosis is wrong, they go back to Epi-Net and repeat the diagnosis process.Often students are incorrect because many diseases have similar symptoms and it requires careful combination of all data to reach a correct diagnosis.
Following a correct diagnosis, students move to the final tasks of recommending treatment protocols for patients and ways to contain the spread of the infectious disease.Their last task is to write a Public Health Report to characterize the disease occurrence as an outbreak, epidemic or pandemic, and to inform the public about it and how to avoid and treat it.This report is orally presented to the CE.Then reports of a different outbreak in a different part of the planet draws them to another analysis to solve a new disease mystery and save more lives.
Throughout the mission the CE tracks team progress both in a software dashboard that displays student work, and by observing their interactions in the classroom via Skype or other video-conferencing software.The CE asks guiding questions, especially if students appear stuck or if they are racing ahead.The questions specifically lead students to critically think about their data and analyses.Also alerts appear during the course of the mission providing images of patients and their symptoms, updates on the number of patients, and disease-caused deaths.After a correct diagnosis, alerts content changes to provide additional information about the disease's past occurrences and treatment conditions which often must be done in the field.
In the week to ten days prior to the mission day, teachers conduct a review of relevant epidemiology content provided as lesson plans and support curriculum in the Pandem-Sim website (http://www.pandemsim.com/).The pre-mission preparation materials review the epidemiological process and the tasks students will do during the mission.Additionally, the support curriculum includes detailed information about the different types of pathogens such as bacteria, viruses, protozoans, etc.This is especially useful if a teacher has not already covered this content in the classroom.The final parts of the pre-mission preparation materials include information on relevant standards and practical details for actually conducting the mission.

PILOT TESTS
Pandem-Sim is a complex interactive activity with 14 different tasks for students to complete.The mission was designed to require collaboration between specialist groups on each team, and for the teams to frequently interact with the Chief Epidemiologist.The question is: Did it all work?To find out, high school biology teachers who had previously conducted our CLC missions were recruited for pilot tests.All the teachers had five plus years of teaching experience and they understood the process of our live simulations, could recognize successful missions, and contribute useful feedback for improvements.The teachers received a free simulation mission for their class and $150 for their participation in the evaluation study.
Four pilot tests were conducted with students and teachers testing the entire process to be used for the subsequent field evaluation, including pre-mission surveys, pre-mission preparation curriculum, participating in the live sim, and finishing with the post-mission survey.Once all of the survey instruments and procedures were established they were submitted to the Wheeling Jesuit University IRB committee prior to pilot testing (IRB 001-16).Teachers collected signed student and parental permission forms, and submitted a letter of cooperation from authorized personnel at their school along with a teacher consent form.All student demographic information and survey results were recorded using a coded ID number to provide confidentiality.
To allow all of the students to experience the live simulation we asked each teacher to randomly divide the class in two, with a comparison group taking the post intervention survey after pre-mission preparation but before doing the mission; the other half of the students took the survey after the mission.This way all of the students could take the mission at the same time.In all, 78 students provided matched pre-and post-surveys.
A variety of types of data were collected to evaluate the effectiveness of the live simulation.For all four missions we observed the students working in their classrooms using the Skype video connection, and for two missions, development team members observed on site.Following each mission the development team debriefed the Chief Epidemiologist and classroom observers, and followed that by interviewing the teacher who had discussed the mission with his/her students.Additionally, we analyzed data from the pre-and post-surveys from students and teachers.Finally, we reviewed mission logs created by the Pandem-Sim software that time-stamped and recorded each decision students made as they conducted their 14 tasks.

Observations.
The first pilot test in a rural school in southwestern Pennsylvania was the most informative for our development team.Although our internal testing had consistently yielded data that allowed informed revisions, testing with high school students naturally generated additional relevant and valuable information.We witnessed many behaviors and actions that we had not contemplated and immediately identified a number of changes to be made before the next pilot test.
For example, to encourage collaborative interactions among specialists on each team we had designed the software to not allow advancement to subsequent tasks until all three specialists groups finished and discussed certain shared tasks.But, of course, the groups did not work at the same speed, and the faster ones had nothing to do while waiting for the slower ones.This happened in three places during the mission; we subsequently removed two of these waiting periods and maintained only the most critical one where students must concur on their initial disease diagnosis.With modified software and student procedures, participants in the next three pilot tests progressed more surely through their 90-minute simulations.
The subsequent three pilot tests were at a medical careers-focused high school in South Carolina, an advanced biology class in a Louisiana high school, and an honors biology class (with a few AP Biology students) from a school in Pennsylvania.All three of these classrooms of students were at a higher level of academic preparation than the first pilot test students, and they used the improved simulation.

Pre and Post-Mission Surveys.
Pre-post student surveys provided details of changes in attitudes towards science, epidemiology and career interests.Student surveys included three components from Inspiring Careers in STEM (Berk et al., 2014): two dichotomous questions about pursuing healthcare careers, two open-ended questions where students noted their first and second career choices, and three items scored on a 5-point, strongly disagree (1) to strongly agree (5) Likert scale.Eleven items from the second instrument, STEM Career Interest Survey (Kier et al., 2014) used a 5-point scale ranging from 1 (strongly disagree) to 5 (strongly agree) to query students about their science attitudes and career interests.From the STEM Semantic Perception Survey (Tyler-Wood, Knezek, and Christiansen, 2010), the surveys used five items on a 7-point semantic differential scale with one adjective at a one and the opposite adjective at the seven on the scale.Student pre-and post-surveys used the sums of each scale to examine significance.There was no significant difference from pre to post survey for the Inspiring Careers in STEM, STEM Career Interest, or STEM Semantic Perception items.
To measure changes in content knowledge, we added 36 additional multiple choice questions (also from reliable and valid sources) focused specifically on the pre-mission preparation content, infectious disease content in the sim, and the CDC process for investigating infectious disease outbreaks as used in our simulation.The post-survey was identical with the addition of four questions appraising the student experience with the sim.
Teachers answered 19 pre-mission questions on their own demographics, preparation to teach the epidemiology content, and their observations of student interests in careers.Following the mission the teachers answered another survey with 29 questions focused on student interest in related careers, the time teachers devoted to pre-mission preparation with their students, how relevant it was, and if the teacher would be interested in purchasing the simulation ($550) for future classes.Students and teachers completed the surveys online using Survey Monkey.
The following results come from the project's external reviewer, Dr. Kristine Chadwick, who analyzed the survey data and provided the demographic information in Table 1.All students showed small but significant gains in epidemiological knowledge from pre-test to post-test, including the comparison group which took the post-test before experiencing the simulation.When queried about career awareness, about half the students reported an interest in learning more about epidemiology in the pre-test survey using a yes/no response.Approximately 20% of the students who originally said they were not interested changed post-test response to a "yes".Similarly, 15% who responded in the pre-test that they were not interested in pursuing a career in epidemiology or biomedicine changed to a "yes" in the post-test.In general, there were no statistically significant differences in knowledge scores by gender, but females scored significantly higher both pre-test (p = 0.006) and post-test (p=0.05) on biomedical career interest.
A majority (56.4%) of the students reported that they enjoyed participating in the simulation a great deal, and 21.8% said that they enjoyed participating somewhat.When asked if participation in Pandem-Sim increased their confidence in their abilitiy to pursue an epidemiology or biomedical or healthcare-related career, the majority of students noted a positive connection (51.7% selected somewhat and 31.0%a great deal).Another question asked students about the extent they felt their participation in Pandem-Sim had increased their interest in pursuing a career in epidemiology or a biomedical or healthcare-related field.Results indicated a majority of the students would consider a career in epidemiology or the biomedical field: 61.7% selected somewhat, followed by 24.1% a great deal, 10.3% a little, and 3.4% not at all.All of these results are based on a small sample, and the field tests surveys encompassing about 1350 students should yield more reliable data.

Teacher Feedback.
Teacher responses from the surveys indicated they felt well prepared to teach the preparatory materials using the Pandem-Sim premission lessons.Teachers augmented mission preparation with textbook information, Pandem-Sim supplemental curriculum, and NIH infectious disease resources.Teachers engaged online with the Chief Epidemiologist during training sessions and reported the activity to be valuable.To integrate Pandem-Sim into their normal lessons, teachers used a variety of approaches including assigning readings and questions from the premission lessons, asking questions to improve learning, discussing biomedical and epidemiology careers, and reviewing prior units on microorganisms and chain of infection.
The four participating pilot test teachers had eight to 25 years of science teaching experience, with an average of 13.5 years.The purposeful sample had previous experience using CLC simulations in their science classes in both public and specialized schools focusing on healthcare curriculums.The teachers piloted Pandem-Sim in the following classes: Introduction to Zoology and Biology II (combined), Biology II, Honors Biology, and Health Science III.Three of the teachers were female and one was male.Teachers reported preparation time with their students in blocks of 20 minute intervals.One teacher reported 81-100 minutes of instructional time while another spent 141-160 minutes, and two spent between 241 and 260 minutes.
When asked to what extent the live simulation helped their students learn content about epidemiology, teachers all replied "very much".Another survey question asked teachers about student engagement during the live simulation; all teachers reported that students were very engaged.Field notes from the Pandem-Sim team in addition to teacher notes and emails identified high levels of student engagement in the live simulation.The project teams' field notes and teacher comments identified students' staying on task during the simulation and high levels of collaboration between specialist teams.One teacher reported, "My students really enjoyed the simulation.Some of my students who I expected to be apathetic or not engaged at all said that they loved it and asked if they could participate again next year even if I don't have them in class." The Role of the Chief Epidemiologist.
During the development of the simulation we had worried that because students received, analyzed and reported data through online forms that there would be too little interaction with the Chief Epidemiologist (CE).We observed that this could happen but also knew the experience was richer and more interesting for students if they periodically reported to and answered questions directly with the CE.The CE would call up a whole team to a camera at the front of the room to discuss their team progress.This produced animated interactions within teams as they considered how to interpret their data.We introduced more of these interactions in subsequent pilot tests, making the student experience more lively and thought-provoking.
The Pandem-Sim project team observed delivery of critical thinking questions by the Chief Epidemiologist.Formative evaluation after each pilot included assessment of the quality of student responses to critical thinking questions.The team also identified possible improvements in the selection and phrasing of critical thinking questions.As part of pilot testing improvements the Pandem-Sim team addressed these improvements with the CE through discussion debriefs; refinements continue through field testing.

Mission Logs.
Mission logs collected by the simulation software precisely tracked times for students to complete each of the 14 tasks (which averaged 4 to 21 minutes).The logs also revealed that students frequently backtracked to review previous data/results needed to make next-step decisions.The mission logs also highlighted student mistakes such as selecting a diagnosis but not clicking the "Submit" button.Also, the observation that students selected various hemorrhagic diseases before finally submitting one as their diagnosis led us to revise the diseases descriptions in Epi-Net to more clearly differentiate symptoms and disease characteristics.The logs documented that each team that completed a disease took about 60 minutes.

The Next Step.
Successful completion of pilot testing and tweaking of mission components were followed by field testing/formal evaluation starting in January 2018.Teachers were recruited from across the USA, with the goal of gaining feedback and results from minority teachers proportional to their occurrence in US school systems.We anticipate testing will be completed by May 2018, with data analysis starting immediately afterwards.Forty-six field tests will be completed in order to determine an effect size.

CONCLUSIONS
Pilot testing identified where the simulation was effective and where software and procedure changes were needed to make it flow more effectively and to enhance critical thinking.It revealed needed changes to teacher training so that all teachers followed pre-mission procedures to improve student performance during their missions.The Chief Epidemiologist, an experienced CLC mission leader, was a major factor in identifying where, when and how to enhance student involvement.The CE observed student interactions and could see from the backend dashboard when work was wrong or faltering, and then could offer suggestions and encouragement.
The Pandem-Sim project was originally funded to develop two learning projects: the live simulation described here, and the Pandem Disease Center, a massive teacher resource devoted to epidemiology and related sciences.In 2015 additional funding allowed adding a third component on the use of Big Data in modeling infectious disease outbreaks.The Pandem-Sim website (http://www.pandemsim.com/)links to content for all three of the project components.Evaluation of the latter two components is ongoing.Screen shot of a scenario video that introduces students to the disease to be investigated and the location of its outbreak.The Pandem-Sim website contains all needed content and instructions for preparation and running of the live simulation mission.

Figure 1 .
Figure 1.Pandem-Sim task diagram lists tasks performed by students in each of three specialist groups.

Figure 3 .
Figure 3.Screen shot of a patient interview video which describes an individual's account of symptoms, disease progression, and how they may have been exposed.

Figure 4 .
Figure 4.Epi-Net is the interactive database used by students to diagnosis the disease they are investigating.