Seattle Children’s Virtual Research Training Program: Pivoting to Remote Science Education for High School Students

: Seattle Children’s Research Institute has built a series of in-person education programs to inspire and empower students to explore futures in biomedical research and healthcare. The COVID-19 pandemic prevented us from offering an in-person laboratory program, forcing a rapid pivot to an online format. The Virtual Research Training Program (VRTP) was a one-week summer experience for high school students, including students from groups that are under-rep resented in STEM. The curriculum introduced topics such as biochemistry, immunology and immunotherapy, and global and public health. Also included were laboratory demonstrations to emphasize cutting-edge applications for healthcare and discussions regarding college and career preparation. Key challenges included converting the in-person curriculum into a digestible virtual format, becoming proficient with the technology to provide for a seamless end-user experience with equi table access, and establishing quantifiable metrics for evaluation. Students reported statistically significant gains with large effect sizes in knowledge about science concepts and laboratory procedures, and in preparation for college and future STEM careers. Students were also engaged by asking questions, indicating their active participation despite the online environment. This article discusses the adaptation of an in-person laboratory program into a virtual program as a potential model for in creasing remote access to science education.


INTRODUCTION
There is great demand for high-quality introductory programs for high school students looking to pursue a career path in the biomedical sciences. Many of the occupations projected to grow the fastest in the coming years are in the healthcare field or require significant STEM training (U.S. Bureau of Labor Statistics, 2020). However, overall high school science literacy has remained low. According to the National Assessment of Educational Progress (NAEP)-also known as the Nation's Report Card-only 21% of students in grade twelve scored at or above the level of Proficient in science subjects in 2009, and only 22% met this standard in 2015 (U.S. Department of Education, 2016). Many under-served high schools lack the resources to prepare students for working in real-world research laboratories. Community colleges and research organizations have stepped in to bridge the gap and offer access to authentic laboratory experiences and interactions with professionals in the STEM fields (Learning Undefeated, 2020; University of Washington, 2020).
The Science Education Department (SED) at Seattle Chil-dren's Research Institute (SCRI) is committed to providing access to high-quality science education for under-resourced schools across Washington state. Since our establishment in 2008, we have helped inspire and train the next generation of scientists and healthcare professionals, focusing on under-served and under-represented students, at no cost to participants. We have built a series of programs that serve students at different stages in their developmental and educational pathways (Jones et al., 2019). One such program is the summer Research Training Program (RTP) for high school students looking to pursue future careers in biomedical research. This competitive program brings students to the Science Discovery Lab, a dedicated classroom for up to 32 students at the Research Institute, to use authentic research equipment and learn from Ph.D.-level scientists and educators. There is also a strong college and career preparation component that lays the groundwork for future pursuits in the fields of science and healthcare. However, the COVID-19 pandemic necessitated the can-cellation of the in-person RTP in summer 2020. We immediately began planning for a virtual replacement, and opened participation to all RTP applicants without limitations. The challenges in its development were many, including figuring out how to rapidly innovate and adapt a hands-on in-lab curriculum to a virtual format that would also be informative and meaningful for students. Other logistical challenges arose such as access to technology and expertise for using that technology, both for SED staff and for students. It was also important to evaluate and measure the successful implementation and execution of the VRTP, and to ensure that students were actively engaged with the program, rather than just passively watching content on a screen. Ultimately, the summer research program is intended to inspire and empower high school students interested in the field of biomedical research, and to provide resources for future learning as they prepare for college and careers in science. Here we describe the adaptations made to the program for a new online format.

METHODS
The scope of the summer program was condensed from the original four-week schedule for the in-person RTP to a one-week, ten-hour format for the online VRTP. The twohour limit per day was intended to minimize fatigue from video conferencing. The one-week session was first conducted in July 2020 and then replicated in August, to provide options for students' schedules. Minor adjustments were made to the second session based on feedback from the first session; both sessions were functionally identical. Students could also choose to watch a recording of the session on their own schedule. The program consisted of scientific lectures, demonstrations of laboratory experiments, and panel discussions.
The Science Discovery Lab was converted into a filming studio with separate areas for lectures and laboratory demonstrations. An adjacent conference room was also repurposed to allow staff to serve as moderators for the live-stream and to provide additional on-site support. Seven SED staff were directly involved in the production of the VRTP, including the senior director, two lead scientists as primary lecturers, one scientist for lab demonstrations and as a guest lecturer, and two scientists and a curriculum specialist as moderators of the WebEx Q&A and for technical assistance. The SED staff members working in the lab observed physical distancing of at least 6 feet and wore regular personal protective equipment (lab coat, gloves), supplemented by transparent face shields when speaking on camera or face masks when not on camera. Other staff members and the guest speakers worked remotely in different rooms on-site or at home.
We utilized the Cisco WebEx Events video conferencing suite to allow for the rapid deployment of assets to host, share, and record virtual lessons with an external audience (Gottlieb et al., 2020).
An important factor leading to the use of WebEx was the ease of accessibility for students using either a computer or a mobile device, including their ability to simultaneously watch the session while also typing their questions. In the lab, cameras in laptops and mobile devices were used to provide multiple views for the lecture and lab demonstrations. Student interaction was most efficiently conducted using the Q&A function within WebEx, wherein students could ask questions without other students seeing who had asked the question. This allowed the moderators to de-identify and filter questions and forward them to the lecturers, who would then repeat the question aloud before answering on camera. The moderators also logged all questions, including questions that could not be answered during the live session, which were then answered later the same day in a Q&A document. The presentation slides, the Q&A document, and supplementary materials were posted to Google Classroom to provide free and easy access for students. Also, recordings were made natively through WebEx and posted on YouTube for students to view off-schedule. In addition to flexibility of scheduling, the recordings also allowed students who experienced issues with internet connectivity to still participate once the connection was restored, albeit without the ability to ask questions in real-time. Finally, WebEx was also used to set up small groups for optional office hours with up to three students and one SED staff member. These office hours were conducted outside the ten-hour standardized program and allowed students more options for asking questions in a less public environment, without a requirement for sharing audio or video. Accordingly, these office hour sessions and the questions asked were not recorded.
Each day of the VRTP had a different scientific focus (Table 1). Days 1, 2, and 4 also included engaging laboratory experiments designed to introduce new students to the subject. The first demonstration related to biochemistry was a gel electrophoresis experiment with connections to gene editing in diabetes treatment. The lab experiment for immunology focused on how scientists use a patient's own immune system to fight cancer; this experiment highlighted the isolation of white blood cells using a Ficoll gradient and quantification with a hemocytometer to show how white blood cells are used for immunotherapy applications. Another demonstration was related to infectious diseases and public health and used an enzyme-linked immunosorbent assay (ELISA) to screen for HIV in simulated patient samples. A supplementary public health demonstration showcased healthy and simulated smoker's lungs. These demonstrations were performed in real-time as much as possible to allow students to make observations and to provide for an authentic laboratory experience; exceptions to this include expedited gel electrophoresis, centrifuge spins, and incubations by using a second identical set of pre-prepared materials prior to the final observations.
The curriculum also included two panel discussions on college and career preparation. Day 3 featured a lecture on career preparation, including how to prepare a resume, cover letter, and elevator speech, and how to prepare for an interview. Day 5 consisted of a discussion on college preparation and featured two guest speakers who are current college students that had previously participated in our summer SED program in 2017, and two SED staff members who had recently graduated from college in 2016 and 2019. Three of the four speakers also self-identify as women of color from groups that are under-represented in the biomedical field. A conscious effort was made to highlight these diverse voices and to provide relevant and recent near-peer experiences for the students regarding the college experience.
Comprehension of the lecture topics was assessed through a quiz with 10 multiple-choice questions hosted on Kahoot (Licorish et al., 2018;Wang et al., 2020). This paid online platform provided the means to create, distribute, and record completion of the quizzes that covered the concepts presented in Days 1-4. Additionally, students could easily access Kahoot from a computer or mobile device. The quizzes were intended to function as checks for comprehension but were not graded nor analyzed. The deadline for completion of the quizzes was in mid-September, four weeks after the end of Session 2, to allow for sufficient time for students who chose to watch the recordings. Students that completed all four Kahoot quizzes received a Certificate of Completion. Upon completion of the program, students were asked to fill out a post-session survey hosted on REDCap, an electronic data capture software for obtaining research data (Harris et al., 2009). This platform allowed for the gathering of anonymous, retrospective data in a rigorous manner, including students' knowledge on various topics on a 4-point Likert scale, both before and after completion of the program. This survey also solicited anonymous and constructive feedback for future improvements.

RESULTS
We received 176 applications for the in-person RTP before the program was cancelled. Open registration for the virtual program was offered to all RTP applicants, as well as students in the Biomedical Research and Global Health Program at Puget Sound Skills Center, a specialized career and technical education school. We confirmed 116 registrations prior to the start of the virtual program. The average attendance was tracked for each day of the live sessions by monitoring the number of participants in the WebEx call. Session 1 had an average of 52 students per day with a maximum of 58; Session 2 had an average of 27 students per day with a maximum of 30. The number of participants remained steady during the sessions. Ultimately, 102 students completed the VRTP's requirement of the four Kahoot quizzes: 62 students in Session 1, 31 students in Session 2, and 9 students who watched the off-schedule recordings. For comparison, the maximum capacity of the cancelled in-person program is 32 students.
Students self-identified as either well-represented or under-represented on their applications for the in-person RTP ( were able to link their VRTP registration to their previous RTP applications. A few students who did not originally apply for the RTP were not asked to provide this information. The language used for the diversity information is adapted from the Notice of NIH's Interest in Diversity (National Institutes of Health, 2019). Of the 102 students who completed the VRTP, 97 students completed a post-session survey and rated the program on a 5-point Likert scale, anchored from Poor (1) to Excellent (5), with 92.8% of students rating the program as either Very Good (4) or Excellent (5). The overall rating is comparable to the overall satisfaction ratings for previous summer SED programs for high school students that were conducted in-person (unpublished data).
Students were able to submit questions during the live sessions using the Q&A function, and the questions were recorded by moderators. Most questions were then repeated verbally for all attendees and answered by the lecturers. All questions were answered either during the live session or in a downloadable document by the end of the same day. The questions broadly covered three topics: lecture and lab content, college and career prep, and program administration. For the average daily attendance of 79 students in both Sessions 1 and 2, 62 students overall asked 310 questions. Of this number, nearly 80% asked at least one question and almost a third of them asked five or more questions. Of particular interest, 7.6% of students asked a question every day ( Figure 1). Furthermore, 35 students in both Sessions 1 and 2, or 44% of the overall average daily attendance, participated in optional office hours with five SED staff members. The questions asked during the office hours were not recorded nor included in this analysis.
To document the immediate impact of the VRTP, we administered a post-session survey. Students rated their level of knowledge, before and after the program, on a 4-point Likert scale, anchored from "None -Have no knowledge of the content" (1), "Low -Know very little about the content" (2), "Moderate -Have basic knowledge; there is more to learn" (3), and "High -Consider myself very knowledgeable" (4). A 4-point Likert scale was chosen to simplify the choices and to preclude a neutral response, while also providing a sufficient continuum to capture changes from pre to post (Lang et al., 2017). This technique, called a retrospective pretest, is used when participants may inaccurately estimate their knowledge or attitudes at the start of an intervention due to limited experience (Pratt et al., 2000).
Students answered six items about the science concepts and laboratory procedures, and six items about college and career preparation. Evaluators measured the internal consistency reliability of the two sets of items using Cronbach's alpha. The pre and post alphas were at least 0.70 (science concepts/labs: pre = 0.765, post = 0.761; college/career preparation: pre = 0.773, post = 0.701), justifying their grouping into scales (Nunnally, 1994). Participants moved from low to moderate knowledge of science and STEM careers before the program to moderate or high levels of understanding afterward (Table 3). They reported greater gains in their understanding of science concepts and lab procedures than in college and career preparation. Changes in both outcomes were large and statistically significant based on conventional effect size interpretations (Fritz et al., 2012).
Exploratory analyses of individual items reveal topic-specific gains. We conducted Wilcoxon signed rank tests and lowered the threshold for statistical significance to account for multiple comparisons (Bonferroni-adjusted alpha per item set: 0.05/6 = 0.008). We have reported individual ordinal response items with nonparametric tests of association (Carifio et al., 2007;Lovelace et al., 2013). Students' ratings increased significantly for all items, with large effect sizes (Tables 4-5). As a general rule, r values of 0.10, 0.30, and 0.50 represent small, medium, and large effects respectively (Fritz et al., 2012). Effect sizes should also be interpreted within the frame of reference of a specific set of comparable intervention studies, or relative to other items in a set of analyses.
Students appear to have learned the most from the immunology and immunotherapy lecture as indicated by the large increase from pre to post scores. Less than a quarter of all students reported either moderate (20.8%) or high   ly on, we identified several key challenges to providing our students with a high-quality program while also ensuring equitable access. First, it was crucial that we convert the curriculum and materials designed for a lengthy in-person laboratory program into a digestible virtual format that would still be engaging, informative, and meaningful for the students. Next, we needed to become proficient with the technology and other virtual tools needed to host a live-streamed event and provide for a seamless end-user experience. Furthermore, it was crucial that the video conferencing technology be accessible for a wide range of devices and various levels of internet connectivity. Finally, we needed to define and establish quantifiable metrics in order to evaluate student content knowledge and engagement, especially in the absence of direct, two-way contact with students. The discussion that follows summarizes the key decisions we made to address each challenge and reflects on lessons learned. When considering what topics to address, we drew upon our expertise as a leading pediatric research institute and the many cutting-edge projects currently underway across our seven research centers. It was important to introduce fundamental concepts in biological research while also providing solid connections to improving healthcare. Most of the lecture content was readily adapted from the cancelled in-person summer program and from lessons created originally for students in grades 9-12 for half-day field trips during the regular school year. Of note, the biochemistry lab demonstration regarding gel electrophoresis and gene editing in diabetes treatment was previously in development and was introduced to students for the first time during the VRTP. Conversely, the public health lecture was newly created with the most recent information regarding the state of the COVID-19 pandemic, as well as the status of various vaccine clinical trials. The broad extent of topics covered in this program was designed to appeal to a wide audience and to foster further exploration into these diverse areas of study.

Paired-Samples t-Test
We were pleased to see that students learned the most (2.1%) levels of knowledge regarding immunology before the program; then after the program, 56.3% of students reported a moderate level of knowledge and 40.6% reported a high level of knowledge. Conversely, students reported the smallest gains in knowledge of pursuing a STEM career, with 37.5% of students reporting a moderate level of knowledge and 29.2% reporting a high level of knowledge before the program. After completing the VRTP, nearly all reported moderate (42.7%) or high (55.2%) levels of knowledge regarding their future career pathways in the STEM fields. This difference in knowledge gains between the items yielding the largest and smallest increases is likely due to students knowing more initially about STEM careers than immunology or immunotherapy ( Figure 2). This is reflected in the higher pre-score for knowledge of college and career preparation (2.68) compared to that for lecture and lab content (2.05) (Table 3).

DISCUSSION
The shift to remote work because of the COVID-19 pandemic has compelled educators to rapidly pivot and adapt to meet the immediate needs of their students in an online setting. To that end, the Science Education Department (SED) planned and implemented the Virtual Research Training Program (VRTP) for high school students in summer 2020. Ear-    about immunology and immunotherapy, as well as the related laboratory demonstration on Ficoll blood separation (Table 4). Previous in-person classes featuring the Ficoll blood separation experiment were also extremely well-received by high school students and teachers (unpublished observations). Seattle Children's has a Good Manufacturing Practice facility -called the Cure Factory -that develops therapeutic cell products for cancer treatments (KING 5, 2019). It is rare for traditional high schools to have the facilities or expertise needed to demonstrate the cutting-edge patient-care applications such as those being developed at the Cure Factory. This reinforces the need for informal education programs like the VRTP in bridging the gap between theory and application. However, it should be noted that evaluating students' proficiency in actually performing laboratory experiments was not possible for the VRTP. As such, the gains reported in knowledge of lab content should not be cross-compared with previous unpublished data from our in-person programs regarding the knowledge of performing laboratory procedures. It was also necessary to ascertain student engagement with the VRTP, especially since we could not see the audience. The accelerated timeline of the VRTP development precluded the use of a dedicated course management system to monitor student interactions (Martín-Blas et al., 2009;Wei et al., 2015). Instead, question asking was used as a proxy for quantifying student engagement (Caton et al., 2020;Witte et al., 2008). First, the ability of students to ask questions in real-time was seen early on as a critical feature of the VRTP; we wanted students to be able to talk with scientists, even if only virtually. Second, the Q&A function built into WebEx allowed for ease of tracking the students' questions. Finally, we regularly encouraged students to actively participate. The audio and video from students were disabled during the live stream, allowing for staff moderation of questions, promoting a safe environment for questions, and protecting students' privacy. As a result, a vast majority of the students attending the live sessions asked at least one question. This suggested that even when not actively monitored, these students were eager to participate and learn about biomedical research. It should be noted that while traditional student engagement could be approximated through direct interactions with students in a classroom or lab, these interactions would be difficult to quantify and compare across cohorts. In that respect, the virtual nature of this program has yielded a novel metric for evaluating future educational programming.
The overall response to the use of the Cisco WebEx Events video conferencing suite was varied. Our staff's previous training in and familiarity with WebEx allowed for a simple and quick transition from small group meetings to large group presentations. However, our students were more familiar with other platforms, such as Zoom Video Communications, which led to many issues with students unable to connect to the live stream. In fact, the issues with WebEx comprised a significant proportion of the feedback from students and subsequent tech support by our staff.
Also, there were many improvements to be made in the quality of the video production, such as using camera, audio, and lighting equipment suitable for video streaming. For summer 2020, we utilized laptop cameras, cell phone cameras, wired and wireless headsets with microphones, and small inexpensive LED lights, which ultimately proved insufficient for showcasing our dynamic lab environment. In the future, we will use higher-resolution cameras mounted on tripods, dedicated recording microphones, and studio-quality lighting equipment. This will lead to a better experience for students to observe lectures and lab demonstrations.

ADVANTAGES AND LIMITATIONS
There were numerous advantages to virtual education, including increased flexibility to accommodate scheduling conflicts and the removal of travel barriers, especially for students with work or childcare commitments. Although the VRTP originally enrolled students with local Washington addresses, future virtual cohorts will include out-of-state and international students. We were able to increase the number of students served to over a hundred and include both under-represented and well-represented students due to our essentially unlimited capacity.
Nevertheless, there were limitations that need to be acknowledged. It was not possible to fully replicate working in an actual laboratory nor physically perform the experiments at home. Also, our students could not tour the many facilities at our Research Institute. We would have wished to showcase our aquatics facility and have students observe zebrafish embryonic development, and for students to visit our therapeutic cell production core to learn about developing immunotherapy cell products for cancer treatments. To compensate, we are continuing to exhibit lab experiments and demonstrations that exemplify the fundamentals of biological research and provide additional resources for students to learn about more advanced topics.
Furthermore, there were many disruptions to the work schedules of other scientists that we had collaborated with in the past, and thus, we could not feature any guest speakers from other departments in the Research Institute. Fortunately, these issues are not anticipated to persist in the future and plans are in motion to feature these scientists -lab technicians, graduate students, and post-doctoral researchers -and showcase their work in fields such as bioinformatics, immunology, and neurology, to name a few. These interactions will allow our students to better understand the many career pathways in biomedical research, as well as highlight our diverse workforce and the innovative projects currently underway at Seattle Children's.
Other notable limitations of virtual programming are the accessibility concerns for students with limited vision or those who are hard of hearing. To address these concerns, future virtual programming will utilize multiple channels for delivery of content. This may include verbally repeating or describing visual content, using alternative text for images, and generating captions for videos and recordings (University of Washington, 2019). To further improve accessibility, we will also explore other video conferencing options, including Microsoft Teams Live Events. This platform will allow for native integration with learning management systems (Canva, Blackboard, etc.), automatic transcripts and translations, and additional tools for audience interaction, including polling (Henderson et al., 2020). Such features will allow for greater engagement and accessibility for students. However, students also require reliable or even highspeed internet access to fully experience the virtual program. At present, students without a good internet connection may only have the option to watch the recordings when able.
The VRTP also did not have the capacity to allow for students to work collaboratively with SED staff or with each other. Typically, our in-person program is a great way to get to know students and to be able to write meaningful and personalized letters of recommendation. While we were able to assess knowledge gains with the post-session survey, we could not evaluate comprehension in real-time or accommodate personal check-ins. These limitations may be inherent to online education, but we hope to minimize their impact by offering more optional office hours, optional breakout rooms to allow for interactions amongst students, or additional resources for career exploration, such as career fairs or workshops, in the future.

CONCLUSIONS
Although the development of the VRTP was accelerated out of necessity due to the COVID-19 pandemic, the response from the students was overwhelmingly positive. In addressing the many unique challenges that arose, we have developed and validated novel methods for reaching audiences beyond the traditional classroom. The VRTP has allowed us to connect with over a hundred high school students, when our previous in-person program capacity was limited by physical space to a few dozen students at a time. For the future, our virtual and in-person summer programs for high school students will be conducted in parallel, providing additional online options for students who may be interested in biomedical research but who are unable to commit to several weeks of in-person programming. The VRTP also has the potential to reach students outside the commute radius of our Research Institute. And finally, the best practices developed will allow our other educational programs to likewise navigate the transition to remote education and address the needs of the communities in which we serve.
Even during the production of the 2020 VRTP, plans were already underway for summer 2021. These plans include adding even more laboratory demonstrations, including PCR and other common molecular biology procedures and their applications in healthcare. We also plan to feature many more scientists and healthcare professionals at Seattle Children's to highlight the diversity of our workforce and to address the challenges that many of our students face as they start their careers. These guest speakers will also showcase the plethora of career options available to our students and provide concrete next steps to help achieve those goals.
The VRTP was an opportunity to gather useful data and improve our programming over the long term, as well as continue to provide educational resources for students. The Science Education Department was successful in converting an in-person laboratory research program into a remote education experience. We were able to continue to inspire, empower, and enable the next generation of biomedical researchers and healthcare workers in pursuing their future careers. With the lessons learned, future in-person and remote programming will enable us to serve even more students across Washington state and beyond.

Author Contributions
The manuscript was written through contributions of all authors. All authors have given approval to the final version of the manuscript.