Conversion of Summer STEM Program from In-Person to Virtual Learning Offers Unexpected Positives and Pitfalls

,


INTRODUCTION
History of WRAIR GEMS. STEM education is essential for the stability of the modern economy, as science and technology are major components for job growth in future occupations. However, racial minorities, low-income, and first-generation students have been historically underrepresented in higher education science programs which are often a prerequisite for high paying jobs in STEM-related fields (Curiale, 2010). Recognizing an opportunity to improve upon STEM education efforts, the Walter Reed Army Institute of Research (WRAIR), with the support of various funding mechanisms, created the Gains in the Education of Mathematics and Science (GEMS) program for rising 7th-12th grade students from Washington, DC and the surrounding suburban areas (Brown et al., 2020), with a focus on equality of access to students whom the Army Education Outreach Program, AEOP, has identified as "twice underrepresented" or "U2" (USAEOP, 2020). U2 students are those who self-identify with two or more of the following: racial or ethnic minorities historically underrepresented in STEM, students who qualify for free or reduced-price meals, female students in certain STEM fields, students who receive special education services, students with disabilities, aspiring first-generation college students, students in rural, frontier, or other federally targeted outreach schools, and students for whom English is not their primary language at home (USAEOP, 2020). GEMS gives students the opportunity to explore science through hands-on activities, simulations, computer applications, and laboratory experiments. Students gain practice in using the scientific method, designing experiments, recording data and forming conclusions based on their data. Moreover, the most important element of GEMS is the use of close-in-age, undergraduate or recent post-baccalaureate student mentors, termed near-peer mentors (NPMs). NPMs are selected for their interest in teaching and research in STEM. They are largely recruited from local counties, and thus often have shared identities with the GEMS participants. NPMs facilitate weekly laboratory investigations, discuss the importance of STEM research, and serve as STEM role models.
Hypotheses and Expected Outcomes. While the WRAIR GEMS program provides science enrichment activities for middle school and high school students, the program is also targeted at removing barriers to access into STEM programming and providing STEM internships for both high school and college students. WRAIR GEMS prides itself on the ability to strongly recruit U2 students into the program and foster a continuation of returning students for multiple years of participation. The conversion of the WRAIR GEMS program to eGEMS not only had to provide science enrichment activities to students, but also maintain the standards and outcomes of the original in-person program. These outcomes include the promotion of positive science attitudes among GEMS participants, high recruitment of U2 students, and professional development for our NPM participants (Tenenbaum et al., 2017;Anderson et al., 2019;Brown et al., 2020). However, the switch from an in-person experience to an online format presents many challenges towards student participation and engagement (Zhang, 2005). Access to an online program requires the use of high-speed internet technology which is not easily accessible to all students. In addition, the lack of in-person activities could also be a deterrent for students that are seeking more of an immersive experience. With these considerations in mind, the WRAIR GEMS staff developed a carefully crafted, learner-centered, interactive online experience (Zhang, 2005), paired with thoughtfully-designed, take-home experiment supply kits (Dickerson et al., 2014), or supply lists for long-distance students, to accomplish similar outcomes as the in-person program. Our overarching hypothesis was that eGEMS would maintain the same level of U2 student participation while promoting positive student attitudes towards STEM education and STEM careers for GEMS participants. We did, however, expect some differences due to the changes in program format across years, such as the amount of group work students would experience, as well as the number of scientific tools they would use. Finally, we also hypothesized that NPMs would have similar gains in skills for STEM education and careers, including transferable skills such as networking, across years.

NPM Training.
Whether it is conducted in-person or online, GEMS is facilitated by a team of NPMs. Once selected, NPMs undergo a two-week training period covering topics including pedagogy, professionalism, ethics, career development and networking, and lesson planning. NPMs are also required to conduct a research experiment or lesson evaluation and participate in a final poster presentation at the end of the summer at a STEM symposium hosted at the WRAIR. All NPMs are closely supervised by the year-round fulltime, doctoral-level science education staff at the WRAIR. Near-peer mentors serve as interns for the duration of the summer program and receive a stipend based on their educational and experience level.
In-Person Summer GEMS. The in-person program is a hands-on science education experience offered to rising 7th-12th grade students with each weeklong session focusing on a different age group (beginning: 7th and 8th grades, intermediate: 9th and 10th grades, and advanced: 11th and 12th grades) on a rotating basis. Each week, students participate in either a biomedical-or an engineering-focused program, with approximately 40 students in each program per week. The program runs throughout the summer for nine to ten weeks. Student applicants are selected based on interest in STEM and geographic location, with an emphasis on welcoming students from populations underrepresented in STEM. There are no experience or merit requirements for entry into the program. Participants receive a small stipend to facilitate their participation by offsetting the cost of transportation and lunch and to provide a sense of achievement at the conclusion of their time in the program, as well as a certificate of accomplishment.
Each day, NPMs lead their group of participating students in hands-on, inquiry-based STEM modules that have been developed by NPMs themselves in collaboration with WRAIR scientists. The program covers a broad range of science and engineering topics each year (see Table 1 for examples of topics that have been covered) and all are adapted to be age appropriate for the beginning, intermediate, and advanced groups. All lessons are designed to be hands-on, allowing students to be actively engaged in their learning and many lessons focus on current events to encourage the connection between science and the real world. Advanced students in grades 11 and 12 have the opportunity to choose their own topics of interest to research further as part of a group and design their own experiments. See Table 2 for an example of a daily schedule.
Virtual eGEMS. The eGEMS format was created to maximize student engagement while combating the barriers of online learning, such as the isolating nature and "Zoom fatigue". The program consisted of a five-day format, with four days used for program activities and one day for program prep and organization. Similar to in-person GEMS, eGEMS was divided into three separate groups to offer grade appropriate activities: beginning (rising 7th and 8th graders), intermediate (rising 9th and 10 graders), and advanced (11th and 12 graders). However, the biomedical and engineering foci were combined, so student participants would get programming in both disciplines. Each week, eGEMS welcomed an average of 65 students to align with the typical 1:6 NPM to student ratio. The total student number was equally divided among three separate Google Classrooms, two of which had biomedical foci and the other had an engineering focus. Written materials were posted to the Google Classrooms, which were designed to complement the real-time  programming. The students then participated in eGEMS programming via Zoom on Monday through Thursday with the days split into two approximately two-hour morning and afternoon sessions. For three of the four days, the morning sessions offered long-form NPM-led investigations and activities and the afternoon activities were a mix of shorter activities (about ~30 minutes each) that the student participants would rotate through (summarized in Table 1, shown in detail in Table S1). Each day, the NPMs would switch classrooms allowing participants to receive a different type of programming facilitated by a new group of NPMs. For the 4th day of the week, the morning session offered a whole group, team-based interactive activity and the afternoon session consisted of survey completion and NPM-student bonding activities (Table S2). The 5th day of the week was reserved for NPMs to plan and prepare for the next week of activities. They used this time to discuss the successes and challenges of the week alongside changes to implement in the future. Furthermore, NPMs were required to attend a weekly professional development series where they received career advice from various STEM professionals.

Near-Peer Mentors. As reported in
During the summer of 2020, 17 near-peer mentors participated in the virtual GEMS program (male=5; female=12). NPMs ranged in age from 18 to 28 (m = 20 years, 7 months). NPMs reported their racial/ethnic background to be Asian (23.5%), Black/African American (17.6%), Latinx (5.9%), White (35.4%), and more than one race/ethnicity (17.6%). One hundred percent of NPMs chose not to report if they had received a Pell Grant. NPMs attended public (82.4%) and private universities/colleges (17.6%). Approximately a quarter (23.5%) of NPMs in 2020 would be the first in their family to graduate from college, while 76.5% percent reported that their parents had graduated from college. Gender trends of NPMs are in line with the higher proportion of female students in biology and life science undergraduate programs (Eddy et al., 2014).

Figure 1. Student and NPM demographics between years. (A)
Participant demographic data is displayed across 6 categories (race and ethnicity, first-generation college student status, gender, FARMS status, school type, primary language used at home) as percent of participants self-identifying in each category in 2019 (n = 517) and 2020 (n = 679). (B) NPM demographic data is displayed across 5 categories (race and ethnicity, first-generation college student status, gender, Pell Grant status, and school type) as percent of NPMs self-identifying in each category in 2019 (n = 15) and 2020 ( n = 17). (C) Demographic data from Prince George's County Public Schools (PGCPS) state report cards for four categories (race and ethnicity, gender, FARMS status, school type, English language learner status -ELL) in school years 2018-19 (n = 132,322) and 2019-20 (n = 132,667). (D) Demographic data from Montgomery County Public Schools (MCPS) state report cards for four categories (race and ethnicity, gender, FARMS status, school type, English language learner status -ELL) in school years 2018-19 (n = 161,546) and 2019-20 (n = 162680).

Data Analysis.
Qualitative Data Analysis. Qualitative data from both the AEOP Youth Survey and the Near-Peer Mentor survey were analyzed using open coding based on Grounded Theory (Glasser andStrauss, 1967, 1999). Open coding involves naming and defining concepts that emerge from a detailed line by line review of raw data (Strauss and Corbin, 1998). This coding method involves a deductive process of identifying and naming themes that emerge from the data (Williams and Moser, 2019).

Quantitative Data Analysis.
Quantitative analyses were performed solely on student data as the NPM exit survey only included quantitative data in 2020 and not in 2019, preventing us from being able to compare between years for the NPM cohort. For the student data and based on our hypotheses, we selected questions from the 2019 and 2020 AEOP student surveys that gave information about: (a) program fidelity; (b) student attitudes toward STEM education; and (c) student attitudes toward STEM careers. Importantly, the only potential U2 identities that students self-report in AE-OP's post-participation survey include race/ethnicity, gender identity (no option for non-binary existed at the time the survey was administered), whether the student's parents finished college, whether the student is a FARMS recipient, and whether or not English is the primary language spoken at home. Therefore, for our analyses, we categorized students as U2 based only upon whether or not they identified with underrepresented groups in two or more of these five categories. To display the frequencies of each survey question response between our determined groups, violin plots were used to effectively visualize the distribution of data wherein a greater curve width represents a higher frequency of response selection. Plots show varied tail lengths due to the predictive nature of the performed analyses.
Initial comparisons of student responses to the AEOP survey by year were performed using Mann Whitney U-tests. The Mann Whitney U-test is similar to a t-test, but is non-parametric and therefore more appropriate for the ordinal response data the study comprised. We additionally used generalized linear models in R to facilitate binomial regressions for analyses in which the response data was either "yes" or "no" and multinomial ordinal models to facilitate analyses when response data was Likert-scale-like, or ordered and multinomial in nature. Unlike the Mann Whitney U-tests, the regression models allowed us to analyze outcomes while modeling more than one predictor at a time, and to test whether there were any interactions between predictor variables. Specifically, we modeled the response data as a function of year (2019 or 2020), or as a function of year and U2 status (U2 or not U2). Reported regression coefficients therefore indicate any occurrence of a positive or negative correlation between modeled response data and these func-

Measures.
AEOP Youth Questionnaire. This evaluation tool (USAE-OP, 2019b) was developed by the United States Army Educational Outreach Program to be completed by student participants in the summer GEMS program (USAEOP, 2019a; USAEOP 2020). The AEOP reported that the questionnaire was aligned with the Army's strategic plan and AEOP's priorities of 1) A STEM-literate citizenry, 2) STEM-savvy educators, and 3) A sustainable infrastructure (USAEOP, 2019b). This questionnaire included both multiple choice and free-response questions and collected information about participants' experiences with and perceptions of program resources, structures, and activities; potential benefits to participants; and strengths and areas of improvement for programs (USAEOP, 2019b). Survey questions varied slightly from 2019 to 2020, but overall topics remain consistent across years.

Near-Peer Mentor Survey.
This evaluation tool was developed by WRAIR Science Education staff to learn about how the NPMs view their internship experience, and their gains from the experience, as well as how we can improve the experience in the future. The 2019 survey was based on a validated survey by Nelson et al. (2017). The 2020 survey was modified slightly to incorporate the virtual aspects of the eGEMS program. The 2020 survey was also shortened to encourage a higher response rate among NPMs. Both of the surveys attempted to gauge how the NPMs' value the GEMS experience for their personal and professional development.
Procedures. The AEOP Youth Survey (USAEOP, 2019b) was administered to student participants on Thursday afternoons during their week of participation. In 2019, during the in-person summer GEMS program, surveys were completed using pencil and paper format. In 2020, during the virtual eGEMS summer program, surveys were completed online. In both years, the surveys took approximately 45 minutes to complete. Both surveys were administered to student participants by NPMs. Using the NPMs to administer the survey helps to motivate the students to complete the survey, as they have developed a rapport with the students during the week. NPMs explained that the survey is optional but important for the success of the program. NPMs were also able to make the survey experience seem more a part of the enrichment experience instead of an additional side task.
The Near-Peer Mentor survey was administered during the last week of participation each year via Google Forms. Researchers counted responses for up to two weeks following the end of the NPM internship. The survey took approximately 20 minutes to complete in 2019 and 15 minutes to complete in 2020. Completion of the survey was optional and no additional incentives were provided.
tions. Participants who did not disclose the information necessary to determine U2 status were excluded from the multinomial regression analyses. All of our statistical analyses were conducted in R (R Core Team, 2021). Both the Mann Whitney U-tests ("wilcox.test") and the generalized linear models ("glm") are part of the basic "stats" package included in R. The multinomial regression ("polr") was conducted in the "MASS" package (Venables and Ripley, 2002).
As the Likert-like scales were changed for some questions across 2019 and 2020 in the AEOP survey, we decided to re-scale the 2020 data (Vossen et al. 2019; Table 3). In some cases, there were different upper bounds of the scales (Table 3), but the highest and second highest scores in 2020 and 2019, respectively, were both "agree." We debated over whether we could match these with one another since they were defined with the same word. However, because the implication with bounded scales is that they go from least to most, with equidistant points for each scale number in between, we decided that statistically it made more sense to not equate "agree" between years, since it was an upper bound in one year and not in the other. Re-scaling, which is further explained in Vossen et al. (2019), was done using the following formula: where S is the score, LB and UB represent the lower and upper bound, respectively, and σ is a standardized score (a proportion between 0 and 1) corresponding to S. In order to maintain integrity of the data across years, all response options were kept in the re-scaling and analyses.
We also divided questions that contained two responses that could be considered "zero" scores. For example, responses to the question, "How satisfied were you with the following GEMS features: Teaching or mentoring provided during GEMS activities?" included both "did not experience," and "not at all" as options. These responses cannot be scored as "zero", and the other as a "one" since these answers are not ordered. The responses also cannot both be scored as "zero" since they do not carry the same meaning.
Therefore, we decided to divide this question into two parts: (1) whether or not students experienced any degree of mentorship, and (2) if so, whether or not they were satisfied with the mentorship they received (Table 4).

Student and NPM Demographics Between Years 2019
and 2020. As reported above and shown below in Figure  1A In 2019 and 2020 GEMS respectively, 0.5% and 1.5% of students reported primarily speaking a language other than English at home, while PGCPS and MCPS had higher percentages of ELL students enrolled (PGCPS 2019 -19.5%, PGCPS 2020 -20.7%, MCPS 2019 -17.7%, MCPS 2020 -18.1%). The AEOP survey has a 'choose not to report' option, though, when the school district data does not have this option, so this may confound the comparison. School district report cards did not contain data for comparing parental level of educational attainment; however, the percentages of GEMS participants whose parents did not graduate from college (2019 -7.5%, 2020 -7.1%) were low compared to the national rate 24% of college students whose parents did not attend postsecondary schooling (Radwin et al., 2018). The NPMs were not asked about primary language spoken at home; however, they were asked about Pell grant

Program Fidelity Across Years and by U2 Status.
Between 2019 and 2020, we did not detect statistically significant differences in median responses for several key "program fidelity" areas, such as whether or not students experienced mentorship, how satisfied students were with the mentorship they received (Figure 2A), the quantity of STEM careers students learned about over the course of the program (Figure 2B), and how often students examined data to form conclusions ( Figure 2C). There were, however, a few statistically significant differences in median responses for other measures of program fidelity between 2019 and 2020. Students reported an increase in the frequency of planning their own research based on their own ideas from 2019 to 2020 (p = 0.037, W=45630, 2019 median = 2.0, 2020 median = 2.67).
However, this potential difference was not detected via multinomial regression when taking U2 status into account. Students also reported declines in frequency of group work (p < 0.001, W = 65059, 2019 median = 4.0, 2020 median = 2.67) and use of scientific or laboratory tools from 2019 to 2020 (p < 0.001, W = 62408, 2019 median = 4.0, 2020 median = 2.67). None of these student responses to "program fideli- ty" questions differed by U2 status; only the variable "year" drove these differences ( Figure 2C). Statistical details associated with Figure 2 are reported in Table S3.

Student Attitudes Toward STEM Education Across Years and by U2 Status.
Students' self-reported gains in attitudes toward STEM education remained consistent from 2019 to 2020 in several areas, including: (a) whether their NPMs helped them learn about STEM in everyday life (b) wheth-er their NPMs helped them to understand how they can use STEM in their communities, (c) whether they are likely to mentor or teach others in STEM, (d) whether they would help with a community project related to STEM, (e) whether they would take an extra STEM class, and (f) whether they feel more confident in their STEM knowledge, skills and abilities (all NS; Figure 3A,C). Interestingly, students reported higher scores in 2020 than 2019, despite the switch to online formatting in 2020 in the following areas: (a) whether or not they were aware of the GEMS program (β = 0.67, p = 0.02, z = 2.24 for the effect of year in the multinomial model taking U2 status into account even though U2 status was not significant in the model; NS Mann Whitney U model, W = 37605); and (b) how interested they were in participating in GEMS in the future (p = 0.01, W = 23200, 2019 median = 2.0, 2020 median = 2.0). However, students in 2020 reported being less likely than the 2019 cohort to participate in a future STEM camp, club, or competition (p = 0.04, W = 34520; 2019 median = 3.0, 2020 median = 2.67, Figure  3B,C). Any statistically significant differences were driven by year, rather than U2 status ( Figure 3A-C). Statistical details associated with Figure 3 are reported in Table S4.

Student Attitudes Toward STEM Careers Across Years and by U2 Status.
Both 2019 and 2020 programming elicited similar changes in student attitudes towards STEM careers. In both years, students reported similar increases in interest in pursuing STEM degrees and STEM careers after participating in GEMS, and these shifts did not differ by U2 status or by year in multinomial regression ( Figure 4A). Additionally, in a new question on the 2020 survey, students indicated interest in working with STEM mentors -with the median response being "strongly agree" (level 3) for all student groups ( Figure 4B). Students were also asked about future interest in pursuing AEOP teaching and research internships. There was an increase in awareness of some of the programs in 2020, specifically the Science and Engineering Apprenticeship Program (SEAP; β = 0.46, p = 0.029, z =2.181) and the NPM program (β = 0.82, p = 0.00027, z=3.379) (TableS5). These increases were not detected via a Mann Whitney U-test; however, they were detected via a binomial general-ized linear model. There was no shift in interest across years or by U2 status for the research internship. However, there was a decrease in interest for the teaching internship (the NPM internship) in 2020 ( Figure 4C). This difference was detected via a Mann Whitney U-test (p=0.046, W=19689, 2019 median = 2.0, 2020 median = 1.0); however, it was not detected via the multinomial regression accounting for variation from both year and U2 status. Statistical details associated with Figure 4 are reported in Table S5.

Student Qualitative Data. Students gave written responses
to two open-ended questions in 2019 and 2020 that asked them to reflect on any perceived benefits from participation in GEMS or suggest any improvements for their respective iteration of GEMS. The benefits question asked, "What are the three most important ways that GEMS has helped you?" while the improvement question asked, "What are the three ways that we could make GEMS better?" There were 360 respondents in 2019 and 338 respondents in 2020. Responses were reviewed by a team of researchers under each question category and several themes emerged through the process of open coding. While there was some overlap, the themes emergent from responses addressing perceived benefits differed from those emergent from responses including suggestions for improvement. Full definitions for each sequence of themes are provided in the Student Survey Qualitative Coding Manual (Table S6 for perceived benefits, and Table S7 for suggested improvements). Between years, themes emergent from student reports of perceived benefits (Table 5) were largely similar, corroborating the reported maintenance of program fidelity between 2019 and 2020 (Figure 2A, B). Main themes that revealed  the similar nature of the overall student perceived benefits were: real world applicability and newly gained perspectives, gained knowledge and/or interest in STEM education, future plans in education or career seeking, personal growth and/or gain, mentorship, and gained knowledge and/or interest in GEMS affiliates. Possible exceptions to this were observed as differences between years through the themes of meeting people, program enjoyment, and gained knowledge and/or interest in STEM careers. Specifically, a lower occurrence in 2020 of statements reflecting the theme of meeting people, including peers, mentors, and professionals (Table 5) was observed, while responses like "[h]ad STEM opportunities that weren't available to me anywhere else" reflected a possible 2020 increase in program enjoyment including feelings of engagement and exposure to novel experiences. Another possible increase in gained knowledge and/or interest in STEM careers was also indicated by 2020 student responses in their own words (Table 5). Together, these corroborate the observed increased interest in future participation of GEMS or an external STEM club, camp, or competition ( Figure 3) as well as increased interest in pursuing STEM degrees and careers after participating in GEMS (Figure 4). There are notable discrepancies between years among GEMS improvements suggested by students pertaining to the themes professional engagement and program logistics. These may have both emerged at lesser degrees from 2020 suggestions (Table 6). Nevertheless, all other themes that emerged from students' suggested improvements appeared similar between years including overall satisfaction; wanting more guidance or information to guide future planning in education or career seeking; and indications of gaps in NPM training, curricular content, mentorship, and knowledge of GEMS affiliates (Table 6). One theme, in-person preference, emerged only from 2020 suggestions at a rate of 15% (Table  6). While 15% of respondents noted an in-person preference for the GEMS program in their suggestions (Table 6), most GEMS students who answered closed-ended survey questions regarding the virtual aspects of eGEMS had a favorable view of being virtual. 2020 students were asked "Overall, how would you rate the virtual session?" From a total of 338 respondents, 99.4% chose positive ratings of 'good,' 'very good,' or 'excellent' leaving 0.6% that chose 'not so good.'

Differences/Similarities in NPM Qualitative Data from 2019 to 2020.
NPMs responded to open-ended questions in 2019 and 2020 that asked them to reflect on how the skills they learned in GEMS would benefit them in the future. There were 8 respondents in 2019 and 12 respondents in 2020. Of note, the 2019 prompt focused on how teaching, mentoring, and communicating would benefit NPMs in the future, whereas 2020's prompt was broader. Nevertheless, the questions, while posed differently, still elicited similar responses. Emergent themes are outlined in Table 7. The questions were as follows: 2019: Reflecting on your teaching experiences this past summer, how have you matured in terms of being able to teach, mentor and communicate science/engineering to others? How do you think these skills will benefit your professional/ career aspirations? 2020: Do you feel that your experience as a near-peer mentor has contributed to your professional/educational development? If so, how? If not, please explain? Data analysis involving a team of researchers using open coding strategies revealed several common themes and subthemes elucidating the experience of near-peer mentors. Main themes that revealed the nature of NPM maturation during their summer experience were: (gained) relationships, broadened perspectives, professional development, "Offer more education on careers in the army and DoD" In-person preference N/A 15% "This was out of anyones (sic) control but GEMS will be better in person" teaching skills and experience, and self-efficacy and confidence. Several subthemes also emerged, which are detailed in Table 7. Between years 2019 and 2020, similar percentages of NPMs reported most of the subcodes within the overarching theme of maturation. There was some discrepancy between the percentage of NPMs reporting themes that fell under the self-efficacy and communication subcode in 2019 (88%) versus 2020 (25%) ( Table 7). Additionally, there were differences between the percentage of NPMs reporting on professional and transferable skills that they had gained in 2019 (88%) versus 2020 (33%) ( Table 7). There were some themes that did not appear in one year or the other. For example, gains in STEM knowledge and practice only appeared in 2020. Themes that appeared in 2019 but not in 2020 in-

DISCUSSION
U2 Participation. Student demographics remained similar across in-person and eGEMS ( Figure 1A). In alignment with previous years (Brown et al., 2020), our demographic data revealed similar racial/ethnic make-up of the GEMS program versus surrounding counties. However, it also revealed underrepresentation among FARMS recipients and English language learners compared to local county public schools. Although the county public school data did not include the percentages for potential first-generation college students ( Figure 1A), the GEMS percentages of potential first-generation college students (approximately 7% each year) were far lower than the national rates of 24% of college students whose parents have no postsecondary education and 56% of students whose parents do not have a bachelor's degree ( Figure 1B) (Radwin et al., 2018). This comparison to national data on potential first-generation college students may be confounded by the fact that Maryland has a high percentage of residents 25 years and older with bachelor's degrees or higher compared to the national percentage (Maryland -40.9%, United States -33.1%, United States Census Bureau, 2019). We will need to assess the reasons for barriers to FARMS recipients, English language learners, and potential first-generation college students to create a plan for improving GEMS' accessibility. Additionally, the demographic question about gender identity on the AEOP student survey does not include an option for students who identify as non-binary. Alternative response options would aid in fostering the inclusion of students who self-identify as non-binary. This is also important from the lens of program evaluation, as we aim to accurately assess our accessibility to students of all genders.
Program Fidelity. Our measures of program fidelity showed several areas in which GEMS remained a stable resource for students between the 2019 in-person version of GEMs, and the 2020 online version of GEMS. Students were similarly satisfied with the mentorship they received from NPMs ( Figure 2A); they learned about similar numbers of STEM careers ( Figure 2B), and they reported similar levels of engagement in key scientific processes, such as examining data to form conclusions ( Figure 2C). This was recapitulated by students in their own words when asked to reflect on any benefits they perceived with participation in GEMS ( Table  5). Responses regularly included themes indicating feelings of enjoyment and engagement with GEMS and NPMs, as well as a strong emphasis on interest and knowledge surrounding STEM education. Some areas where GEMS can improve how we serve students online include finding a way (a) to allow students to use more scientific equipment at home and (b) to give students more opportunities to engage in group work. The former might be done by applying methods from the field of frugal science. For example, one of our 2020 NPMs developed a lesson where students could create their own centrifuges at home using simple materials such as paper and string. These simple centrifuges are used around the world in applications such as testing for malaria, and therefore this highly accessible activity gives students access to actual scientific equipment. We can infuse similar lessons into future eGEMS programs. Furthermore, access to proper computer technology and high-speed internet is an ongoing challenge for all virtual learning (Roth, 2020). We have partnered with organizations, such as those in collaboration with the AEOP, and schools, such as Montgomery College, to build a collection of loaner laptops and internet hot spots. However, these items are limited and cannot fully address the great technology disparity facing students in our local population.
Pertaining to the need to provide students with more group work engagement opportunities while participating online, a few statistically significant differences were observed between 2019 and 2020. One encouraging result was that students in 2020 reported a higher frequency of planning their own research based on their own ideas ( Figure  2C). Because a lot of the work in 2020 eGEMS was solitary in nature due to physical limitations, this result seems reasonable and could have been a positive artifact of these circumstances. To encourage this benefit for the in-person sessions as well, it might be helpful to provide students with more opportunities for individual projects during in-person GEMS. On the other hand, students in 2020 less frequently reported meeting peers, mentors, and professionals when asked about benefits of GEMS, another likely reflection of the solitary nature of eGEMS (Table 5). While it is not possible to create the same experiences in person and online, we can address this in future iterations of virtual GEMs by creating more opportunities for online teamwork. We did not detect differences in program fidelity measures by U2 status, and therefore concluded that U2 and non-U2 students had similar outcomes across years (Figure 2A-C). This is encouraging, as U2 populations have been differentially affected by the COVID-19 pandemic in general (e.g., Hooper et al., 2020;Lee et al., 2021). We had hoped that the pandemic's effects would not carry over into U2 students' GEMS outcomes as well.
Student Attitudes toward STEM Education. Our measures of student attitudes toward STEM education remained consistent from 2019 to 2020 in several areas. In their own words, students from both years indicated, to a robust degree, that gaining knowledge and interest in STEM education was a perceived benefit of participation in GEMS; 81% of responses reflected this theme in both years (Table 5). Additionally, students from both years felt similarly confident in their STEM knowledge, skills, and abilities due to participation in GEMS ( Figure 3A). We are optimistic about this result since students still had gains despite having decreased access to scientific tools in eGEMS. Students also reported receiving similar levels of help from their NPMs in learning about STEM in everyday life and understanding how they can use STEM in their communities ( Figure 3C). In fact, similar levels of students from both years indicated real world applicability and newly gained perspectives to be another perceived benefit of participation in GEMS (Table 5). Interestingly, student responses from 2020 reveal that they were more aware of the GEMS program, and more interested in participating in GEMS in the future, than their counterparts in 2019 ( Figure 3B,C). Since students have spent a lot of time online due to the COVID-19 pandemic, it is possible that they had more time to investigate the GEMS website. The increases in interest may also be due to increased accessibility of the online format, in terms of reduced travel time, reduced cost of travel, etc. (though there are decreases in accessibility for some students too, largely due to connectivity issues). Students also reported being less likely to participate in a future STEM camp, club or competition; we are unsure of how to reconcile this with being more likely to participate in future GEMS programs. Student Attitudes toward STEM Careers. The GEMS program maintained similar elevations in STEM career interest across 2019 and 2020, while further increasing awareness of many of AEOP's other internship opportunities in 2020 (Figure A,B). In the open-ended responses, a higher degree of students reported gaining knowledge and interest in STEM careers in 2020 than 2019. In future iterations of the in-person GEMS program, we can work to increase access to the online resources that may have led to increases in program awareness and career knowledge during eGEMS. The only program that saw a decrease in future interest in 2020 was the NPM program, which may reflect the lack of choice between online, hybrid, and in-person teaching in the eGEMS format. Participants may show more interest in future years of the NPM program that offer more flexibility in the format of the internship. Importantly, the shifts in attitudes towards STEM careers did not significantly vary based on U2 status, indicating that both in-person and eGEMS are able to provide similarly positive shifts across student groups. NPM Outcomes. In 2019 and 2020, NPMs reported similarly-themed outcomes in the areas of professional development, teaching skills and experience, self-efficacy and confidence, relationships, and broadened perspectives. These themes all suggest personal and professional maturation as an outcome of this internship. There appeared to be some differences in levels of reporting of self-efficacy and confidence subcodes (88% in 2019 versus 25% in 2020), as well as of professional and transferable skills (88% in 2019 versus 33% in 2020). These differences could be accounted for by the fact that the 2019 and 2020 questions were not posed in exactly the same way, or they could reflect real differences in the internship, potentially due to the in-person versus online formats of the GEMS. Since eGEMS, NPMs spent less time both interacting with students and conducting experiments in our professional teaching laboratory than their 2019 counterparts, it is possible that this could be responsible for some of the differences in both of these themes across years.
Limitations of this Study. We acknowledge that we were limited in our analyses, and their interpretation, by changes that were made to the student and NPM surveys from 2019 to 2020. The changes to the scale on the student survey, in particular, were difficult to resolve, and interpret. For example, the definition of the upper bounds and other scores on several questions were different across years (e.g., a score of 5 could mean "agree" or "strongly agree" depending on the year the survey was taken). Additionally, in 2019 there were neutral options for some questions, while in 2020 there were not, which creates challenges when comparing across years. We ultimately decided to re-scale the student surveys to make the upper and lower bounds match, but with any rescaling, we lose some information. Additionally, as with most educational programs, this program was refined throughout its implementation so there may be minor changes in how students experienced the program over the course of the summer (e.g., due to NPMs gaining experience throughout the summer), but no data that were collected would allow us to assess this. Future Directions. One of GEMS' foremost goals is to increase access to STEM programming. To do so, we must take into account both the increases and decreases in accessibility that come with virtual programming. Going virtual in 2020 allowed for GEMS to expand its reach beyond the local counties, to students from other cities and states. In fact, of 159 student participants who reported their school state in WRAIR's eGEMS post-participation survey, approximately 40% attended schools located outside of Maryland.
Despite the majority of students reporting satisfaction with the eGEMS program overall, many repeat GEMS students reported liking the in-person GEMS program in 2019 better than the online GEMS program in 2020. While this could have been for many reasons, we suspect that at least some of the students were tired of online learning, since most had spent the whole semester online at that point. These differences may have also been due to differences in program formatting or supplies. Several students and NPMs reported issues with missing or inadequate quantities of eGEMS supply kit materials. Toward the beginning of eGEMS we had not developed a refined system for packing the contactless curbside pick-up supply kits. However, our packing process became increasingly well-organized over the course of the summer; therefore, we do not anticipate this being a problem going forward. Another reason that students may have reported preferring in-person GEMS over eGEMS is that many students (over 30%) had technical difficulties, mostly due to unpredictable internet connection. We are working to develop ways to ensure all students have the same level of access to eGEMS. This will include connecting parents with federal programs such as the Federal Communications Commission Lifeline Program which provides support for low-income subscribers to access services including broadband internet (Federal Communications Commission, 2021), state programs such as Maryland's Technology Assistance Program which provides free access to and training about assistive technology devices for residents with disabilities (Maryland Department of Disabilities, n.d.), and local programs such as Montgomery County Public Library's mobile hotspot lending programs (Montgomery County Government, 2021).
With its broader reach, and with a solution for some of the technical difficulties encountered by students, eGEMS could be a promising tool to reach more students whose families cannot afford the time or money it takes to drive their dependents to WRAIR, as well as neurodivergent students, and students with disabilities. Therefore, we will continue to offer and improve upon both the in-person and online formats of the GEMS program in the future.

ASSOCIATED CONTENT
Supplemental material mentioned in this manuscript can be found uploaded to the same webpage as this the manuscript. The originally published version of this article can also be found in this section.

DISCLAIMER
The opinions or assertions contained herein are the private views of the author, and are not to be construed as official, or as reflecting true views of the Department of the Army or the Department of Defense. The investigators have adhered to the policies for protection of human subjects as prescribed in AR 70-25.