Gains in the Education of Mathematics and Science: A Summer Program Designed to Address Systemic Inequities and Barriers to STEM Pathways

Educational inequity is widely prevalent in United States (U.S.) public schools and creates barriers to STEM education for underserved and underrepresented populations, including racial minority, low-income, and first-generation college students. Scientists at the Walter Reed Army Institute of Research (WRAIR) recognized a need in its community, the greater Washington, DC metropolitan area, to improve access to high-quality STEM education, and founded the WRAIR Gains in the Education of Mathematics and Science (GEMS) program for this purpose. In pursuit of this goal, the GEMS program engages students using innovative learning strategies and authentic STEM experiences with the hopes of fostering interest in STEM and motivating more underserved and underrepresented students to persist in STEM education and career pathways. By 1995, WRAIR scientists began investigating how to bring science-enthusiastic but novice high school (HS) students into their laboratories where they could be mentored, along with providing a stipend to mitigate financial barriers; importantly, this process was funded by two Science Education Partnership Awards from 2001-2006. Each year, WRAIR’s GEMS program now guides about 600-700 local middle and HS students to take part in hands-on, inquiry-based STEM laboratory investigations led by undergraduate “near peer mentors” (NPMs), so named due to their close age to participants. GEMS is now sustained at 14 sites nationwide through the U.S. Army Educational Outreach Program, and serve about 4,000 students and near-peer mentors annually. Recent evaluations of the program explore service to underrepresented groups in STEM and growth of the local program. Further evaluation of the nationwide programs reveals positive student feedback, especially regarding working with NPMs and hands-on learning. INTRODUCTION STEM education is deeply impacted by funding inequities that plague our public education system (Bloome et al., 2006; Wagner, 2019). “While some young Americans— most of them white and affluent— are getting a truly worldclass education, those who attend schools in high poverty neighborhoods are getting an education that more closely approximates school in developing nations” (Equity and Excellence Commission Report, 2013; also see Nielsen, 2013; O’Day and Smith, 2016). To address systemic inequities in science education, the Walter Reed Army Institute of Research (WRAIR) aimed to develop a summer experience for underserved and underrepresented groups in STEM. This effort strove to increase students’ interest and persistence in STEM education and careers, and ultimately led to the creation of the Gains in the Education of Mathematics and Science (GEMS) summer program. It has long been established that teaching strategies that foster active learning (e.g., inquiry-based learning, cooperative learning, problem-based learning, etc.) are effective at improving students’ learning (Acar and Tuncdogan, 2019; Chan et al., 2016; Johnson and Johnson, 1986; Rakow, 1986). However, access to inquiry-centered education and well-supported and trained teachers is rare for low-income and minority students, who are disproportionately subjected to didactic, teacher-controlled instruction, a phenomenon termed the “pedagogy of poverty” (Haberman, 1991; Thadani et al., 2010). Furthermore, research has shown that students can lose the desire to pursue a STEM career as early as middle school (Maltese et al., 2014; Sadler et al., 2012; Wyss et al., 2012). GEMS was created in 1995 as targeted enrichment to supplement classroom learning and keep students engaged in STEM. WRAIR scientists prioritized six Gains in the Education of Mathematics and Science Brown and Morris Vol. 3, Issue 2, August 2020 Journal of STEM Outreach 2 key programmatic elements that aid in equalizing STEM experiences for all students. Figure 1 highlights these elements: (1) accessibility for underserved students, (2) nearpeer mentors (NPMs; so named by GEMS founders due to their close age proximity to GEMS participants), (3) novel lessons and learning modules, (4) hands-on laboratory activities, (5) connections to local scientists and engineers, and (6) sustained pathways to STEM programming. Twenty-five years later, GEMS is now a well-established summer STEM enrichment program for middle and high school students that takes place annually. The next section details the history of building GEMS from a small local effort to a nationwide program. Historical Background. While GEMS was developed and piloted at a single Army research facility (WRAIR), GEMS is now nationwide with annual funding and support provided through the US Army Educational Outreach Program (USAEOP). Early funding of WRAIR GEMS was derived from small, indirect infusions from ongoing research in the founding scientists’ laboratories and a pilot grant through the Armyand Navy-funded Science and Engineering Apprentice Program coordinated by George Washington University. These funds covered limited supplies and personnel costs and provided participant stipends. In 1999, the Maryland Superintendent of Schools learned of the program and connected founders to a funding opportunity from the National Institutes of Health (NIH) National Center for Research Resources called the Science Education Partnership Award (SEPA). Seeking and receiving this sponsored research funding allowed for substantial expansion and resulted in rigorous evaluation of the GEMS program. Under the initial SEPA, from 2001 to 2003, every aspect of GEMS was expanded and improved, allowing greater outreach and participation for both mentors and mentees, better NPM training and more diverse module development with increased sophistication for a broader range of HS students. The program leads began the process of evaluating GEMS outcomes for participants by developing and collecting participant attitude surveys toward STEM (preand post-participation), conducting near-peer mentor exit interviews and implementing concept-learning assessments after each STEM module and at the end of the program (Bliss et al., 2007; Hammamieh et al., 2005; Yochelson, 2006; Yourick and Jett, unpublished evaluations). Whether at WRAIR or dissemination sites, evaluations of the initial SEPA-supported GEMS program showed its efficacy in STEM teaching. GEMS students’ responses overwhelmingly revealed that they highly valued their NPMs, enjoyed challenging handson activities to learn science, had gained more positive attitudes about science, and learned many science concepts and specific laboratory skills in the GEMS program that were not taught in their school curriculum (Bliss et al., 2007; Hammamieh et al., 2005; Yochelson, 2006; Yourick and Jett, unpublished evaluations). With the awarding of a dissemination SEPA grant from 2004 to 2006, WRAIR expanded previous GEMS connections and founded new partnering sites (see timeline in Figure 2). The results of the pre/post GEMS intern evaluation at each of these sites largely mirrored the findings from WRAIR GEMS and provided assurance that the GEMS formula translated readily among various institutions. The work of the GEMS program and many other long-standing Army STEM outreach endeavors combined to further interconnect Army programs for high school and college students and, ultimately, led to the creation of the new US Army Educational Outreach Program (USAEOP) under the Office of the Assistant Secretary of the Army (Acquisition, Logistics and Technology). The positive evaluation outcomes from years of program development (e.g., student-NPM rapport, gains in students’ STEM skills and science attitudes, etc.) were presented to the Director of Research and Army Laboratories in 2005, which led to the Army-funded program that exists today and the successful inclusion in and funding from the USAEOP in 2007. GEMS has continued to expand to new sites in the years since (Figure 2). GEMS now serves around 4000 students and nearpeer mentors annually with the USAEOP funding all programs. GEMS and other high school and college programs in the USAEOP are managed through cooperative agreements with academic and nonprofit partners such as the National Science Teachers Association, Battelle Memorial Institute, Rochester Institute of Technology and other past and present partners to provide evaluation support, marketing, administration and additional innovation. Here, we report our findings from evaluating the GEMS program at both the WRAIR site in the Greater Washington, Figure 1. Key programmatic elements of the summer GEMS program. The six key programmatic elements of the GEMS program are highlighted to showcase the founding principles of the program. Gains in the Education of Mathematics and Science Brown and Morris Vol. 3, Issue 2, August 2020 Journal of STEM Outreach 3 DC metropolitan area and on a national scale with aggregate data from all of the other USAEOP GEMS programs. This work highlights and evaluates the impact of the founding principles of WRAIR GEMS and explores how the national USAEOP GEMS program influences student change. Despite variations in the GEMS program model across sites, all sites are centered on a shared theory of change. This theory postulates that through providing accessible, high quality, inquiry-centered science enrichment, we can improve students’ attitudes toward science and confidence in their own science skills, and ultimately motivate them to persist in STEM pathways and pursue STEM majors and careers. Presently, GEMS invites new and returning middle and high school students each summer for progressively more challenging STEM learning. This approach creates a step-wise STEM pathway toward enrollment in college-level science courses along with experiential learning through a continuum of programs, within and beyond the USAEOP, all the way to post-doctoral fellowships, for the possibility of STEM career support at all levels. What follows in this manuscript is further explanation of the GEMS program elements and implementation, a detailed history of the program evaluation in its broader capacity, and detailed analysis of the continuity of experiential learning and teaching that begins with the original GEMS program site at WRAIR. METHODS While GEMS sessions and programming only occur in the summer, significant efforts and planning are required year-round. Figure 3 details the annual timeline for implementing the WRAIR GEMS program. The planning phase begins in the early fall, as we evaluate our implementation of the previous summer GEMS program at the USAEOP GEMS review meeting. Once we obtain a finalized and approved budget from USAEOP (December/January), we begin the processes of interviewing (December-February) and finalizing staff hires (March), ordering program supplies (April) and preparing program spaces (May), all before the program starts in June. Other major components of GEMS implementation center around NPM and GEMS participant recruitment, which are described in the key programmatic element sections below. Every summer, WRAIR GEMS offers both biomedical and engineering programs, which run concurrently (8-10 weeks of one-week sessions) and at three levels: Beginning (rising 7th-8th grade students), Intermediate (rising 9th-10th grade students) and Advanced (rising 11th-12th grade students). This approach provides varied STEM programming through which student participants can matriculate for mul-


INTRODUCTION
STEM education is deeply impacted by funding inequities that plague our public education system (Bloome et al., 2006;Wagner, 2019). "While some young Americansmost of them white and affluent-are getting a truly worldclass education, those who attend schools in high poverty neighborhoods are getting an education that more closely approximates school in developing nations" (Equity and Excellence Commission Report, 2013; also see Nielsen, 2013;O'Day and Smith, 2016). To address systemic inequities in science education, the Walter Reed Army Institute of Research (WRAIR) aimed to develop a summer experience for underserved and underrepresented groups in STEM. This effort strove to increase students' interest and persistence in STEM education and careers, and ultimately led to the creation of the Gains in the Education of Mathematics and Science (GEMS) summer program.
It has long been established that teaching strategies that foster active learning (e.g., inquiry-based learning, cooperative learning, problem-based learning, etc.) are effective at improving students' learning (Acar and Tuncdogan, 2019;Chan et al., 2016;Johnson and Johnson, 1986;Rakow, 1986). However, access to inquiry-centered education and well-supported and trained teachers is rare for low-income and minority students, who are disproportionately subjected to didactic, teacher-controlled instruction, a phenomenon termed the "pedagogy of poverty" (Haberman, 1991;Thadani et al., 2010). Furthermore, research has shown that students can lose the desire to pursue a STEM career as early as middle school (Maltese et al., 2014;Sadler et al., 2012;Wyss et al., 2012). GEMS was created in 1995 as targeted enrichment to supplement classroom learning and keep students engaged in STEM. WRAIR scientists prioritized six key programmatic elements that aid in equalizing STEM experiences for all students. Figure 1 highlights these elements: (1) accessibility for underserved students, (2) nearpeer mentors (NPMs; so named by GEMS founders due to their close age proximity to GEMS participants), (3) novel lessons and learning modules, (4) hands-on laboratory activities, (5) connections to local scientists and engineers, and (6) sustained pathways to STEM programming.
Twenty-five years later, GEMS is now a well-established summer STEM enrichment program for middle and high school students that takes place annually. The next section details the history of building GEMS from a small local effort to a nationwide program.
Historical Background. While GEMS was developed and piloted at a single Army research facility (WRAIR), GEMS is now nationwide with annual funding and support provided through the US Army Educational Outreach Program (USAEOP). Early funding of WRAIR GEMS was derived from small, indirect infusions from ongoing research in the founding scientists' laboratories and a pilot grant through the Army-and Navy-funded Science and Engineering Apprentice Program coordinated by George Washington University. These funds covered limited supplies and personnel costs and provided participant stipends. In 1999, the Maryland Superintendent of Schools learned of the program and connected founders to a funding opportunity from the National Institutes of Health (NIH) National Center for Research Resources called the Science Education Partnership Award (SEPA). Seeking and receiving this sponsored research funding allowed for substantial expansion and resulted in rigorous evaluation of the GEMS program.
Under the initial SEPA, from 2001 to 2003, every aspect of GEMS was expanded and improved, allowing greater out-reach and participation for both mentors and mentees, better NPM training and more diverse module development with increased sophistication for a broader range of HS students. The program leads began the process of evaluating GEMS outcomes for participants by developing and collecting participant attitude surveys toward STEM (pre-and post-participation), conducting near-peer mentor exit interviews and implementing concept-learning assessments after each STEM module and at the end of the program (Bliss et al., 2007;Hammamieh et al., 2005;Yochelson, 2006;Yourick and Jett, unpublished evaluations). Whether at WRAIR or dissemination sites, evaluations of the initial SEPA-supported GEMS program showed its efficacy in STEM teaching. GEMS students' responses overwhelmingly revealed that they highly valued their NPMs, enjoyed challenging handson activities to learn science, had gained more positive attitudes about science, and learned many science concepts and specific laboratory skills in the GEMS program that were not taught in their school curriculum (Bliss et al., 2007;Hammamieh et al., 2005;Yochelson, 2006;Yourick and Jett, unpublished evaluations). With the awarding of a dissemination SEPA grant from 2004 to 2006, WRAIR expanded previous GEMS connections and founded new partnering sites (see timeline in Figure 2). The results of the pre/post GEMS intern evaluation at each of these sites largely mirrored the findings from WRAIR GEMS and provided assurance that the GEMS formula translated readily among various institutions.
The work of the GEMS program and many other long-standing Army STEM outreach endeavors combined to further interconnect Army programs for high school and college students and, ultimately, led to the creation of the new US Army Educational Outreach Program (USAEOP) under the Office of the Assistant Secretary of the Army (Acquisition, Logistics and Technology). The positive evaluation outcomes from years of program development (e.g., student-NPM rapport, gains in students' STEM skills and science attitudes, etc.) were presented to the Director of Research and Army Laboratories in 2005, which led to the Army-funded program that exists today and the successful inclusion in and funding from the USAEOP in 2007. GEMS has continued to expand to new sites in the years since (Figure 2). GEMS now serves around 4000 students and nearpeer mentors annually with the USAEOP funding all programs. GEMS and other high school and college programs in the USAEOP are managed through cooperative agreements with academic and nonprofit partners such as the National Science Teachers Association, Battelle Memorial Institute, Rochester Institute of Technology and other past and present partners to provide evaluation support, marketing, administration and additional innovation.
Here, we report our findings from evaluating the GEMS program at both the WRAIR site in the Greater Washington, DC metropolitan area and on a national scale with aggregate data from all of the other USAEOP GEMS programs. This work highlights and evaluates the impact of the founding principles of WRAIR GEMS and explores how the national USAEOP GEMS program influences student change. Despite variations in the GEMS program model across sites, all sites are centered on a shared theory of change. This theory postulates that through providing accessible, high quality, inquiry-centered science enrichment, we can improve students' attitudes toward science and confidence in their own science skills, and ultimately motivate them to persist in STEM pathways and pursue STEM majors and careers. Presently, GEMS invites new and returning middle and high school students each summer for progressively more challenging STEM learning. This approach creates a step-wise STEM pathway toward enrollment in college-level science courses along with experiential learning through a continuum of programs, within and beyond the USAEOP, all the way to post-doctoral fellowships, for the possibility of STEM career support at all levels.
What follows in this manuscript is further explanation of the GEMS program elements and implementation, a detailed history of the program evaluation in its broader capacity, and detailed analysis of the continuity of experiential learning and teaching that begins with the original GEMS program site at WRAIR.

METHODS
While GEMS sessions and programming only occur in the summer, significant efforts and planning are required year-round. Figure 3 details the annual timeline for implementing the WRAIR GEMS program. The planning phase begins in the early fall, as we evaluate our implementation of the previous summer GEMS program at the USAEOP GEMS review meeting. Once we obtain a finalized and approved budget from USAEOP (December/January), we begin the processes of interviewing (December-February) and finalizing staff hires (March), ordering program supplies (April) and preparing program spaces (May), all before the program starts in June. Other major components of GEMS implementation center around NPM and GEMS participant recruitment, which are described in the key programmatic element sections below.
Every summer, WRAIR GEMS offers both biomedical and engineering programs, which run concurrently (8-10 weeks of one-week sessions) and at three levels: Beginning (rising 7th-8th grade students), Intermediate (rising 9th-10th grade students) and Advanced (rising 11th-12th grade students). This approach provides varied STEM programming through which student participants can matriculate for multiple summers.
At WRAIR GEMS, students explore STEM through hands-on laboratory investigations, simulations, computer applications, and engineering design projects. The first day of GEMS includes an overview on safety and a collection of activities, termed "training wheels," designed to train student participants in basic research techniques, use of common laboratory equipment, and safety procedures. For the remainder of the week, student groups conduct several research investigations or engineering design challenges, facilitated by NPMs. These activities vary in length, from 45 minutes to multi-day projects. Students also gain insight into a variety of STEM pathways, careers, and educational opportunities through question and answer ("Q&A") panels or discussions with NPMs and guest speakers, as well as through advertising additional USAEOP programs. Each GEMS session culminates with student presentations, during which each student group presents the results of their multiday group projects in front of their peers, NPMs, and science education researchers.
Successful implementation of GEMS relies on all six of our programmatic elements (Figure 1), as well as rigorous evaluation of program outcomes. These essential program components are discussed in detail in the following subsections.
Accessibility for Underserved Students. USAEOP's definition of underserved includes at least two of the following ("U2"): low-income students; students belonging to racial and ethnic populations that are historically underrepresented in STEM; students with disabilities; students learning English as a second language; students with no parents who attended college; students in rural, frontier, or other federally targeted outreach schools; students who identify as female (USAEOP, 2019). The process of recruiting and selecting middle and HS students is paramount to making GEMS accessible to U2 student populations. We advertise GEMS broadly through school districts within the local area. Additionally, we work diligently to identify U2 populations within local communities and directly recruit students from these areas. GEMS staff also use Spanish language advertisements to accommodate non-native English speakers in our local area. We participate in local outreach events throughout the school year to engage directly with students, parents, and school officials. Solicited students and parents are encouraged to apply for GEMS through an online portal; we then select students and assign them to one week of GEMS. Students generally only participate in one session per summer to ensure that participation in GEMS is available to as many individuals as possible. However, we will make exceptions if space allows. While preference is given to students that meet the U2 criteria, our staff aims to accommodate as many student applicants as possible. Careful tracking of students' schools, residential zip codes, and previous experience with GEMS helps staff achieve the goal of serving U2 students while also maximizing the number of individual students who can attend the GEMS program. For example, a new U2 applicant may be selected before a returning student of the same zip code who does not meet the U2 definition (see participant demographics and acceptance rates in Figures 4-6).
Much as with our student participant recruitment strategies, we work with local universities and colleges to advertise NPM internship opportunities, as well as attend local fairs and events where candidate NPMs are likely to attend. Recruiting from local universities with large populations of students from within the state is important, as it allows us to recruit from the same communities as our GEMS student populations. We also specifically work to recruit NPMs from local Historically Black Colleges and Universities (HB-CUs), Minority-Serving Institutions (MSIs), and commu- lesson plan development, and more. NPMs also participate in team-building exercises, professional development activities (e.g., workshops for resumes and elevator pitches, etc.) and networking activities (e.g., "Faces and Spaces," which is an introduction to WRAIR staff and facilities, in the format of a scavenger hunt; a meet-and-greet event with scientists and engineers from other WRAIR departments or institutions; research laboratory tours; etc.). These activities prepare them for and foster their understanding of their roles in the context of our institution and expose them to other STEM careers. Importantly, NPMs are given ample time to practice their legacy lessons, culminating in a run-of-show on the last day of training, where NPMs facilitate their GEMS lessons in full for their peers and support staff, who provide structured feedback and constructive criticism. The full details of the NPM recruitment, selection, hiring, and training process, for both summer GEMS and other related programs, are beyond the scope of this publication and will be published separately in greater detail (in preparation).
WRAIR GEMS also hires upper-level high-school students in the position of "Assistant NPM". These are typically recent high school graduates, many of whom formerly participated in summer GEMS, and they serve in a similar role as the undergraduate NPMs, with several key differences. First, their internship duration is shorter (four or eight weeks) and does not include the two-week training period as most high schools are still in session during this period. The shorter internship duration is designed to accommodate the reduced time these students have available for internships, due to a shorter summer in the high school to college transition, and additional commitments many students typically make for extracurricular and freshmen activities. Additionally, Assistant NPMs have fewer internship requirements: they are not asked to develop an original module (although they are asked to partner with an undergraduate NPM to contribute to some portion of the lesson plan development, implementation, or presentation); they are not required to teach independently (although many grow comfortable enough to do so); and they are given fewer student supervisory roles (due in large part to the fact that many Assistant NPMs are still minors themselves). The position of Assistant NPM has proven to be a key transition step for many of our former GEMS participants who then return to WRAIR as undergraduate NPMs or research laboratory interns in subsequent years.
Novel Lessons and Learning Modules. NPMs begin the first 3-4 weeks of GEMS teaching their legacy lessons. However, each summer WRAIR GEMS offers novel lessons and activities to participating students. The goal is to feature current, cutting-edge U.S. Army research while also leveraging the expertise and interests of each program year's group of NPMs and resource teachers. To this end, over the course nity colleges; both in order to increase the participation of underrepresented populations in a STEM internship opportunity, and to represent diverse learning pathways to GEMS participants. Through these efforts, we hope students will be able to "see themselves" in their near-peer mentors, and that college students will gain valuable experience mentoring U2 students in STEM (Murphey and Arao, 2001).
Importantly, GEMS is free for all student participants and includes a small stipend to cover associated costs such as transit and food. GEMS NPMs and other staff (who are largely interns and fellows) receive competitive stipends for their experience and education level. The ability to provide monetary support for participants and staff mitigates financial barriers to this summer STEM experience.
Near-Peer Mentors. During each week-long GEMS session, small groups of 6-8 students work alongside college-aged NPMs, who are pursuing STEM degrees and careers, to explore STEM topics. NPMs are mainly responsible for daily facilitation of laboratory activities and investigations, answering student questions about STEM pathways, and classroom management.
High-quality near-peer mentorship begins with recruitment of appropriate NPM candidates (see more about recruitment strategies in the previous subsection), and training in a variety of topics and skills. Our NPM interview and selection strategy is designed to gauge and maximize applicants' abilities or philosophies in several key areas: (1) STEM content knowledge, (2) mentorship and teaching experience and/or desire, especially with young students, (3) laboratory and technical skills, (4) teamwork and leadership, and (5) working with underserved populations in STEM. NPMs are hired at a ratio of one NPM per six participants within each weekly program; years of programming and feedback from both students and mentors have provided anecdotal evidence that when groups are any larger, it is hard for NPMs to provide individualized attention and mentorship to students.
After hiring selections are made, NPMs review pre-written, or 'legacy' lessons, selected by WRAIR GEMS staff to correspond to the NPM's interests and skills, and to ensure that a diverse and cohesive set of lessons are offered. Most legacy lesson plans are designed by current or former NPMs, with help from WRAIR scientists and engineers, and resource teachers (local teachers who help with curriculum development and design, teaching and classroom management); or they are sourced from peer-reviewed publications and professional organizations. At the start of their internship, before interacting with students, NPMs participate in a two-week paid training period, which includes training in the use of laboratory equipment, care and use of a live-animal teaching collection, equitable and inclusive teaching techniques (including student-centered, active learning techniques), fostering growth mindsets, science communication, of the summer, NPMs also design, teach, and assess the outcomes of their own novel lessons. These efforts are guided and supported by assigned research mentors (WRAIR staff, typically post-doctoral researchers, scientists, and engineers). At approximately week four or five (though sometimes earlier), NPMs transition from facilitating legacy lessons to implementing their original lessons. The NPM internship culminates with an in-house summer STEM Expo, for which NPMs prepare a poster of their science education research, detailing student outcomes following participation in the original lesson plan. The poster is then presented to WRAIR scientists, engineers, and other interns and fellows at the institution. Some of the more novel lessons are prepared for publication in relevant journals (Hammamieh et al., 2005;Morales et al., 2017). This structure has a variety of benefits: (1) it allows returning students to experience novel lessons and STEM content; (2) NPMs are given freedom and autonomy in their lessons, which serves to motivate and engage them; (3) NPMs gain valuable experience in lesson development, implementation, data collection and presentation (4) WRAIR GEMS staff expand their options for future legacy and outreach activities; and (5) current STEM research and practices can be incorporated into lessons, allowing students to engage with the most up-to-date scientific practices and knowledge.
Hands-on Laboratory Activities. At WRAIR GEMS, we help NPMs to use and develop lessons that center around hands-on, scientific inquiry and/or the engineering design process, to foster active learning in small student groups. It is sometimes difficult to train educators to lecture less; this may be due in part to the fact that lecturing is often the default manner by which we disseminate information throughout our years in academic (and other) settings. However, because practices that foster active, student-centered learning have been shown to have positive effects on student outcomes, including content knowledge gains and attitudes toward STEM (e.g., Freeman et al., 2014), it is important to continually encourage NPMs to frame lessons in such a way that students are active participants, conducting their own exploration, and generating answers to their own questions.

Connections to Local Scientists and Engineers.
As part of each weekly GEMS session, we invite a guest speaker-a DoD scientist or engineer-to give an age-appropriate research talk and answer questions about themselves regarding career choices, challenges, and how they overcame them. The scientists and engineers at WRAIR are from diverse backgrounds and have a wide variety of subject-matter expertise, including entomology, infectious diseases, neuroscience, pharmacology, toxicology, blast-induced neurotrauma, ecology, microbiology, veterinary medicine, psychology, STEM education, and more. We advise speakers to bring items that students can see or touch (e.g., demonstrations, pictures, or laboratory equipment) to engage students. One goal of this weekly talk (with a Q&A session) is to expose students and NPMs to STEM careers that they may not have been aware of before, and to expose them to a variety of pathways into STEM careers, particularly those within the U.S. Army, Department of Defense, or other federal agencies, which are often underappreciated and unadvertised in academic settings.
Sustained Pathways to STEM Programming. WRAIR GEMS offers a continuum of STEM experiences for middle and HS student participants. WRAIR also offers research laboratory internships for HS students through the USAEOP Science and Engineering Apprenticeship Program (SEAP). At the end of HS and during college, several former GEMS participants apply for education or laboratory research internships at WRAIR, via NPM internships and USAEOP College Qualified Leaders (CQL) internships. Upon graduating from college, former NPMs may return to WRAIR for post-baccalaureate fellowships through the Oak Ridge Institute for Science and Education (ORISE), either in our biomedical research laboratories or our STEM education programs. Finally, WRAIR offers postdoctoral research fellowships through ORISE and the National Academies of Sciences, Engineering and Medicine (NASEM). Providing free STEM programs (with stipends) for 7th-12th grade students, and STEM internships/fellowships with competitive stipends for individuals, from 11th grade students to Ph.D. graduates, helps WRAIR to sustain continuous pathways into STEM careers while benefiting WRAIR with their services.

Evaluation of WRAIR GEMS Program Goals.
During early development of the GEMS program at WRAIR, funding through NIH SEPA was leveraged to establish and disseminate the program. Over the years, WRAIR GEMS has conducted site-specific evaluations to elicit feedback from both students and NPMs, which have allowed for recent publications regarding student participant perceptions of the WRAIR GEMS program (Tenenbaum et al., 2017) and long-term impact to NPMs who participated in the program (Anderson et al., 2019). Importantly, site-specific evaluation measures have allowed our team to track the self-reported demographics and participant numbers from our own program, allowing us to evaluate our accessibility to underrepresented populations in STEM and our influence on students' persistence in STEM. As much of the more current WRAIR GEMS evaluative work has been published, we use this publication as an opportunity to showcase WRAIR GEMS participant numbers and demographics in recent years (2013)(2014)(2015)(2016)(2017)(2018).

Evaluation of GEMS on the National Scale.
Every year, USAEOP hosts a post-implementation annual review of the GEMS programs across the nation. This meeting is critical as we discuss program fidelity, evaluation strategies, stipend levels and other processes. The annual review is the ideal time to prepare and set goals for the upcoming program year, particularly regarding how to increase program impact, identify partners, and achieve program goals. Critically, at this point, discussions begin regarding program sites, target participant numbers, and NPM and resource teacher recruitment, allowing for a draft budget to be completed.
Annually, beginning in 2012, USAEOP has managed and reported a nationwide GEMS program evaluation in which GEMS students, with appropriate assent and parental con-sent, are asked to participate. Data from all GEMS sites are aggregated and evaluated, and therefore we were unable to obtain site-specific data for WRAIR GEMS from USAEOP reports (USAEOP, 2015(USAEOP, , 2016(USAEOP, , 2017(USAEOP, , 2018(USAEOP, , 2019. Presented here are exit responses from a subset of USAEOP survey items, from years 2014-2018, which collectively speak to the impact of the nationwide GEMS program (Table 1). These survey items were selected to showcase changes in students' interest in STEM, feelings of STEM identity and self-efficacy, and understanding of important aspects of the scientific process after program participation. The items were also chosen prior to assessing any proportions of positive or negative responses in each of the five annual reports from 2014-2018.

Statistical Analyses of Data from USAEOP GEMS.
Upon graphing survey responses over the five years for which we pulled USAEOP-wide GEMS data, WRAIR researchers identified a possible negative trend; learning gains for the last two survey items appeared to decrease over time. This potential trend would not have been easily identified via analysis of within-year data from the USAEOP annual reports. To address the question of whether or not this trend was statistically significant, researchers used beta regression to model proportions of students choosing each response ("learned a lot", "learned more", "learned a little", and "no [learning] gains") by year, for each of the two questions, using the R package "betareg" (Cribari-Neto and Zeileis, 2010; R Core Team, 2019). Beta regression was selected because it works well for responses that are constrained between 0 and 1 (e.g., proportions), and that may not be normally distributed (Cribari-Neto and Zeileis, 2010).

Results from Evaluation of WRAIR GEMS Program Goals.
Previously published studies demonstrated that GEMS was largely a positive experience for students (Tenenbaum et al., 2017), and helpful for persistence in STEM fields among former NPMs (Anderson et al., 2019). Students reported developing a strong rapport with their near-peer mentors and receiving guidance and support in the areas of STEM learning, educational planning, and life skills (Tenenbaum et al., 2017). Many students also felt that the program offered engaging lessons that encouraged them to learn while developing independence and self-efficacy with a group of like-minded peers (Tenenbaum et al., 2017). Another important goal of the GEMS program is to foster interest and equip students with the tools to attain STEM majors or careers. We evaluate this goal by measuring a student's completion of a STEM degree or attainment of a STEM job/career after participation in our program. We classify this measurement as a student's persistence in STEM. We recently reported our findings in a study and found that former NPMs overwhelmingly tended to persist in STEM and/or STEM education fields (39 out of 40 online survey respondents) and had an increased commitment to mentorship and STEM education (Anderson et al., 2019).
Since 2014, WRAIR GEMS has grown from accepting around 400 student participants each year to well over 600 participants in recent years (Figure 4). Furthermore, the number of returning students ranges from ~30% to 50% of the total student number, highlighting the degree of student persistence in the program ( Figure 5). Despite a growing applicant pool, our program has been able to accept >50% of applicants for most of the reported years ( Figure 4). Demographic information demonstrates that the majority of GEMS participants have been from racial groups that are underrepresented in STEM ( Figure 6A), with the two largest groups being African American/Black as ~45% and Hispanic/Latinx American as ~20% of the participants. This finding supports the program's mission. Parallel to GEMS participants, many WRAIR NPMs have been from underrepresented groups in STEM as well ( Figure 6A). These data result from the intentional recruitment of participants from underrepresented groups in STEM and reflect the fulfillment of the main goal of the GEMS program. Further, these demographics reflect that of local county populations ( Figure 6B).
WRAIR GEMS relies upon resource teachers, volunteer speakers (scientists and engineers), NPMs, and assistant NPMs to help implement the program. As the number of WRAIR GEMS participants has increased throughout recent years (Figure 4), so has our need for support staff and volunteers (Figure 7). The number of GEMS staff and volunteers has nearly doubled in recent years compared to 2013.
Results from Evaluation of USAEOP GEMS on the National Scale. Nationwide, across years 2014-2018, a majority of students agreed that USAEOP GEMS contributed to increased interest in taking STEM classes in school (68.9%) and participating in STEM activities outside of school requirements (76.2%), with the exception of the year 2015 ( Figure 8). A majority of students also credited GEMS with their increased confidence in their STEM knowledge, skills, and abilities (88.3%), and their increased interest in pursuing a career in STEM (79.1%) (Figure 8).
Further, the vast majority of students felt that GEMS contributed to learning important 21st century skills (Bell, 2010;Bellanca, 2010), such as (1) how to make changes when things do not go as planned, and (2) how to view failure as an opportunity to learn. For example, 88-97% reported that they learned at least a little, while 33-69% felt that they learned a lot with respect to these two skills (Figure 9). Despite these positive outcomes overall, there was a statistically significant decrease in the number of students who "learned a lot" about these skills during GEMS from 2014-2018. In this time period, the estimated odds that students "learned a lot" about making changes when things did not go as planned decreased by 51.2% (SE = +/-1.1%; z = -6.37; p <0.0001). Similarly, the odds that students "learned a lot" about viewing failure as an opportunity decreased by about 55.4% (SE = +/-1.1%; z = -5.52; p < 0.0001).

DISCUSSION
We have reported several positive outcomes of the WRAIR GEMS program and the nationwide USAEOP GEMS program. WRAIR GEMS participant demographic data suggest that we are recruiting individuals of diverse racial and ethnic backgrounds in proportions similar to the surrounding counties ( Figure 6). WRAIR GEMS attendance data also strongly suggest that the program is accommodating both new and returning students, indicating that the program is promoting students' persistence in STEM education programs and creating a continuous STEM pathway ( Figure  5). Not only has the number of WRAIR GEMS participants increased since 2014, but the number of support staff, including scientists and engineers who volunteer to speak at GEMS, has also increased in this time period (Figures 5 and  7). This result is important as a major component of GEMS is to facilitate connections between students and local STEM professionals (Figure 1). Additionally, USAEOP GEMS survey data support that nation-wide, GEMS students learn two things that are important to authentic research experiences: that they must adapt when things do not go as planned (e.g., due to flaws in experimental design), and that failure provides the opportunity to learn (e.g., failing to find the predict-ed outcomes) (Figure 9). The USAEOP survey data strongly suggest that GEMS contributes to increases in students' confidence in their STEM knowledge, skills, and abilities, as well as their desire to persist in STEM (via taking additional STEM courses, or pursuing STEM careers; Figure 8).
We have uncovered some areas in which WRAIR GEMS and USAEOP GEMS could improve. Relative to the local population, Hispanic and Latinx Americans are still slightly underrepresented among WRAIR GEMS students and NPMs ( Figure 6). However, recent, more formalized, partnerships with HBCUs, MSIs, and their local public school districts will hopefully help us to address this concern. We have also started to use Spanish language advertisements to recruit students for GEMS.
On a national level, the USAEOP GEMS survey data suggest that the proportion of students learning "a lot" at GEMS, in response to two example survey items has steadily decreased over time from 2014-2018 (Figure 9). These items are: (1) how to make changes when things do not go as planned, and (2) learn to view failure as an opportunity to learn. This is somewhat disconcerting and we can only speculate as to why this might be the case. From direct site visits and the annual GEMS review, we know that fidelity to WRAIR's original model for GEMS is variable across sites. If there are some sites that offer the same laboratory experiments year after year, rather than offering novel content, this may be one possible explanation for the pattern. Another possibility is that budget constraints might make it difficult to mentor NPMs as they teach, and continuously encourage them to lecture less and make more room in their modules for true scientific inquiry, as the latter should lead to learning gains in the two example survey items. Thirdly, recruitment of more students advantaged by high-performing school districts, instead of underserved students, by other GEMS sites could result in a program cohort that is already well educated in STEM. These possibilities will need to be investigated with further program analysis. Future Directions. We know from observation and discussions with GEMS staff across USAEOP sites that GEMS runs differently at different sites. It is imperative to directly assess aspects of programs that mitigate learning gains in order to be sure that we are investing in a program that adequately addresses systemic inequities and barriers to STEM pathways across the nation. As the GEMS program was founded with the priority of serving underserved students, it is important to ensure that this goal is accomplished at every additional site. This effort will help to ensure that the GEMS program is truly promoting science interest and engagement for underserved students. Furthermore, our research has found that the relationships between the participants and NPMs is a crucial component to the success of the program (Tenenbaum et al., 2017). Considering this finding, it is recommended that our NPM recruiting and training model be implemented across sites to ensure replication of success. As different AEOP GEMS programs have the freedom to adapt the GEMS model to fit their geographic area, the development of a method to ensure program fidelity through adherence to the core principles of the original proven program and incorporating evidence-based strategies is recommended.
As interest in the WRAIR GEMS program continues to grow, the challenge of serving an increasing student population has emerged. Additionally, we have noticed from informal program feedback that the ability for students to travel to our remote site on a daily basis is a potential barrier for many underserved students. To combat both of these issues, we are exploring the concept of implementing GEMS at WRAIR-sponsored community sites. We have begun forming partnerships with colleges, particularly HBCUs and community colleges to provide additional workspace for students in their own communities. Linking our college partners to young students in their communities presents an innovative solution to serve more of our target populations and aid in the continuity of STEM pathways by exposing students to a college environment. Furthermore, partnering with local colleges allows us to directly recruit NPMs that are more familiar with our students' local environment.
Further reflection upon the WRAIR GEMS program has exposed additional challenges that prevent us from serving U2 students. Within the greater Washington, DC metropolitan area, the $100 dollar stipend may not be sufficient to offset the travel and food cost associated with attending the week-long program. WRAIR GEMS has consistently advocated for increased student stipends which has resulted in an increase to $125 for the 2020 summer program. With the expansion of WRAIR GEMS into community sites, we have used our college partners to sponsor meals for participants, which has been well received by students and their families. In addition to advocacy for increased stipend funding and meal sponsorship, we will continue to explore options to provide better access to transportation and to breakdown any other barriers to participation in our programs.
WRAIR GEMS legacy modules explore a mix of topics centered on NPMs' own areas of expertise and designed to feature DoD-related research. However, highlighting current DoD research in GEMS modules is a challenge for NPMs who are newly-exposed to DoD research themselves. We have begun a new initiative to provide NPMs with more DoD laboratory experience by creating a dual program (DP) of research and science education service. The DP fellows are funded through a WRAIR leadership process to distribute Army and DoD funds for laudable and underfunded purposes. ORISE then manages these funds to support ninemonth fellowships in WRAIR research laboratories, followed by a three-month period serving in GEMS as NPMs. During GEMS, they are asked to develop an original GEMS lesson based on their WRAIR laboratory experience. This effort brings awareness of and access to DoD research directly within the GEMS experience.
Broader Impacts. Ultimately, the GEMS model is a unique approach providing sustainable and continuous programming in STEM, while simultaneously offering mentorship opportunities and evaluating student change. Inequitable educational opportunities place underserved and underrepresented populations at a distinct disadvantage when it comes to pursuing STEM degrees and careers. Implementation of the GEMS model has led to sustainable programs nationwide that promote shifts in students' science attitudes and persistence in STEM education. This work has shown that the implementation of this model can contribute to efforts to address systemic inequities in STEM education, and provide a blueprint for other institutions to disseminate evidence-based outreach programs that contribute to educational reform and student success.