A Survey of STEM Outreach Programs During the COVID-19 Pandemic

: The recent COVID-19 pandemic required schools around the country to shut their doors to student and teacher learning, impacting both formal and informal education. STEM outreach programs scrambled to revise their programming to reach the “new normal” needs of teachers and students. STEM Outreach directors from Vanderbilt University and West Virginia University developed and implemented a survey to determine the impact of the COVID-19 pandemic on STEM outreach programming around the country. Sixty-one programs responded to the survey, describing a total of 115 outreach projects in partnership with K-12 schools. Each Principal Investigator (PI) provided a description of their programs, and indicated the impact of the pandemic as well as the adaptability of the programs to a virtual platform. PIs also noted new programs that they had started as a result of the pandemic, challenges that they had to overcome, and innovative strategies that might enhance the use of technology in future STEM outreach partnerships. More than 80% of the PIs indicated that they had made changes to move completely to virtual approaches. Results from this survey strongly suggest that although the pandemic has been an ongoing challenge for many STEM outreach programs, many have risen to the challenge and created virtual and hybrid programs that can be a model of change for the future.


INTRODUCTION
"The pandemic has upended the present. But it has given us a chance to remake the future." (F. Zakaria, Washington Post).
In 1937, a severe polio epidemic hit the U.S. In response, many cities closed public places such as pools, playgrounds, and theaters. In at least one city -Chicago -schools were closed for 15 weeks. In response, educators and experts came together to develop and deliver the first large scale "radio school" experiment to send out lessons across all subjects to the city's more than 300,000 elementary and middle school children (Foss, 2020). Classes were short (15 min) with an objective of being "entertaining yet informative." The program ran for only three weeks, but introduced the concept of "remote learning" for K-12 students.
Fast forward to the COVID-19 pandemic that has contin-ued since December of 2019. No one could have imagined the impact that this pandemic would have in all aspects of our lives. As researchers raced to find new treatments, to understand the biology of the virus, and to develop vaccines that might finally slow or halt the disease progression, humanity was forced to adapt to the continuing changes that impacted our lives. One major impact was certainly on the education of our children. Schools worldwide began shuttering their doors and sending students home to carry on their learning from their homes, through self-guided virtual lessons or with the help of parents who were often sheltering at home as well (Frenette et al., 2020;Peterson et al., 2020;Kolb, 2021). Many schools attempted a mixture of virtual and in-person classes (hybrid), while others went to totally virtual learning. These choices led to significant challenges for all involved. Parents who had to work were faced with difficult decisions about leaving their younger children at home, or tried to find out-of-house options for assistance. A recent Pew study has shown that most parents of K-12 students worry that their children are falling behind because of disruptions due to the pandemic (Horowitz and Igielink, 2020). Garbe and colleagues reported that while parents agreed with school closures and were satisfied with the level of support from school districts, they were less confident of providing a balanced and supportive learning environment while at home (Garbe, 2020). Teachers, who were suddenly faced with teaching virtually, had to deal with limited knowledge of the technology involved and little time to adapt classroom practices to the virtual world (Kaden, 2020;Middleton, 2020;Stephanile, 2020).
One important collaboration that has evolved over the past several decades is the partnership between STEM "experts" at universities and informal STEM education institutions and the preK-12 world. Hundreds of these partnerships have established innovative and highly effective approaches to enhance teacher STEM content and confidence in teaching STEM; increase student achievement in STEM areas; and enhance student excitement about STEM and interest in pursuing STEM careers (Tomanek, 2005;Laursen, 2007;Meyer, 2017). As the pandemic took hold and educational institutions were shuttered, these partnerships were suddenly confronted with the challenge of maintaining their connections with teachers and students through adaptation of their programs to virtual platforms. Although technology has infiltrated our educational practices as new strategies have become available over the past several decades, researchers, teachers and students have found that many learning approaches simply cannot be reproduced virtually. Students cannot participate in hands-on research opportunities in university labs; students are limited in their usual collaborative teamwork in classrooms; and teachers struggle with having to direct hands-on activities in virtual or hybrid situations. This results in limitations to partnerships across the spectrum including student-student, teacher-student, teacher-teacher, and student-researcher connections.
Online learning is of course not new. A plethora of online learning opportunities have appeared over the past decade or more including online curricula (Brodersen and Melluzzo, 2017), online lab simulations (Callaghan et al., 2021;labster.com); historic and real-time online data collection and analysis (Pennisi, 2020); and online courses for teachers and students, etc. (Callaghan et al, 2021;labster.com;Pennisi, 2020). In addition, increasing numbers of school districts have noted an increase in the number of students entering online K-12 schools, and universities are moving toward changing their partnerships with K-12 schools to online (McKenzie, 2020).
In the current article, we report the findings from a survey administered to researchers in the STEM outreach commu-nity in May of 2020. The focus of this survey was to determine the impact that the pandemic had on ongoing researcher-K-12 partnerships, and what strategies these programs were using to overcome the pandemic challenges, with a goal of this study of learning how our colleagues are adapting and innovating during the pandemic.

METHODS
Survey Description. A survey was developed to send to over 100 STEM Principal Investigators (PIs) at universities and informal education institutions (survey included in Supplemental Material). The total population was difficult to quantify, since the survey was sent through a variety of blinded listserves and posted on social media. A total of 61 responses were received. The majority of respondents represented large public universities (54%), with 28% from small private universities. The remainder were spread evenly across informal institutions, large private universities, small public universities, and historically Black colleges and universities.
The first survey question asked each participant to list up to three partnership STEM programs, to describe each program in less than 100 words, and to indicate how much each program was impacted on a scale of 1 (low impact) to 100 (high impact). Participants also described if each program could be adapted to a virtual platform, again on a scale of 1 (low adaptability) to 100 (high adaptability). An additional seven open-ended questions asked Principal Investigators (PIs) to describe new programs started because of the pandemic; list successful virtual approaches; indicate if there was an increase in program requests; describe challenges that researchers anticipated in adapting programs to online; describe potential equity challenges; describe innovative or out-of-the box approaches that have been tried or planned; and indicate how the use of the virtual space has introduced new strategies that might alter how we teach and learn STEM.
Coding. Qualitative analysis was used to develop a coding framework from themes that emerged from the survey open-ended questions based on methodology previously used by this research group (Ufnar et al., 2017). Two researchers collaboratively developed a preliminary scheme to capture each primary and secondary emergent theme from all data sources. For the purposes of coding, instances were defined as each survey response. The three authors randomly sampled 10% of the data set to test the coding scheme. The scheme was created and refined by categorizing participant comments, adding categories when emergent themes were not captured, and eliminating or collapsing categories when instances were extremely rare or it was difficult to make reliable distinctions between categories. After a rate of agree-ment between coders of over 80% was achieved, the scheme was judged to be stable.
Two primary themes emerged from this analysis: Program Adaptation (PA) and Challenges (C). Each primary category was further subdivided into several secondary categories to capture the themes emerging as most significant. Through fine-grained analysis, secondary categories were further subdivided into tertiary categories. This final analysis was conducted by examination of open-ended survey questions (Table 1). The inter-rater reliability for the coding using the tertiary categories was greater than 85%, with 198 open-ended survey responses analyzed.
The primary theme of Program Adaptation (PA) was assigned to an open-ended question when the PI indicated that they had adapted their program to be delivered virtually. The secondary categories were assigned to describe the primary approaches that the PI was implementing. For example, several of the programs that focus on in-person instruction for middle and/or high school students have now adapted the curricular units to a virtual platform; these programs were coded as Curriculum Development (CUR). The secondary code "participation (P)" was used if the PI noted a change in participant recruitment (PR), an increase in participation (IP), or a change in student or teacher engagement (SE, TE). Responses were coded as a Challenge (primary category), with eight secondary options and tertiary codes within the secondary series (in parentheses following the secondary code). The Access (A) code was used when access to the internet was negatively impacted, either due to Equipment (ER) or Wi-Fi access (TECH). The Access secondary code also included the impact of equity when attempting to learn via the virtual approaches. Another common secondary category under Challenges was the issue of Connections (UC), assigned to those responses that indicated a negative impact on several connections (student/student, student/researcher, student/teacher, etc.).

Confidentiality of Data.
Electronic files were kept on a secure server maintained by the Collaborative for STEM Education and Outreach at Vanderbilt and password protected. Identities of the survey respondents were known only to the authors. The Vanderbilt Institutional Review Board designated this study as exempt without the need for informed consent.

RESULTS AND DISCUSSION
Program Descriptions and Adaptability. In question one of the survey PIs were asked to describe up to three programs, discuss the impact of the pandemic on each program, and score each program for the level of impact (1=low to 100=high) and the level of adaptability (1=low to 100=high). A total of 115 programs were described, with 61 respondents providing information about one program, 32 descriptions of a second program, and 22 for a third program. Respondents represented universities or colleges in 28 states and one in Canada, with an additional four informal agencies including

Challenges (C)
Logistics (  museums, zoos and aquaria, camps, or groups focusing on extracurricular programming. Universities included small private top-tier universities, small private colleges, and large state universities. Program descriptions and impact of the pandemic were analyzed using coding analysis as described in Methods. Two primary categories were defined -Program Adaptation (PA) and Challenges (C) -with seven and eight secondary categories for each primary code, respectively. Tertiary codes were also identified during the analysis and are discussed for each secondary code.
Program Adaptations. A total of 66 secondary instances under the primary category of Program Adaptation were identified as shown in Figure 1. The highest category was teacher Professional Development (PD) (22 instances), with most programs adapting summer workshops to a virtual format. For example, one PI noted: "Teachers are looking for support to implement this curriculum in online format, and we are working to accommodate them." Most of the virtual workshops reduced time that teachers would spend online to avoid "Zoom fatigue" (Blum, 2020), and provided a balance of synchronous and asynchronous activities. Several sites stated that they offered time for teachers to create professional learning communities to promote teamwork and teacher-teacher collaboration. PIs also mentioned that the focus of the PD workshops had shifted to providing support for implementing curriculum virtually. As one PI noted: "We were unable to do the hands-on, outdoor activities and opted instead for sharing ideas and tools for engaging students in virtual environments with easy adjustments to switch to face-to-face." Additionally, as noted by one PI, "Teachers enjoyed not having to drive to the university to "meet". The time saved was more effectively used to address the assigned projects." Twenty sites were continuing Curriculum Development, with some adjustment for implementing the curriculum virtually. This category includes a wide range of curricular materials including lectures, presentations, and discussions that connect students with project staff in real time combined with posted resources such as videos, readings, and activities for students to examine independently. One PI stated: "We have adapted longer curricula into bitesized modules that are amenable to providing to students as self-directed independent assignments in distance-learning format." One group that focuses on curriculum around river ecology developed a virtual unit for elementary/middle school students. While their field trips to the river had been cancelled, they commented that: "We did, however, create a lesson set revolving around the food web, including a game through Powerpoint, a drawing activity, and a recorded lesson video." One project commented that "All the materials for teachers and students are already downloadable and freely available, so teachers can use them in a virtual format." The program staff is providing support for teachers to use the activities for teaching students virtually. The PI commented that "We believe the pandemic will result in even greater interest in this type of program as they look for motivating, engaging projects that can be delivered…online." Hands-on Science was examined as a separate secondary category, since many of the camps, informal activities, and curriculum implementation involved development and use of kits. Five sites were able to move both in-classroom and informal projects that involved hands-on activities to a virtual platform, combined with kits that were either dropped off at public sites (libraries, schools, etc.) or mailed directly to participants' home. The project staff provided synchronous or asynchronous instructions on how to use the kits. As one PI noted: "we assembled kits that were sent home with the students so that they could still participate in real hands-on science." Sixteen instances of virtual Laboratory Experiences substituting for on-campus student research were identified, including approaches such as simulated labs, and data collection and analysis. One PI noted: "With remote learning instituted in mid-March, we had to find ways for students to collect data and collaborate on research projects online." Another project commented that "A result has been bringing together youth from all over the state…because the travel barrier is gone. Youth are set up to continue their data collection after the series ends by nature of the virtual platform used." Further comments emphasized online simulated laboratories, and the use of "historic data online to create a research project centering around…water quality." There was at least one lab that was "open to taking teachers and working with them to do some projects remotely." Most of the virtual programs combined synchronous didactic instruction for students using Zoom or other similar platforms with data analysis or lab simulations to replace the research projects. home environment for working remotely; inability to adapt in-classroom hands-on experimentation; a belief on the part of researchers that virtual programming would not provide authentic research experiences for students; and limited interest from teachers or students in participating remotely.
For those programs that were able to adapt or continue their projects, the challenge instances were assigned to six secondary categories ( Figure 2). Loss of Connections among participants was the highest with 25 instances. As noted by most of the PIs who are attempting a virtual student program, the students lose the connections with the university faculty, as well as interactions with their peers and their teachers. This secondary category was subdivided into seven tertiary categories: mentoring (five instances); campus culture (five); labs (six); student-student (three); student-teacher (two); student-researcher (two); and community (one). PI comments highlighted the challenges in this category: "We are concerned primarily with the person-to-person (peers, near-peers, role-models, caring and supportive faculty and staff, college dorm and campus experiences) and formation of identities that the students are not able to experience." "Students did not experience being on University campus and being immersed in different lab experiments." "In person teaching has been cancelled which limits the ability of conducting experiential learning programs where interactions between student and teacher facilitates the learning." Engagement of students and teachers in virtual learning also presented challenges for outreach programs with 18 instances. Instances were further assigned to six tertiary Mentoring via the internet guided students through basic research techniques, provided simulations and videos, and instructed students how to collect and analyze data. As one PI noted: "This summer, we have a "virtual" version that is completely online." Thirty students meet two times per week for six weeks for 90 minutes each session; the focus alternates between biomedical topics and careers. The PI added: "Quite different, but as interactive as possible." The virtual laboratory category was also broken down into two tertiary categories: data collection (five instances), and laboratory simulations (two instances). For example, participating mentors in one project "guided students in learning the scholarship/scientific underpinnings of their research and then offered data analysis and writing projects to students as an e-learning experience." A common theme across many of the PI responses was the importance of Connections including the tertiary categories of student-student (three instances), student-researcher (two instances), teacher-teacher (one instance), teacher-researcher (two instances), and mentoring (one instance) as indicated in Table 1. This became particularly difficult to maintain through a virtual platform, but several projects specifically focused on this issue, and pointed out how their program had managed to maintain and/or increase connections. For example, one PI commented: "A result has been bringing together youth from all over the state… Youth are set up to continue their data collection after the series ends by nature of the virtual platform used." Another emphasized the importance of connections: "We had to find ways for students to collect data and collaborate on research projects online." Six programs focused on offering out-of-school or informal programs such as summer camps or after school programs. Although several of these were cancelled, a few successfully adapted to a virtual format. In one project, graduate students introduced 9th graders to computer programming, learning in the process the importance of active learning strategies. In another program adapted virtually, students conducted kitchen science activities at home and documented their activities using photography. They then met with program staff in the afternoon to share their results. One program continued to meet in person, hiring additional staff to monitor safety regulations and reducing the number of students from 30 to ten. Program Challenges. There were a total of 74 secondary instances under the primary category of Program Challenges as shown in Figure 2. Included in challenges were those projects that cancelled a portion of their programming (PC) and those projects that had to cancel entirely (NP) (22 instances each). Cancellations fell primarily in the following categories: lack of funding to adapt programs; reduction in staff; concerns about safety of adult-minor interactions over the internet; lack of access to reliable internet and/or supportive categories: reduced participation (eight instances); informal student participation (three); student engagement (three); burnout (two); teacher engagement (two); and researcher/ scientist engagement (one). Examples of the engagement category include the following: as PD workshops had to go online, PIs were finding it difficult to engage teachers: "teachers are too exhausted and overwhelmed to do any PD." Students struggled at times with virtual learning: "Students do not like taking classes online and would prefer being in the classroom." Additionally, PIs found it challenging to ensure that students would attend the virtual programs: "one of the more difficult problems is how to make sure that students are attending to the learning." In the Grant Requirements category, PIs expressed the challenges associated with fulfilling required goals in their funded grants (14 instances). Included in this category were issues with completing or carrying out evaluation plans (nine instances), and lack of funding to adapt to a virtual platform. For example, one PI noted "Assessment was curtailed as programs ended and IRB prohibited data collection." Another PI stated that "the current pandemic has impacted our formative evaluation plans as well as plans for conducting focus groups with teachers and students." Additionally, a PI noted that "evaluation data collection has been greatly curtailed because school systems don't want to spend classroom time on discretionary activities, given how much learning time has been lost." One program that had moved their PD to an online format stated that there was a significant impact on their budget since a large part was designated for stipends and travel costs for teachers.
Ten instances were assigned to the Logistics secondary category, which included the tertiary categories of cost of adaptation (one instance); staffing (two); legal issues (two); planning time (two); and participant recruitment (two). Two programs responded that their entire outreach staff had been laid off by the university, and they were uncertain if their program would continue if and when schools reopen. There was also an impact on recruiting; as one PI noted: "there appeared to be a large drop-off in the number of students applying for the program for the fall of 2020." Finally, at least two PIs were concerned about legal issues in working with minors online: "Remote (web) interactions have been challenging because of concerns about…policy/safety when adults interact with minors." Seven programs noted the challenge of lack of Access to reliable Wi-Fi and lack of the required equipment to access internet-based outreach. As one PI commented: "After speaking with schools about a virtual option, we decided not to offer that -schools noted that many students lacked internet access and/or computers and were unlikely to participate in an online program." There were five instances of challenges in the School-Based category. These programs relied on scientists going into schools and working specifically in classrooms. In a GK12-like program (Ufnar et al., 2017), scientists (fellows) are now working on developing virtual lessons. In another program, in-person visits were cancelled for 2020 and the PI is uncertain "whether Innovators in Residence [fellows] will be allowed into the schools" in the fall of 2021.
Ranking of Impact and Adaptability. Researchers were asked to rank the impact of the pandemic on their programs and the adaptability of their program to a virtual platform on a scale of 1 (low impact) to 100 (highest impact). As shown in Figure 3A, impact ratings ranged from 1 to 100, with an average rating of 79, indicating a significant impact of the  pandemic on adapting to virtual programs. A shown in Figure 3B, the overall adaptability rating ranged from two to 100 with an average of 60.

New Programs and/or Approaches in Response to the
Pandemic. Twenty-two PIs responded that they had started new programs in response to the pandemic, while 23 PIs stated that they had not started new programs but were actively adapting their projects for remote learning. The new programs fell into ten categories as shown in Table 2. Five programs were developing videos in support of online cur-riculum, while four programs were developing new curriculum with two courses specifically focused on COVID-19. Other programs had moved student skills development associated previously with on-campus internships to a virtual platform, while several had developed online research modules or virtual journal clubs.
PIs were also asked to list virtual strategies that they have found most successful. Over half of the PIs outlined new adaptations they had begun; specific comments outlining their strategies are shown in Table 3. Many PIs reported using Zoom to connect with classrooms, primarily for delivery of interactive/synchronous lectures. In addition, the breakout

Games
Redesigning the game-based learning environment of our existing game to make sure it is scalable and is able to support very large simultaneous implementations Creating new game content focused on COVID-19 PD PD workshop to assist teachers on tools for online teaching Working with teachers to support online/remote instruction  feature on Zoom has been useful in creating small groups to promote collaboration. Several PIs reported using Google Classroom to deliver curriculum, an approach that they felt teachers were most comfortable with. Several projects are using Zoom for their PD, limiting the number of hours that teachers spend online together with materials and activities posted online. Several groups mailed or delivered hands-on kits together with online guidance on implementing the kits virtually. Two PIs mentioned the effectiveness of using near peer mentors (undergraduate and graduate students) to develop new interactive tools as well as provide online role models/mentors for students. Several PIs noted that they were using novel online learning platforms including Schoology, Microsoft Teams, Zoom Breakout Rooms, Paddle, Direct Poll, Flipgrid, StoryMaker, and Labster. In addition, PIs reported using the user-friendly Google Suite applications that can enhance engagement and collaboration including Google Docs, Google Slides, Google Sheets, Google Drawings, Google Forms, Google Sites, and Google Keep. PIs mentioned several points that help to make online strategies more effective including shorter online/synchronous time to avoid burnout for teachers and students; asynchronous resources to enhance both teacher and student engagement; smaller groups together with online tools such as Zoom Breakout Rooms to foster collaboration; and creating interactive resources that teachers and students can easily access.
PIs were asked if they had received any requests for virtual programming. Nearly half of the responses from PIs indicated that they had gotten no increase in requests. From the ones that responded yes, a typical response talked about teachers asking for help in dividing existing lessons into 2-hour virtual bites, corresponding to the synchronous contact time they have each week with students. Another need expressed from teachers was instruction on how to do virtual teaching and how to engage youth in these programs. Specifically, teachers wanted advice on how to implement curricula in an online format. They expressed a need for help with online recorded instruction videos, Zoom realtime meetings, access to curriculum, access to webinars on video production skills for teachers, and access to more interactive/dynamic teaching modules. One response reported "Teachers who participated in the summer intensive and professional development have requested we continue to offer regular Zoom events to expand their network and create opportunities to engage with colleagues from different grade levels, content areas, and districts." In addition to teachers' needs, families were requesting help to find activities and learning opportunities for their children.
Impact of the Pandemic on Program Implementation. In their responses to the impact that the pandemic has had on their program implementation as discussed above, PIs de-scribed a number of challenges that were quantified through coding as shown in Figure 2. PIs were asked to describe in more depth the specific challenges that they anticipated in continuing their programs virtually. Responses were coded as described in Methods, and the number of instances quantified. A total of 61 PIs responded, with 53 describing specific challenges, and four programs replying that they were already conducting their programs virtually with plans to expand in 2021. Six of the responses were not codable. Of the 53 challenge responses, the number of instances in each of the six secondary categories were Engagement (21 instances), Access (17), Logistics (13), Connections (12), Grant Requirements (8), and School-based (Figure 4). In the Engagement category, maintaining student engagement was the top response (12 instances), followed by teacher engagement (4), burnout (3), and recruitment of participants (2). With respect to student engagement, PIs commented that "It is much harder to engage high school students in conducting research when we only have their attention and time virtually." Another program holds a large student research conference each year; the PI expressed concern about how to "replicate the excitement of students at the actual event." With respect to teacher engagement, one PI commented "I'm not sure how to engage teachers and request more time, when already so much is asked of them." Access challenges were a consistent theme throughout this survey. In response to challenges going forward, both access to Wi-Fi as well as the equipment necessary to participate online were mentioned a total of 17 times. Access issue comments pointed out that this challenge impacts rural and urban schools, and is specifically a difficult challenge for economically disadvantaged populations. For example, one PI stated: "Some students in our rural community are simply not able to participate because of unstable or inaccessible [Wi-Fi]." Another mentioned that "Most of the youth in low income schools can only have computers and [Wi-Fi] at school." Student access to sophisticated laboratory gramming. A total of 53 PIs described specific challenges that that anticipated as they adapted their programs from in-person to virtual. Responses were coded as described in Figure 2. equipment was also mentioned (three instances), specifically related to students working in research laboratories during summer and academic year internships. As virtual learning continues, these experiences will be among the most difficult to replace. Some PIs as noted above are using computer-based simulations, but this does not replace the on-campus experiences for the students, or the actual in-lab research that is the focus of many high school programs.
PIs mentioned several tertiary categories in Logistics. Recruitment was mentioned four times, with comments such as: "Recruiting teachers may be an issue. After spending 40+ hours engaged in virtual teaching, they may not wish to participate in a virtual professional development program." Comments also included student recruitment: "The challenge we face is recruitment into the program. Students want to be on campus with a teacher and peers." PIs suggested that more planning time was required to adapt to the virtual environment (three instances): "Added time [will be] needed to successfully plan and deliver programs in the face of additional institutional demands due to [the pandemic]." Three PIs mentioned potential legal issues when involving minors: "there are legal/protection of minors concerns about students and adults connecting online." Finally, two PIs mentioned that their programs lacked sufficient staff to adapt their programs to virtual learning: "Staffing resources to build new curriculum is my biggest concern." Many of the outreach partnerships rely on important connections among the participants including students working in teams on research projects; students and teachers working together in university research laboratories; and researchers visiting classrooms to give presentations. PIs mentioned the negative impact on these connections 12 times, with student internships in labs (6), researchers as mentors (5), student-researcher collaborations (4), and student-student interactions (4) as the top categories. It should be noted that several PIs mentioned more than one challenge in their responses. One PI noted that "Students…lose the secondary and tertiary interactions of being integrated into a laboratory, project, and campus environment." Another stated: "Our students really benefit from contact with mentors that we use in our Academy. It is hard to give them this time virtually." One PI mentioned several connections that would be negatively impacted: "Our programs have heavy focus on near-peer mentors, in-class activities, and small group discussion." Finally, one PI pointed out that "Building esprit de corps between students and among students and mentors may be difficult." Several PIs pointed out that grant requirements presented specific challenges when attempting to adapt programs to a virtual platform. Evaluation was mentioned five times, with comments such as "Additionally, we anticipate challenges focused on our ability to complete our classroom implementation evaluation." Four PIs mentioned that they anticipated difficulty in meeting obligations of the grants that funded their programs.
Three PIs indicated no challenges since their programs were designed specifically for virtual learning in an effort to broaden their audiences. For example, one PI noted: "Our new program (in development) was actually designed to provide virtual role models and mailed materials to underserved children who do not have access to STEM materials and human resources." Another group responded: "Because [our program] leverages game-based and virtual learning resources, we are strongly positioned to continue our research programs virtually."

Equity Challenges in Adapting Programs to a Virtual
Platform. The primary challenge for many PIs during the current pandemic has been in reaching underserved populations. Many of the programs have by necessity turned to the internet, causing concern expressed by 63% of the PIs about inequities in reaching these populations. Only four responses indicated no equity issues and they were working inside institutions that dealt with the issues for them. As one PI stated: "Digital inequity is alive and well. We are reaching out to partners that are invested in closing that as much as possible such as Microsoft and also local entities that have been fighting hard in this arena for a while." Access to STEM content and learning opportunities is often dependent on reliable computers, internet, and software, each of which must be present to allow fully engaged participation. Students in these populations frequently do not have access to stable and reliable Wi-Fi, even if they have smart phones or have been provided with a laptop.
Although the focus of this question was to find out if programs had new or innovative approaches to overcome the equity challenge, over 60% of the programs are still in the mode of recognizing the challenges, with no answers yet for how to overcome them. Among the programs with solutions were several interesting approaches by PIs to reach underserved students. Almost half of the solution-oriented responses clustered strongly around providing computers and hot spots and cell-phone friendly content. One PI responded, "We have thought about bringing our MobileLab to a central site (school or library parking lot) to provide internet access." Another PI talked about "using programs for distributing virtual content that work on cellphones, recording sessions for viewing at any time, technical support for connecting, virtual office hours, one-on-one support." Frequently laptops and connectivity are provided for all participants.
Underserved populations are frequently suffering from food insecurity and are missing basic needs for survival. STEM enrichment is not their primary concern. This theme ran through both the challenges and the solutions where PIs mentioned creative ways to bundle STEM materials with survival supplies. Examples of strategies are seen in the following excerpts from PI comments: ". . . I'd definitely say our focus is on providing food, technology, and cooling needs to scholars and families." "We have encouraged partners to leave SEPA books at food banks or other entities reaching undeserved children.
". . . partnered with a school feeding program to send home science kits at meal pick up sites." "…delivering food and science education packet pick up from local libraries and the Piggly Wiggly." To overcome the lack of technology, some PIs have resorted to drop-offs of materials and hands-on science kits to homes. Another designed a program that can happen off line where families can share in the results. Another offered alternative options to on-line resources by switching from using 3D printing to household materials to model proteins. Although all of these strategies are working to reduce disparities in access to online resources and programs, we are still in need of creative solutions and ways to cover the costs of these solutions that are covered by entities other than the populations lacking sufficient resources.

Innovative Strategies to Capture the Power of the Inter-
net. Many of the STEM outreach programs to the K-12 community depend on direct contact with students and teachers, including programs such as after school clubs, summer science camps, teacher workshops, scientists and engineers bringing hands-on activities to classrooms, and on-campus student research internships. As highlighted over and over again in the survey responses, these types of programs have either been entirely shut down or PIs are working to adapt to a virtual platform. In the final two questions of the survey, we asked PIs to think about how the use of distance learning can lead to new and more effective strategies for outreach/ partnership programs. Admittedly our survey was administered early in the pandemic when many programs were working furiously to determine how they could adapt to the new virtual domain. However, several quotes from PIs underscored their belief that virtual learning is here to stay, and the challenge for all of us is to reimagine ways that the internet can be a positive component of educational programs: "…It will be a dominant force in education for the foreseeable future. It has opened up great opportunities for parents and teachers in homeschooling situations." "…since our program offers a range of authentic activities outside the laboratory that might be done on-line, the on-line version of our program may be attractive to some schools nationally that otherwise could not gain such experience for their students." "I think it will emphasize how important a feeling of community and closeness is to education and helping students succeed." "A student, who may sit/work silently during a hands-on experiment or in a large classroom, may engage more readily in chat and thus provide insights into their learning. With certain parameters set around chat, it could offer a lot of background data for evaluation" Importantly, while the inequity issues inherent in technological approaches have been discussed above, there are significant advantages to virtual/distance learning that a number of PIs pointed out. The most common response was the increased access to diverse audiences using distance learning. PIs adapted programs quickly to integrate synchronous and asynchronous learning for both teachers and students. While in-person programs are often limited by geography and cost, turning to a virtual platform reduces or eliminates the cost for the participants, and reaches a much broader audience. For example, several PIs mentioned opening up their teacher programs to rural communities that previously would have had to deal with barriers such as travel and costs. One area that was mentioned in response to several of the survey questions, and repeated here as an approach that might change how we think about our outreach programs is the delivery of hands-on kits to students using public spaces as drop off points (food centers, libraries, etc.) or mailing kits to students. This could be a very effective way to reach students who might not have access to a university or informal outreach center. For example, one PI commented that "Virtual lessons are feasible as long as you can package and deliver materials necessary for the lessons to students" and "It provides a broader array of opportunities to students in rural areas who might be too far away from a college." Finally, there was definitely a feeling of not only commitment to using the internet effectively, but PIs exhibited excitement about the challenge to define innovative ways to capture the power of technology:

"I love that virtual opens up the world. Our virtual camps have kids from across the country. While virtual adds real barriers, it knocks down others."
"This has the potential to make STEM outreach a national program that is not limited to proximity to major academic health science centers."

CONCLUSION
In the current study we report the results from a survey of PIs that highlighted how they tackled the challenges of partnering with K-12 students and teachers in a virtual world that they were presented with during the global COVID-19 pandemic. With no options for in-person programming, PIs had to adapt their partnership to a totally virtual platform. A total of 61 PIs described the impact on and adjustments made in 115 programs. Although a variety of challenges were acknowledged and discussed above, the overwhelming response by PIs was their commitment to trying new strategies and approaches to reach their audiences. Often the responses were not only positive, but enthusiastic. As one PI stated, "The gift out of all this is that we DO have to reimagine and implement new strategies! The mission never changes -this is how we grow." Admittedly, responses to the survey were submitted early in the pandemic -at a time when all we knew was that schools had been completely shuttered. What is still unknown is what K-12 education will look like in 2021 and beyond, and how we as STEM experts can fit into this new landscape.
It must certainly be acknowledged that the overarching challenge for all programs was confronting the inequity of access to reliable internet and equipment resources of underserved populations. Working with school districts, outreach directors worked diligently to overcome this challenge by setting up hotspots, handing out computers to all students, and mailing or delivering hands-on kits to homes. With these efforts and challenges in mind, PIs continued to adapt and innovate, with a variety of approaches that have the potential to positively impact how partnership programs are implemented in the future.
Many of the successful innovations that PIs have used have contributed in a positive way, and could definitely alter how we partner with students and teachers. Several examples of these successes were highlighted in the survey: teacher PD presented virtually, at least in part, breaks down geographic boundaries and reduces costs of housing and travel. Placing portions of student internships online such as study skills, required university certifications, and initial instruction on research skills and data analysis reduces time on campus. These modules could be asynchronous, reducing demand on faculty time. Developing totally virtual student labs, although losing valuable connections, allows students in non-university areas to participate. In addition, certain projects could become global research projects that would actually promote important interactions (albeit virtual) among students from varying geographic locations. Development of online courses for teachers would again eliminate the requirement for teachers to travel, and could reduce faculty time by using a mix of asynchronous and synchronous approaches. Online curricula have already been started by several groups, and realizing the need to assist teachers to actually implement the curricula in the classroom, PIs have added teacher support to their PD programs. Several groups have used historic online data to introduce students to data analysis and presentation. Finally, several groups have developed online games to provide innovative online learning options for students.
The resiliency in these STEM outreach professionals during this time of crisis is heartening, and the future beyond COVID-19 has been enriched by their creativity and energy. New ways of doing things will certainly be integrated into our outreach programs, and will be interwoven with the faceto-face strategies we know work well. In the end, STEM outreach may be better for the struggles that COVID-19 has precipitated. Necessity is the mother of great invention, and good ideas spread like the virus in this community of professionals, now better connected than ever.

ASSOCIATED CONTENT
Supplemental material mentioned in this manuscript can be found uploaded to the same webpage as this the manuscript.

Corresponding Author
Jennifer A. Ufnar, Ph.D., Department of Teaching and Learning, Vanderbilt University. 230 Appleton Place, Nashville, TN,

Author Contributions
The manuscript was written through contributions of all authors. All authors have given approval to the final version of the manuscript.