The Health Sciences and Technology Academy: Re-Imagining Programmatic Delivery During the COVID-19 Pandemic

: Out-of-school time academic/STEM programs provide educational enrichment to a myriad of student populations with some designed to assist those underrepresented and at-risk who desire to purse post-secondary studies. One such program in West Virginia (WV) is the Health Sciences and Technology Academy (HSTA), which provides hands-on, in person and engaging educational enrichment with the intent to increase the college going rates of Appalachia’s most vulnerable youth. In March 2020, HSTA key personnel encountered the task of redesigning program delivery due to the im-mediate shutdown of all in person operations resulting from the novel coronavirus, SARS-CoV-19 (COVID-19) pandemic. This paper discusses the programmatic changes of program delivery during HSTA’s key program components—the Summer Institute and the Community Based Programming. This paper also utilizes repeated measures one-way multivariate analysis (MANOVA) and paired t-test analysis to examine participants’ perspectives on learning, enjoyment and satisfaction of the aforementioned components prior to and during the COVID-19 pandemic.


INTRODUCTION
For 26 years, the Health Sciences and Technology Academy (HSTA) of West Virginia (WV) has provided academic outreach to African American, first generation college, financially disadvantaged and rural Appalachian high school students with the intent to "home grow" health science/science, technology, engineering and mathematics (STEM) practitioners to serve in their local communities. Due to the Coronavirus Disease of 2019 (COVID-19) global pandemic, local and national educational entities faced the daunting task of providing alternative ways of teaching and engaging students in learning at various levels (i.e., elementary, secondary, college) while protecting them from the COVID-19 virus. As such, the purposes of this paper are to examine HSTA's key program components: Summer Institute and Community Based Programming while comparing the traditional program to the virtual program created and disseminated in response of COVID-19. We will discuss pitfalls, learning curves, lessons learned, adaptations and adoptions of components as well as retained and eliminated elements for future program implementations. Finally, we present analyses of student perspectives on learning, enjoyment, and satisfaction prior to and during the pandemic.
In March 2020, WV schools closed due to the pandemic cancelling all sports, after school activities and in person contact. To ensure that students continued to receive instruction, school officials resorted to various virtual learning management systems such as Blackboard Collaborate. However, due to WV's rural landscape, some students had limited access to the internet, further highlighting social and educational inequities (Allen and McCormick, 2020). For the most part, WV schools utilized Schoology and Google Classroom along with Zoom, Google Meets, and Teams for online learning and allowed for teachers to print and mail packets and worksheets to students and for students to return completed materials when virtual options were not feasible.

OVERVIEW: TRADITIONAL HSTA
Summer Institute. The first HSTA experience for a new participant is a Summer Institute (SI) camp, which is usually before their first year of high school. These camps convene during the months of June and July. An important mission of HSTA is to nurture intellectual growth through innovative and challenging, experiential (problem based) learning opportunities designed to develop and build skills for school and college success within a mentored collegial environment. Organized by grade level, the purpose of the SIs is to create an environment that allows students to mature as they matriculate through the experiences, moving them from a broad scope of topics to more focused subject matter. By the time they reach their senior year, they are prepared to take a credit-based college math course and graduate with the skill sets for mastery in college. Near peer mentors, teachers and professors create an atmosphere of acceptance and success, which establishes the tone for the camp. Traditionally, HSTA students attend SIs on a WV college campus (i.e., West Virginia University, Marshall University, Glenville State College, WVU Institute of Technology, and Potomac State College) exposing them to the rich experiences of college life through research and mentoring opportunities. The SI camps are 1-week (i.e., Freshmen, Sophomores, and Juniors) to 3-week (i.e., Senior) experiential learning opportunities. All camps match small student groups with near-peer mentors, college students, who are their campus and dorm leaders and supervisors. Mentors are often former HSTA students and help bridge the gap between the adult supervisors and students. Mentors receive training in leadership, homesickness, team building, etc. They also serve as math and science tutors assisting participants to complete their final presentations delivered at the end of the camps. Year 1 SI Camp. The first HSTA experience for the majority of rising 9th grade participants is the Freshmen Camp. For many, this is the first time away from home and often the first encounter in a laboratory setting. The curriculum provides exploration of various STEM experiences broad in scope and content as well as fun and engaging. This camp integrates study skills for high school success, multicultural education to build compassion for different experiences and perspectives, math skills, and an introduction to HSTA community research projects.
Year 2 SI Camp. The Sophomore Camp and second campus-based experience is for rising 10th graders focusing on students who have had one year of the HSTA community club experience (see discussion in HSTA community club experience), have completed and presented a research project, and come to camp with a readiness to study more indepth subject matter. Many, but not all, have attended the Freshmen camp. The curriculum design provides subject focused, fun, lab enriched experiences (i.e., Forensics, Coding, Engineering) with an emphasis on using ascertained skills to develop the community research project. This camp builds on study skills, multicultural education, leadership skills, math skills, and ACT preparation.
Year 3 SI Camp. The third university-based experience is the Junior Camp, a National Institute of Health (NIH)-Science Education Partnership Awards (SEPA) funded program. Designed for rising 11th graders, students work in small groups to study a specific subject matter, thus, no two groups are alike allowing participants to become the experts on their topics, translating their experience into potential proposals for community research projects for the upcoming school year. They share these proposals with their peers during the end of the camp presentations. Students also explore potential STEM and health science college majors/careers, multicultural education, fundamentals of statistical analysis, and community based participatory research principals.
Year 4 SI Camp. The Senior Camp is the final campus experience for rising 12th graders who have attended at least one, or perhaps two, of the previous camps. As students matriculate through the program, they are ready to spend three weeks on campus where they enroll in a credit-earning college math course, take a college preparatory course, and/or enhance their ACT/SAT skills for final testing. This extended time of living on campus, freedom, and responsibility, albeit, supervised, provides the students with first-hand knowledge of what to expect when they transition to and how they can successfully handle college. Three weeks of dorm life and class work builds confidence in how to handle struggles, seek assistance, and achieve success.
Teacher Pre-Camp Experience. Prior to the student camps, HSTA teachers attend a one week on campus training for professional development. They work with faculty and staff to prepare for students as they learn and adapt the content, experiments and activities.
At the close of each camp, students make a final presentation of what they learned and how to utilize the ascertained information to create a community-based research project (discussed in the HSTA community club experience).
Programmatic Changes to the 2020 Summer Institute. To protect participants from exposure to the virus, HSTA made programmatic adjustments at all levels. After suspension of program activities in March, HSTA Community Research Associates (CRAs) and Field Site Coordinators (FSCs) worked diligently to transform the SI curriculum and activities into a synchronous and asynchronous distance-learning platform (Bernard et al., 2004;Beyth-Marom et al., 2005). The CRAs primary role in the program is to design curriculum to enhance students' understanding of health and STEM concepts, whereas, the FSCs are tasked with assisting all aspects of program delivery at the local and statewide levels. They also have a unique role in recruitment and retaining students who are at a high risk for attrition (McKendall et al., 2021). HSTA's four CRAs have a combined 75 years of classroom experience and over 50 years of HSTA experience. In conjunction with the LGB/JGB, SIAB, FSCs, teachers, staff, and university staff, the CRAs developed objectives to provide the most nurturing environment for students without the risk of physical contact.
The restructuring process began almost immediately in late March in preparation for putting protocols and standards in place to host interactive and engaging SIs held during the months of June through the end of July. By May, it quickly became apparent that worldwide, everyone would encounter a major life-style adjustment. Thus, HSTA forged ahead in the planning process of creating a remote 2020 Summer Institute as well as the 2020 Community Based Program. The transition of a completely in person, on-campus summer educational experience for upwards of over 400 9th through 12th grade youth into a completely remote off-campus learning platform was daunting to say the least. One of the first phases of transition was to determine if HSTA participants had access to internet services at the local level. Thus, HS-TA's CRAs called upon local staff, namely FSCs and HSTA teachers to determine the landscape of internet accessibility among HSTA students and trouble shoot at the local level. This involved identifying local internet hot spots and working with school staff to allow students to keep computers/ iPads over the summer to measure the feasibility of a remote summer program. An important objective was to provide ample learning experiences that participants could share with siblings and parents while maintaining program expec-tations, program consistency and ensuring that participants continued to build on their skills for success. These endeavors required the CRAs, HSTA staff, teachers, and affiliated faculty members to collaborate in formulating and implementing the processes of virtual program delivery. HSTA held weekly meetings with field and office staff to provide a statewide view of what was happening in the HSTA program as it related to the communities, students and teachers. This allowed the HSTA team to respond pro-actively to real-time problems with viable solutions.
A vital and first question asked was "How many students could the program serve?" Ideally, HSTA desired to provide the opportunity for attending camp to all students. Prior to 1998, when the program was a much smaller organization, this opportunity was open to all students; however, it would not be possible to do this for the virtual camp without all HSTA teachers. When on campus, there are 8 to 10 teachers attending each camp and leading groups. To implement successful camps, we quickly realized that all teachers would need to assist with teaching during the virtual camp. Thus, prior to opening camp registration to participants, all of the 62 HSTA teachers were hired using the funds normally used for housing and travel to teach the virtual summer camps allowing maintenance of small and personal group settings. All of the HSTA university/college affiliates were contacted to ascertain interest in our idea and if they could adapt. They were able to quickly modify and arrange for faculty to participate. In the meantime, recruitment took place and all students that applied were accepted.
In order to streamline and manage the 2020 SI experiences, the HSTA team combined the 9th and 10th grade camps working with four SI university/college Camp Directors to create an online experience. HSTA and SI Camp Directors worked via email to develop lessons and order materials. This restructure allowed the four colleges to combine their strongest components for virtual delivery to three times more students and teachers than the traditional on campus HSTA programs. CRAs combined all lessons into one notebook and trained HSTA students on the content and experiments. Students attended these virtual camps with their regular HSTA teacher. This technique allowed for a stronger bond and nurturing experience since half of the attendees had a prior connection with a HSTA teacher. For many of the sophomores and all the incoming freshmen, this was their first camp experience so the HSTA team decided the junior and senior camps should remain separate.
The team created teacher and student notebooks and corresponding kits for each camp to provide students with opportunities to experience "hands-on" STEM activities through online meetings and home experiments. HSTA designed unique educational kits for each SI group, 9th/10th graders, 11th graders, and 12th graders. SI Kits included HSTA Teacher/Student Summer Notebooks articulating the expectations for the summer program, a schedule of virtual activities that students needed to attend, at-home experiments, activities to complete, as well as equipment and supplies to complete these experiments. HSTA teachers and students received packages from HSTA staff at drop off events and/or through the mail. The team adhered to all COVID-19 safety protocols during drop off events.
The SI teacher training workshops were quite different and minimal in scope. During normal HSTA operations, teachers spend a week with their faculty, prior to students arriving on campus, but the format of the virtual program proved this format infeasible. As an alternative, teachers attended two online meetings with CRAs to discuss the notebook materials and to formulate an action plan to teach the lessons. HSTA teachers along with FSCs organized all of the club meetings and guided students through the lessons and experiences.
The combined 9th and 10th grade camp included approximately 312 participants and covered a vast array of subject matter, some of which were: • the importance of handwashing through a Glow Germ mini lab • exploring statistics of the coronavirus pandemic through analysis activities • chemiluminescence and bioluminescence experiments as well as a citizen science activity tracking of fireflies in WV • how genetics can influence addiction • parts of the brain and how the brain works under the influence of drugs, and • collecting cloud data using the globe website (https:// www.globe.gov).
The Camp Directors at the four different WV universities/colleges worked with their colleagues and staff to prepare the lessons and hands on activities for HSTA students. In addition, the universities/colleges created recruitment activities so that HSTA students could browse websites and request information relevant to the university/college.
The Junior camp hosted approximately 112 students. Unlike the traditional 11th grade camp, all students experienced the same lessons/activities in the following topics: • the role of zebrafish in research • experiential design and the scientific method • heart anatomy and diseases • lung anatomy and diseases • infectious diseases • prevention of disease provide potential research questions to students. The CRAs design, implement and deliver this workshop as well as assist students and teachers throughout the academic year.
During the school year, teachers are required to maintain a report reflecting student attendance, participation in at least six hands on activities and encounter with at least one guest speaker a semester. Recording student attendance ensures maintenance of a 70% or greater attendance rate per semester. HSTA teachers implement hands on activities and recruit guest speakers for their HSTA clubs. Hands on activities are STEM and/or health science focused with the objective to engage students in experiential learning. Guest speakers are community members and STEM/health professionals who are experts in their field and talk with students about careers, college, and often they bring a hands-on activity that relates to their topic.
In addition to the hands-on activity and guest speakers, HSTA teachers guide students through their annual community-based research projects. Students explore issues in their community and select a topic based on their own observations. CRAs approve all research projects to ensure adherence of ethics and lab safety guidelines. The CRAs validate the studies to determine appropriateness of human and non-human data collection as well as approve the general research design to ensure accuracy in data analysis and conclusions. During the first semester, students design and develop their project by identifying community observations, completing a mini literature review on the topic, writing their research question, variables, and hypothesis. Finally, students describe their procedures including all survey and recruitment materials.
Participants dedicate the second semester to collecting data from school peers and community members. As they finish the semester, students complete analysis, draw conclusions, and present their final presentations first at a local practice symposium then a state-wide symposium. Statewide symposiums are held on WV campuses where students spend half the day presenting and watching peer presentations and the other half exploring the college campus. This exploration includes hands on activities and recruitment talks with the university/college personnel. The FSCs also perform a fundamental role in ensuring the local regions have all of the needed materials and supplies for student success and completion of activities and projects. In addition, they recruit new students and teachers, guide the local governing boards that govern the local region as well as communicate information between the local governing board and the state level board, known as the joint governing board. Table 2 provides a synopsis of comparisons between the traditional and restructured Community Based Program experiences.
Programmatic Changes to the Community Based Programming. Important structural components to the HSTA • viruses and antibody tests • inequity in health in WV • the relationship between movement of people and the spread of diseases • the needs of preschool children and the needs in their community • how to eat and drink healthier items • the relationship between mental and physical health • how exercise impacts lung health and aerobic capacity • understanding exponential growth The Junior camp faculty worked individually to prepare their lessons and activities based on their 'normal' topic, with the majority incorporating a COVID-19 element for the virtual experiences.
During the Senior SI camp, HSTA leadership decided to provide an online course for 36 participants to prepare them for the SAT exams through Kaplan. In addition to attending the on-demand Kaplan program and receiving Kaplan materials, students had a college graduate mentor that led them through a series of Kaplan math lessons. Classes were three hours long and included lessons, time to work through problems, and baseline and post SAT tests. Community Based Programming. HSTA's Community Based Programming is traditionally OST consisting of in person after school meetings where students and teachers engage in hands on activities, guest speakers, and receive training in how to prepare a community-based research project. HSTA students in 9th -12th grades spend four years in the program where they must complete two of the four SIs, attend 70% of the after-school meetings, complete a minimum of 75 community service hours by their senior year, and complete and present an annual community-based project.
After the summer program, HSTA teachers gather at an annual Teacher Professional Development Workshop for training in Community Based Participatory Research Principles during the fall prior to the academic school year. HSTA convenes these workshops on a yearly basis to provide teachers with the principles that incorporate an understanding of how to teach research to students. As such, teachers receive training on the preparation of hands on activities that engage students with STEM and health sciences, the recruitment of community members to serve as guest speakers, guidance of students through the research process, and assisting students in preparing and presenting their annual community-based project. Teachers receive a resource notebook containing necessary paperwork, polices, and procedures of HSTA, examples of hands on activities and types of guest speakers, workshops, and activities to help them teach research, and program are providing students with "HSTA Club" experiences designed to expose them to "hands-on" STEM activities, peer to peer and student to teacher interactions, and active involvement in their community through community service hours, and research projects geared towards improving health in their community and around the state. Since these activities were no longer possible to implement in person, HSTA leadership had to discover innovative ways to deliver the program remotely while attempting to retain those critical components. For the 2019 HSTA Freshmen class, this resulted in their experiencing only 6 months of direct contact and the 2020 incoming class having no faceto-face contact with HSTA personnel.
HSTA CRAs created a Problem Based Instruction (PBL) statewide curriculum for delivery to participants, which was iterative, sequential, and steeped in hands on activities. According to Miller and Krajcik (2019), PBL uses a "driving question that is meaningful to learners." This question ignites student exploration and sustains motivation across time," resulting in an end product (e.g., writings) that answers a pivotal question culminating in "a learning sequence" which connects to the community (Helle et al., 2006;Miller and Krajcik, 2019). Essentially, HSTA students across the state were engaged in curriculum designed to build research skills resulting in a final product-their community-based research project.
As previously mentioned, HSTA teachers attended a professional development workshop in September of 2020 to receive training in the revised attendance policy, statewide curriculum, and lesson plans along with resources, strategies, and materials for conducting HSTA club and receiving resources and strategies to assist HSTA students in completing their 2020-21 research projects. In the Fall of 2020 HSTA students received a notebook containing a welcome letter, information about changes to the research project, a list of community service opportunities, the student contract, activities to help prepare them for the Collaborative Institutional Training Initiative (CITI) and Institutional Review Board (IRB) trainings, and other homework assignments. In addition to the notebook, students also received another kit with all the necessary materials to complete all HSTA related activities. The "Fall 2020 HSTA Student Notebook" contained activities to help students prepare for the CITI/IRB trainings pertaining to the Belmont Principles and Laboratory Safety Rules, etc. Other teaching tools were Laboratory Safety Activities, directions for at home experiments demonstrating chemical reactions (e.g., making bouncy balls), collecting sleep data, and literature review exercises. Other learning activities included an egg osmosis experiment and students utilized microscope skills by examining paper money, different types of fibers (e.g., wool, cotton, etc.), different types of hair samples, and different spices (e.g., salt and sugar). The program provided most of the materials, including the pocket microscope for those that did not attend summer camp.
Attendance. As previously mentioned, in the traditional program, students attend at least 70% of in person HSTA meetings to satisfy program requirements. The CRAs transformed this requirement into a point system; whereby, participants earned points for attending Zoom HSTA Club meetings and completing at-home activities allotting for 70% attendance. For each meeting attended and assignment completed, participants received 2 or 1 point(s) for club credit. If students left meetings before the allotted time or did not complete assignments, they earned only ½ of a point. Essentially, the point system, for the virtual program, determined if participants satisfied attendance policies.
Hands-on Activities. One type of experiment in particular was "the bouncy ball experiments" which instructed students on the science of polymers. Participants were given detailed instructions and a list of materials (from their HSTA kits received during Fall 2020), to make three bouncy balls (i.e., Control Bouncy Ball, Experimental A Bouncy Ball, and Experimental B Bouncy Ball). The students recorded  components of the experiment (e.g., amount of each material remaining after making each ball), where they hypothesized which ball would bounce the highest (A or B), explain why they chose which ball bounced the highest and provide the procedures they followed to determine the best way to measure the bouncy ball heights. Students also entered their data into the REDCap database (Harris et al., 2019), and the CRAs created a statewide graph of the Bouncy Ball Experiments. HSTA disseminated the graphs to participants, and they explored the graphs by answering questions such as "What type of graph is this?" "What other types of graphs are there?" "How can we create this type of graph in Excel?" "What does this graph tell us?" and "What does it not tell us?" HSTA students also participated in a mini research study about sleep. The CRAs created a mechanism by which HSTA students collected "sleep data" on themselves and an adult in the household during the Fall of 2020 then used the data in lessons to learn about the research process.
Guest Speaker Experience. Due to safety protocols, outside speakers were not permitted into the schools, thus HSTA leadership partnered with WVU Extension in which four pre-recorded online lessons on a concept called "My Hometown is Cool" were posted virtually. These lessons included discussions on what makes WV communities cool, what a state seal means, what is community capital, and what things would they change about their community. WVU Extension also shared information regarding their statewide competition with HSTA students. HSTA students were able to submit a three-minute video on how to make their community 'cooler.' The competition rewarded students with a small grant to translate their ideas into reality.
Community Service. Traditionally students volunteer at food banks, hospitals, blood drives, etc. to earn community service hours. HSTA suspended in person community service from May 2020 until further notice; however, leadership did not adjust the community service requirements of 75 hours, rather providing online opportunities. Every month students could join up to 4-6 zoom meetings steeped in career awareness and college readiness skills as well as access to near peers who are attending college. Guest speakers included medical, pharmacy, and dental rural residents from WVU Community and Rural Rotations Support Program (CARRS), WV college representatives who discussed attending their specific college and what classes to take in college including discussions of AP classes, financial aid, etc.
After attending the optional Zoom meetings, students were required to submit short reflections of what they learned to earn community service hours. Depending on the length and comprehensiveness of these workshops, it accounted for 1 to 3 hours of community service towards the allotted requirement. HSTA also offered students three hours of community service if they completed their CITI training. Finally, the HSTA Website (www.wv-hsta.org) and the Student Notebook also provided a list of innovative ways to satisfy community service hours, which ranged from calling a shut-in lonely neighbor to starting a virtual club.

Community Based Research
Projects. An important component that students must complete prior to beginning the research projects is the CITI program and IRB trainings, The CITI and IRB trainings prepares participants to conduct their HSTA research project. After these training are completed, participants were ready to begin the research process. However, a major transformation to the community-based research projects is the metamorphosis from a full year project process to half-year virtual projects completed in the Spring 2021 academic year. The HSTA teachers provided limited research project options to participants. The primary goal of the new virtual projects was to assess the health of West Virginians.
During Spring 2021, students completed a second notebook interactively explaining the research experience. Students formed groups of three or less to work on projects related to prevalence, collecting data through online REDCap surveys. Research topics included the following: self-esteem, stress, general health questions, sleep, energy drinks, nutrition, physical activity, My Hometown is Cool, Photovoice, and COVID-19 vaccinations. All projects captured anonymous responses from community members including demographic questions about gender, ethnicity, age, education level, and zip code.
Through mini research lessons, HSTA teachers instructed students on setting up their projects by first discussing project selection for each student and/or group. Afterwards, teachers followed up with a key question for students/groups regarding their selected topic: Is there a problem in your community? In Fall 2020, students completed a mini literature review on the importance of sleep utilizing this exercise to complete a mini literature review for their project topic. A requirement for completion was the inclusion of at least two facts from four different reliable sources and the use of APA format for citations. In the meantime, HSTA teachers continued lessons using the sleep data collected in Fall 2020 to practice writing research questions and identifying variables and hypotheses. Afterwards, students completed research worksheets on their project as they reformulated if necessary, the research question, variables, and hypothesis. Throughout this process, participants engaged with hands on activities in preparation for writing the research procedures (i.e., write the procedures on how to make a peanut butter sandwich). Theoretically, this is an easy task; however, they were required to write the procedures such that the instructions would result in the peanut butter sandwich made the same way regardless of who made the sandwich. This exercise helped students give details in how they would carry out their project and collect data.
Continuing with the sleep data, students practiced graphing, by examining the research question, variables, and hypothesis to determine the appropriate data analysis procedures, and interpretation of results. Then students completed data analysis, graphing, and interpretation of their own data. Finally, students joined together for an online regional symposium over zoom to present their findings. Prior to presenting their research projects, participants viewed an opening pre-recorded video from the HSTA Director after which they went to break out zoom rooms for their presentation. HSTA teachers and community members scored projects for completeness and provided feedback. After presentations, students completed their annual evaluation.

and 2021 Community Based Student Perspectives
of COVID-19. Traditionally, HSTA collects formative and summative data from participants to assess program effectiveness and impact allowing for continual assessments across program platforms (i.e., Community Based HSTA Clubs and Summer Institutes). HSTA data collection occurs via the REDCap (Research Electronic Data Capture) database system hosted at West Virginia University. The secure, web-based software platform supports data capture for research studies (Harris et al., 2009, Harris et al., 2019. We capture both qualitative and quantitative data in these surveys to understand if there were differences in participants' satisfaction, levels of learning, and enjoyment of HSTA's online programming. We utilized The IBM SPSS Statistics (Version 26) software package and QDA Miner Lite, by PROVALIS RESEARCH to code the quantitative and qualitative data, respectively (Provalis Research, 2020).

Participants' Learning Perceptions During the Community Based Programming.
Each year, HSTA disseminates online surveys to participants examining their HSTA and/or school experiences. The May 2020 and 2021 surveys (i.e., Community Based Survey) specifically asked participants how the COVID-19 pandemic affected their HSTA experience, in which we examine participants' perceptions of online programming.
The Community Based Survey is a yearly survey disseminated to HSTA participants at the end of the academic school year. This formative survey queries participant's perceptions of the program's impact and effectiveness in providing services. In order to understand the impact of the COVID-19 pandemic on student learning, we wanted to know if there was a difference in reported learning across three time intervals representing prior to and during the COVID-19 pan-demic. Thus, we conducted a longitudinal assessment of the Community Based Survey highlighting two questions on this yearly instrument for 2019, 2020 and 2021: • Overall, on a scale from 1 to 5, how much have you learned in your HSTA Club meetings?
• Overall, on a scale from 1 to 5, how much have you learned from working on your HSTA Project?
We conducted a one-way repeated measures multivariate analysis of variance (MANOVA) to analyze the aforementioned questions. In 2019, 100% of the program was in person; however, in 2020, approximately 76% of the program was in person and in 2021, 100% of the program was virtual. Respondents included in this study answered key questions across all three years for this longitudinal study as well as those who agreed to have their responses used for research purposes yielding a 33.7% response across all three years (N=205) with approximately 27.3% males and 72.7% females. Only 30% of the participants indicated that their parents had completed college and 78.5% indicated that they received free or reduced lunch. Approximately, 63.4% of the participants identified as White, 23.4% African American, 11.2% Bi/Multi-Racial and 2% Other.
The MANOVA showed a statistically significant difference in the reported levels of learning in the HSTA club meetings and working on HSTA projects over the three separate time intervals (i.e., pre-COVID-19, 7 months into program suspension, and complete virtual program). In order to analyze results, we first examined Mauchly's Test of Sphericity for the conducted test on both learning variables. Mauchly's test looks at the equality of variances between the conditions to ensure that there is sphericity. We found that the assumption of sphericity was violated for both learning variables: learning during meetings, χ2(2) = 9.319, p = .009 and learning from projects, χ2(2) = 10.182, p = .006.
Thus, we concluded that there were significant differences between the variances over the time intervals. Since, sphericity was not achieved for either of the learning variables, we utilized the multivariate, repeated measure MANOVA since n (n=205) is greater than a + 10 (a is the number of levels for repeated measures (Maxwell and Delaney, 1990). Also, when sphericity is violated, an alternative to repeated measures ANOVA is the MANOVA (Laerd Statistics, n.d.).
In other words, there is a difference in the learning from working on projects over time-pre-, suspended-and full virtual programming-at the separate time intervals during COVID-19, which was not the case for learning from club meetings. Figures 1 and 2 provide estimated marginal means and the grand means for each year. Now that we know there is significant differences between learning from working on projects over time, we determined where these differences were by examining the pairwise comparisons as shown in Table 3. The results show that there is a significant difference between years 2020 and 2021 regarding participant's reported levels of learning from working on projects during these time spans.

Qualitative Reflections on COVID-19's Impact on the HSTA Experience.
We used coding frequencies on the qualitative question "Please reflect on how the COVID-19 epidemic has affected your HSTA experience," invoking multifarious comments from student participants. Of the approximately 185 coded comments, there were four major themes: The Overall Experience, Projects, Virtual Meetings, and School. See Table 4 for the total percent of coded cases for each category.
Below are key comments reflecting the variables mentioned above:

"The COVID-19 epidemic has affected my HSTA experience by making it difficult to work in my group, stay in contact with my HSTA teacher, and keep a schedule for working on my project."
"Covid-19 has made my last year in HSTA difficult and isolated. We had no in-person help nor experience, making it more difficult and taxing than previous years." Despite the difficulties, some students provided positive feedback regarding the overall experience.
"Although COVID-19 was very difficult this year, I believe my HSTA teacher did everything she could to make it feel as normal as possible. I have learned a lot this year and discovered different career interests even though we have been living in a pandemic." "It was challenging to complete my project, but overall the experience made me a better student."   The primary variable mentioned under the category "Projects" was that they were difficult, hard, and stressful to complete. Other concepts mentioned were that there were limited research opportunities, not enough time to complete the work, and a positive aspect was the mention of liking virtual presentations. Although some indicated that they liked to make virtual presentation, meeting virtually was not as accepted. For the "Virtual Meetings" category, participants' stated that the meetings were difficult to attend, hard to keep scheduled, and overwhelming, especially since they were accustomed to meeting after school during a normal HSTA program year. The normal HSTA after school clubs provided stability and continuity by which the students collaborated with fellow HSTA members and teachers in order to complete their research projects. As one participant so effectively states, It changed everything compared to what it was like freshman year, but as a group I feel we have a lot more than what most people thought we would. It wasn't easy not having meeting in person but it was definitely good to still have them to learn more.
The final category "School" specifically reflects the difficulty to keep good grades, the difficulty to focus, the impeding of learning, and the erosion of work ethic due to no in person schooling as shown in this student's comment, "The pandemic has affected my work ethic this year. Being at home makes it difficult to concentrate compared to a school setting." COVID-19's impact has impeded learning, decreased motivation and social interaction and increased isolation and anxiety for HSTA students while they go to school and when they participate in the program.

Perceived Enjoyment and Satisfaction of the Summer Institute: Prior to and During the COVID-19 Pandemic.
As previously mentioned, participants complete surveys to reflect on their HSTA experiences. HSTA participants complete the Summer Institute (SI) survey at the end of their one-week or three-week experience. Similar to the Community Based surveys, the SI surveys also examines participant's perspectives on impact and effectiveness in providing services. We wanted to know if there was a difference in the reported levels of satisfaction and enjoyment of the SI experience across two separate time intervals representing prior to and during the pandemic. Since, there is overlap in the various camps participants attendance across their HSTA experience; we conducted paired t-tests on data from respondents who answered the key questions in 2019 compared to 2020 and 2019 compared to 2021: • Overall, did you enjoy the Online HSTA Summer Camp? • Overall, how satisfied were you with the Online HSTA Summer Camp?
In order to conduct our analyses, we matched respondents by their unique identifier across the four separate HSTA SI camps (i.e., Freshmen, Sophomore, etc.) Table 5 provides totals across the matched camps. The majority of participants attended two camps, with only 6 participants having a SI experience for three years. Thus, we have a total of 168 and 27 matched respondents for 2019 compared to 2020 and 2019 compared to 2021, respectively. Respondents included in this study answered key questions across at least two camps, approximately 27.4% males and 72.6% females. Approximately, 57.7% White, 21.4% African American, 17.9% Bi/Multi-Racial and 3% Other.
Comparison means of 2019 to 2020/2021 participants' levels of enjoyment and satisfaction are shown in figures 3 and 4. There was a statistically significant difference between 2019 and 2020 enjoyment; t(167)=4.957, p<0.001, with a difference of .149. However, the enjoyment levels for 2019 and 2020 were not correlated (r = 0.060, p = 0. 437). A significant difference was shown in the reported levels of satisfaction in 2019 compared to 2020 with higher levels in 2019 compared to 2020, t(167)=4.274, p<0.001, with a difference of .393. Furthermore, the levels of satisfaction with the summer institute for 2019 and 2020 were also weakly and positively correlated (r = 0.238, p = 0.002). There was a significant difference in the means for enjoyment for 2019 compared to 2021; t(26)=2.126, p = .043, with a difference of .070. A statistically significant difference was also found in the means for satisfaction between these years as well; t(26)=2.865, p = .008, with a difference of .233. However, the satisfaction levels for 2019 and 2021 were not correlated (r = 0.055, p = 0. 785).
Since the total numbers (N=27) for the 2019 versus 2021 comparisons were small, we also conducted a power anal-  ysis in G*Power 3.1.9.7 software Buchner, 2007 and to determine the power of the achieved results. We were not able to conduct an effect size analyses on the enjoyment variable since 2019 data has a standard deviation of 0. We tested the difference between the 2019 and 2021 independent group means for satisfaction using a two-tailed test with a calculated effect size, and obtained an achieved Power (1-β err prob) of 0.79 as shown in figures 5 and 6.

DISCUSSION AND CONCLUSIONS
Although it has and continues to be a challenging task, HSTA innovatively reimagined the delivery of an in person academic enrichment program into a full virtual platform to Appalachia's most vulnerable youth. This paper addresses some of the overall programmatic changes (Fall 2020/Spring 2021 academic year) and challenges encountered during the Community Based and Summer Institute program components.
Virtual Format Features HSTA Will Maintain. After a review of the COVID-19 virtual format features of the program, HSTA leadership has decided to retain all of the virtual features with adjustments. HSTA will retain the notebook feature to serve as a research guide and organizer for club meetings. Prior to the beginning of the 2021 school year, HSTA will design notebooks to incorporate a research development component that will lead to discussions structuring the final research project.

Research Kits.
Another virtual format feature HSTA will retain is the use of research/hands-on kits for the club experience. HSTA will design and create kits for use during the in person club meetings as well as individualized kits for teachers and students with the exception of including consumables, for possible use during remote meetings. Teachers received a full kit during the fall professional develop-    Satisfaction with Summer Institute. Note: critical t = 2.05553. ment training workshop in preparation for the club lessons. HSTA prepared clubs kits containing all of the materials to perform experiments during the in person club meetings. Finally, HSTA will prepare individual student kits in preparation for remote programming as we move forward in these COVID-19 times. Student kits will allow HSTA to continue in the event of quarantines and/or school shutdowns.

Research Projects.
Unlike the virtual program, students will begin their research projects at the beginning of the HSTA club meetings. However, we have decided 9th and 10th graders will continue to participate in the statewide wellness surveys allowing the younger HSTA participants to learn about the research process and gain confidence in data analysis and interpretation. The veteran HSTA participants (i.e., 11th and 12th graders) will independently select their research topics. The CRAs will provide guidance and give final approval prior to the data collection process. In the event of a COVID-19-related shutdown, we will have virtual symposiums; however, the plan is to return to in person, college hosted symposiums.
Community Service. Another important component is the community service, which will return to an in-person format, with the exception of HSTA leadership hosting two to three monthly community service events. From our virtual experience, we learned that this would help HSTA students earn community service as well as allow the CARRS medical college students and college/university recruitment staff to reach many HSTA students simultaneously.
Guest Speakers. Pending local school policies, it is HSTA's intent to have in person guest speakers during the club meetings. In the event of COVID-19-related shutdowns and/or school policies, which will not allow outside visitors, HSTA will offer online guest speakers. We will encourage HSTA teachers and FSCs to work with local community members to accrue speakers and/or HSTA leadership will assist in this endeavor.

Weekly Staff Meetings.
Prior to COVID-19, the field staff and central administration did not hold weekly meetings. Central administration conducted weekly staff meetings. However, the field staff and central administration only met during the monthly community meetings. HSTA central administration will continue the post-COVID-19 virtual weekly staff meetings with field staff due to the distance of our staff. These weekly meetings have allowed our communication and program success to continue.

Transition of Student Records.
Prior to the pandemic, HSTA had begun to transition student records into the REDCap database. In order to allow the administration and field staff to have real time updates of students' progression and regression, HSTA ramped up the record transition process and club reports to a virtual format, which allows everyone to see in real time if students are in trouble. Rescue interventions are the result of this process and are vital to reaching students who need help so that we can ensure student success. Field staff and teachers could expeditiously contact students and parents who may have missed meetings, lost internet connectivity, or needed help. The rescue intervention is effective and practical, which is why HSTA will retain this successful component.

Streamlined Curriculum.
Since we have streamlined the HSTA curriculum so that it is contained into universal teacher and student notebooks as well as implemented assigned lessons/hands on activities related to the research process, we can ensure that participants receive complete program dosage. In previous years, teachers had the task of ensuring that participants completed six hands on activities and encountered at least one guest speaker per semester in addition to working on the research project. Now, those activities are the same across the program, and students receive more guidance in developing research skills. Furthermore, local guest speakers showcase to HSTA students the skills and talents of their own community members.

Best Practices for a Virtual Experience with High School Students.
Interactive. It was very important that the virtual club meetings were interactive. HSTA achieved interactive meetings by creating and sending home science kits. Going forward (i.e., fall 2021 semester), we ensured that activities are a good mixture of technology and hands on activities. We are preparing students for college with engaging activities such as dissecting an earth worm. We will instruct students in four types of data analysis (t-test, ANOVA, chi square, and correlation) through hands on experiments.
Technology is a wonderful tool, which allows for multifarious avenues for learning; however, HSTA recognizes the importance of encouraging students to move. For example, during summer camp 2021, students completed an environmental water testing kit on a stream, lake, etc. This activity had our students out and moving, and we used technology to deliver the lesson and perform data comparisons.
Visual Contact. It can be easy to lose a student in an online environment. If students are quiet and/or not use their cameras, they can be disengaged or log off early. In the beginning, we thought cameras should be optional, due to maintaining students' privacy family environments, but quickly learned cameras were important for student's engagement. Online meeting platforms have different options to hide the background and/or mute to block out sound. It was also im-portant for teachers to call on different students to talk and/ or share. Speaking during virtual sessions may be intimidating for some students and for those who are naturally soft-spoken having to speak during virtual sessions might be more stressful and difficult. Teachers can help alleviate the stress by coaxing students out of their shyness and creating a non-intimidating virtual environment.
Consistency. Fundamental to successful virtual meetings is setting consistent meeting dates and times. During COVID-19, HSTA observed many more students went to work after their regularly scheduled school day ended. We also observed this prior to COVID-19 regarding extracurricular activities (i.e., band, sports, etc.). Rather they took on these jobs due to extra time in their schedule and/or needed to help their families with finances, our students needed a consistent HSTA schedule.
Online Platforms. Many of our teachers utilized Zoom and/ or Team Meetings during their regular teaching. These platforms were a natural transition for delivery of HSTA programming. Furthermore, we found using Zoom break out rooms allowed students to work in groups for their project. Teachers could navigate back and forth helping individual groups.
Curriculum Notebook. Another important feature for program delivery was providing participants with a printed notebook to take notes about their assignments and research project. This resulted in heightened engagement while students followed along with the teachers as they disseminated lessons online. Although participants completed the research components online, they had the printed resources materials and activities at their disposal.

Expectations and Accountability.
Two important components to creating a successful online or in person after school program are accountability and flexibility. Programs need to hold students responsible for adhering to policies and regulations but should have grace to understand that there may be circumstances beyond the participants' control. HSTA students are at risk students by nature and often their situations can be difficult if not heart breaking. We strive to hold students accountable but also understand that, if necessary, we can help them by offering flexibility on a case-by-case basis. Finally, all expectations were clear and simple and repeated often. Students were aware of the ways in which to earn club credit as well as what assignments needed completing for successful completion of their research projects.
Final Thoughts. Data supported that there were differences among the time intervals (pre-COVID-19, 7 months into program suspension, and complete virtual program). There were also differences among the summer camp experiences pre-COVID-19 and during COVID-19. There are two major factors to consider when looking at outcomes, grade levels and attitude. For younger students who have very little in person or no HSTA experience, they did not have a complete baseline for comparison. Older students remember a very different experience, and we know that change is difficult. Further analysis into cohort differences may tell us more about the differences found among the time intervals. Analyses of qualitative results revealed both negative and positive student perceptions. All students realized the online program was not ideal, but some had a more positive outlook compared to others. This may have been due to maturity levels and/or an environmental impact.
When considering virtual programming, HSTA's advice is to examine what you have done traditionally and talk with students. The student evaluations are shaping how HSTA is moving forward. We know that getting students back on campus for summer camp is very important. In addition, students are more comfortable working in groups when meetings are in person. They feel they receive more guidance in person. HSTA also realized that online delivery of HSTA programming is achievable, yet we lose the peer-to-peer interaction as well as the benefit of teacher nurturing.
The idea of having an online/in person curriculum plan is ideal in our current times. This allows the program to continue to move forward whether the students are in school or not. However, the fundamental problem HSTA will encounter this coming year is consistency with meeting times. HSTA teachers will schedule a "no school" day and time for meetings in the event of cancellations of in person meetings. The curriculum will be consistent in the delivery but if school opening/closings fluctuate due to COVID-19, students will know their alternative meeting day and time also providing consistency.
During these unprecedented times, HSTA was able to move their program online continually moving forward. This encounter has taught HSTA leadership valuable lessons, and we have strengthened the program by continuing to develop a statewide curriculum to guide annual community-based research. In addition, HSTA was still able to connect with WV universities/colleges through online summer camps and zoom meetings so that participants could satisfy community service requirements. Most importantly, HSTA was able to deliver curriculum and respond to participants' needs in real time. This experience has strengthened HSTA revealing that it is possible, even during COVID-19 times, for this 26-year program to continue the long-standing goal of helping vulnerable and at-risk youth to achieve their academic and life dreams.