Development of a Virtual Summer Youth Program in Climate Change Resiliency

: Climate change is a timely topic and offers an overarching theme that can include a variety of STEM disci plines and technologies. In response to the pandemic, we developed and implemented an online science-based summer youth program in Climate Change Resiliency (CCR) for 11th and 12th graders in July 2020. Two sessions differing in content level and delivery were offered. The first session (S1) provided students with an introduction to the issues and scientific fields related to CCR. The second session (S2) explored CCR issues in more depth and utilized specific case studies to deliver con tent. Optional online informational and social activities were also offered including: UF admissions and financial aid panels; UF student life panels, climate change organization panels, and virtual game nights and socials. Overall program satisfaction was positive, with 97 percent of S1 and 81 percent of S2 participants reporting they were somewhat or extremely satisfied with the CCR program. Approximately 83 and 85 percent of S1 and S2, respectively thought the program’s approach to teaching and learning was very or extremely effective. Future recommendations to improve the program include increasing synchronous face-to-face instruction time and participant-led learning opportunities.


INTRODUCTION
For 60 years, the Center for Precollegiate Education and Training (CPET) at the University of Florida (UF) has provided high school students with high quality science education outreach offerings. Our major goals are to foster interdisciplinary science education learning, expose pre-college students to the STEM fields, and provide a window into university-based research. The impacts stemming from the COVID-19 pandemic left a substantial gap in summer educational development opportunities for high school students in 2020, and like many other university summer programs, CPET had to cancel all in-person summer research experience programs for pre-college students since in-person residential programs were not allowed on campus. To address the need for virtual pre-college summer programming, CPET developed, implemented, and evaluated a unique online, science-based collaborative learning program in Climate Change Resiliency (CCR), for rising junior and senior high school students during summer 2020.
The time between development and implementation of an online program was less than three months and meant either quickly converting existing in-person programs or creating a new virtual program. To meet the needs of our students and match the capacity of our staff, we decided to concentrate on delivering one virtual program that covered various STEM fields rather than convert various CPET programs to virtual. Similar to Bergsman and Chudler (2020) we utilized or modified existing online learning resources from other organizations, coupled with remote learning resources we designed ourselves. We also leveraged CPET's existing connections with UF faculty members across various STEM departments who expressed interest in sharing their research with precollege audiences.
We chose to focus on climate change resiliency as the overarching theme for the program since it is a timely topic, encompasses a multitude of STEM disciplines and technologies, addresses natural and social science issues, encourag-es experiential learning and problem-solving, and extends existing science education that youth receive in Florida schools. We defined climate change resiliency as the ability of natural, human-built, and social systems to rebound following a major disturbance, (e.g., natural disaster) and adapt to external pressures, (e.g., invasive species) brought about by a changing climate. We used the concept of resiliency to identify potential hazards to the system as well as understand the underlying vulnerabilities that may affect recovery.
There is growing interest in education to provide an informed response to climate change (UNESCO, 2010). Galindo-Gonzalez et al. (2011) state that developing a more climate literate society is a critical need and that by understanding human influence on climate and how climate affects society, people can make better informed decisions in response to climate change. Additionally, to successfully deliver climate change education, Monroe et al. (2019) conclude that focusing on making climate change information personally relevant and meaningful to learners is one of the key strategies to a program's success. As a center within the University of Florida, CPET has access to a broad range of statewide research, technology, and information that can highlight the direct effect of climate change within Florida and engage learners using place-based, culturally inclusive learning.
While components of climate change science are included in Florida's educational standards (https://www.cpalms. org), having the time and resources to combine these into specific climate change science lesson plans can be challenging for teachers to deliver when faced with other academic requirements during the school year. By providing a summer program focused on the multidisciplinary aspect of climate change science, we aimed to provide an opportunity for students to bridge knowledge from previous science classes and experiences and connect them with new perspectives, technologies, and research.
Learner-centered experiences (e.g., inquiry-based, and experiential activities) are often the emphasis for summer STEM programming since it allows learners to create their own understandings and develop new skills (NAAEE, 2021). The challenge during the pandemic and beyond is how to translate those in-person learner-centered experiences into the virtual environment. Due to some of the advantages of online learning, such as increased flexibility, self-pacing, and accessibility (Adedoyin and Soykan, 2020), virtual programming will most likely remain a component for many outreach and non-formal education programs even when in-person programs return. We offer a description of our program to highlight lessons learned over the course of developing and implementing a virtual summer youth science experience and as an example of what virtual components can work and/or could be improved upon to provide an engaging, interactive, and meaningful learning experience.

METHODS
The CCR program sought to increase participants' knowledge and understanding of the impacts of climate change on natural, human-built, and societal systems while providing an opportunity for participants to explore solutions to these impacts and learn about the various scientific fields engaged in climate change research. The specific objectives of the program were to (1) expand participant knowledge on the science of climate change, (2) inspire students to take action towards climate change, and (3) encourage the exploration of STEM-related college majors and careers. The CCR program offered two 2-week sessions, with the first session (S1) geared toward providing students with an overview of the issues and scientific fields related to climate change, while the second session (S2) explored climate change resiliency issues and research in more depth.
Program Application and Participant Cohort. In response to the pandemic, the application process for the virtual CCR program was much less rigorous and comprehensive compared to our in-person programs. Traditionally, CPET requires participants to provide essays, short answers, transcripts, and teacher recommendations. Due to time constraints, we streamlined the CCR program application, requiring only demographic information, current unweighted GPA, a list of STEM-related coursework that the student had completed or was currently completing, and a response to what they hoped to learn from the program. The application was open to U.S. residents who had completed at least two years of high school. Participants were recruited through emails to eligible students who had applied to cancelled in-person programs; students who participated in other CPET programs; emails to teachers on the CPET mailing list; and through CPET social media, including Facebook, Instagram, and Twitter. Students could apply to one or both sessions.
We had a total of 64 participants in the CCR program: 34 in S1 and 30 in S2. Participants came from 16 different counties throughout Florida and four other states: California (n=3), Illinois (n=1), Ohio (n=1), and Texas (n=1) (Figure Identified as non-white 21 21 Identified as white 13 9 From urban areas 7 9 From suburban areas 27 21 1). Across both sessions, more participants identified as female compared to males, as non-white compared to white, and coming from urban areas versus suburban (Table 1). There were 10 students who attended both S1 and S2. The cost of the program was $350 per session, which covered the cost of program staffing and science kits and materials that were mailed to selected participants prior to the start of the program. Scholarships were provided to seven students who demonstrated financial need.

Special Considerations for Online Youth Programs.
Working with minors in an online program required some special considerations different than in-person youth programs. We worked closely with the UF General Counsel and the Office of Youth Compliance to consider issues of online safety and participant privacy. Parents were required to sign a Participation Agreement that outlined potential risks from participating in online programs and using digital platforms, as well as a consent form for the use of students' images and recordings. Additionally, a minimum of two background screened and youth compliance approved staff members were present during all synchronous activities to provide program services, as well as to monitor participant behavior.
Initial Planning and Program Development. Our program design challenge was to create an online program that reflected the engagement of our in-person summer research experiences. At the onset, we determined components and activities from our in-person programs that we wanted to translate into the virtual environment and how we might achieve that. To translate the immersive on-campus expe-rience, the CCR program provided virtual group panel sessions for campus services and student life, as well as virtual social activities, such as games and talent shows. Instead of hands-on experiences in UF laboratories, the program provided take-home science kits and virtual interactive labs. To address the varied academic experiences and interests of participants we provided two sessions that differed in content breadth and depth. Lastly, as an alternative to meeting in-person for a full-day, we limited the amount of screen time to two hours per day and mixed both synchronous and asynchronous activities as a means of engaging participants while minimizing screen fatigue and/or loss of interest (see Table 2 and Figure 2).
To address the comprehensive, multi-disciplinary nature of climate change research and deliver our first program objective, the content for each of the sessions was divided into three themes 1) resiliency in natural systems, 2) resiliency in human environments or human-made systems, and 3) resiliency in social systems. A list of daily topics specific for each session are listed in Table 3. Both sessions of the program included online content and discussion, experiential activities and laboratory exercises, virtual field trips and tours, guest lectures from faculty and graduate students, and participation in optional activities, including socials, office hours, and informational panels. We used Canvas (Canvas ©2021; Instructure, Salt Lake City, UT) as our online learning management tool and Zoom (Zoom ©2021; Zoom Video Communications, Inc, San Jose, CA) as the videoconferencing platform.
To address our second program objective to inspire students to take action towards climate change, we provided opportunities for participants to engage in virtual panel discussions with UF student-led climate change organizations; contribute to analysis and discussion of various environmental footprint calculators, e.g., carbon and energy footprints; reflect on video and podcast stories of youth taking action in their communities; spotlight successful solutions brought about by community action; and develop civic skill-building activities through debates and role-playing.
To meet our third program objective of encouraging exploration of STEM-related college majors and careers, we asked guest lectures to speak to their personal journeys as to how they became involved in their respective STEM fields. Additionally, participants had opportunities to attend virtual panel sessions from UF students and recent graduates who presented and discussed their experiences in various STEM majors and graduate programs. Session Content. The first session of the CCR program was designed to provide a broad overview of the content, with primers in the various topics (Table 3). Each day included an introduction to the day's topic provided by one of the two co-leaders for the program, interactive online activities, and a lecture by a guest speaker. Guest speakers included faculty and graduate students from the UF Department of Biology, Center for Aquatic and Invasive Plants, UF School of Forest Resources and Conservation, UF Department of Chemical Engineering, UF Department of Geography, the Florida Sea Grant Law Conservation Clinic, and John Carroll University's Department of Environmental Science.
In the culminating project for S1, participants prepared for and engaged in a mock city council meeting to decide what the city of Jacksonville, Florida could do in response to climate change and sea-level rise. The project was adapted from Kluver et al. (2018), with participants being given a community role in each of the following climate change strategies: prevention, mitigation, or adaptation. Community roles consisted of both public and private characters, e.g., tourist shop owner or public utility worker. Each strategy group met throughout the two-week program to discuss and decide on a proposal or action plan for their strategy to present to the city council. On the final day of the program,   ties and presentations (e.g., lectures, interactive activities, assignments, etc.) and provide short answers to questions regarding what participants found most engaging and what new concepts they learned. On the last day of each of the program sessions, participants completed an overall program evaluation which included multiple choice and short answer questions.

EVALUATION RESULTS
Twice-Weekly Evaluations. Mean participant rating percentages for mandatory activities were similar across sessions ( Figure 3). Guest lectures presentations, staff presentations, and final projects received the highest ratings. Participant reflections mirrored rating responses with many participants expressing they found guest speakers to be their most engaging activity and that they enjoyed learning about the different areas of research and how that connected with real world problems: After all groups had presented, city council members used a Zoom break out room to deliberate and choose an action plan. They then returned to the meeting and read a written statement justifying their decision. The second session of the CCRP went more in-depth into specific climate change topics and focused on participant-led activities to deliver content (Table 3). Each day included a case study presentation and discussion led by student participants, followed by a lecture by a guest speaker. Guest speakers included faculty and graduate students from the Florida Museum of Natural History, Alachua Conservation Trust, John Carroll University's Department of Environmental Science, the UF Department of Chemical Engineering, the UF Department of Geography, and the UF/IFAS Nature Coast Biological Station and Department of Agricultural and Biological Engineering.
In S2's culminating project, participants developed and carried out scientific experiments to demonstrate the effect of changes in the environment on the growth of basil seedlings under different experimental treatments of their choosing. Participants grew the plants in selected or simulated environments such as high salinity, cold, heat, flooding, or drought and observed the responses of the seedlings to compare the growth of treated plants to that of the control plants grown under optimal conditions. Participants wrote experimental reports to discuss their rationale for the study as related to other scientific studies, provide their hypotheses and methods for the study, report their findings, and discuss the implications of their results, i.e., how the responses to environmental factors related to potential climate change impacts and strategies.
Optional activities for both sessions included virtual socials (Climate Change Trivia or BINGO, Outburst Game Night, and a talent show), guest panels (UF undergraduate students, climate organizations, and UF Admissions and Financial Aid), virtual farm tours, and office hours with the program co-leaders. In addition to the synchronous activities, participants engaged in a variety of activities on their own time. These included online interactive activities, science kits and laboratories, listening to podcasts, and completing assigned readings of case studies and scientific journal articles.
Evaluation. Program participants provided feedback and evaluation of the program twice per week, and at the end of the program session via surveys administered through Canvas (Canvas ©2021; Instructure, Salt Lake City, UT) and Qualtrics (Qualtrics XM ©2021; Qualtrics, Provo, UT). Twice weekly evaluations had participants rate key activi-

idea, it was amazing to hear everyone's proposals and see how much research and effort we all put into our plans.
Optional activities and participants' ratings are shown in Figure 4. Staff observed that optional activities had low attendance, and this is reflected in the large values for the "No Response" rating with more than half of the participants in each session not providing an evaluation for these activities. However, among participants that did attend optional activities, these activities were generally ranked "Great" and received positive feedback:

I thought the student panel was the most engaging. I really enjoyed hearing about everyone's college experiences and how they realized their passions (also the advice was helpful)!
The virtual farm tours were so cool! Having both a large farm and a small farm provided different perspectives on how climate change affects their farm and plants.

I am happy that I attended the UF Admissions Panel. The information shared was very valuable all around.
The game social was also very fun! I'm glad I got to hang out with the other students since the program is ending.
Negative feedback for the program mostly focused on a need for more time to discuss or review topics and activities. For some, the activities were not challenging enough and the information presented was too basic. Examples of negative comments included: If I had to change anything, I think the assignments should be talked about more in the zoom (what was learned, how they went, etc.).

I did not learn much anything new from any of the introduction sessions for the first hour and everything I learned or enjoyed was from the guest speakers. The science kits felt very basic and elementary, and I did not learn much anything new from the science kits.
I don't like the case studies because they are long, difficult to understand, and not very interesting. I found that the podcasts and videos from last session were more interesting and effective in sparking curiosity.
Final Projects.The final projects were integral learner-centered activities that generally received positive responses from participants. Sessions 1 mock city council meeting project gave participants an opportunity to critically think about, defend, and extend their ideas; consider alternative viewpoints, and gain confidence in what they know. Feedback included:

This mock trial allowed me to experience what is like to be in a city council meeting and how people would react to climate change solutions. It was an interactive activity that helped reinforce what I learn during the program. Additionally, I was able to learn more about my classmates and meet new people.
I liked being able to hear from every separate group and compare their approaches. It was interesting to see how they prioritized different things whether they were involved in prevention, mitigation, and adaptation. I especially liked seeing how their different approaches could be combined for a solution.
The most interesting part of the day was listening to everyone else's presentations during the council meeting, in order to get aspects from typical roles of the city's population and how that effects the city council's decision on how to combat the effects of climate change.
The plant experiment final project in the second session, was an experiential learning activity that allowed students to investigate a problem and experience the scientific process for themselves. The main challenge for this project was participants getting the seeds to grow in their various remote environments and having enough time to run the experiment. The lack of seedling germination led to mixed reviews for the project with some enjoying the experience of conducting the experiment stating, "I really like the implementation of the plant experiment because it is really fun and engaging to watch the plants grow and react to different environments." Or "I also like how I can go through the process of conducting hands-on experiment and writing hypothesis and reports!". Other participants felt frustrated at not being able to complete the experiment expressing, "I wish I'd had more time with my plant experiment, the short time frame on the experiment kind of skewed my results." Or "My experiment didn't really go as planned at all, it experienced many drawbacks and made it so that I barely had any data to properly report in my experiment report." Session Comparisons. Ten students attended both program sessions and provided insight into how the two programs compared: Session 2 was harder than session 1 because we had more major projects but it was worth it. I learned a lot about climate change in both sessions. I don't regret attending it at all. I really enjoyed both sessions. I think that session one gave me a good overall understanding of the issue, and session two delved further into the solutions as well as some of the larger issues that may come as a result of climate change. Although I did prefer session one due to how it was formatted. I also enjoyed the mock city council meeting a bit more than the plant experiment, but I am biased in this as I did not get results from my experiment due to the plants dying.
I liked session 2 a bit more because of the group projects being spaced out and everyone got to present. It was a bit more interactive and engaging for me. Session 1 was a little bit more lab work focused which I enjoyed, so maybe incorporating a few other labs into this session would have made it "better" than session 2.

End of Session Program Evaluation.
Evaluation responses for the end of session program surveys are listed in Table 4. Generally, both sessions were well-received by participants, but S2 generally showed greater heterogeneity in survey response distribution across ratings and had lower ratings relative to S1. For example, ratings provided for "how effective the program's approach to teaching and learning was" were more varied in distribution between sessions (44, 41, 7, 4, and 4 percent in S2 versus 60, 23, 10, 7, and 0 percent in S1). Overall, participants from S1 and S2 reported they were satisfied with the CCR program with 97 and 81 percent somewhat or extremely satisfied, and 87 and 78 percent reporting that the program was either very or extremely interesting. Almost all participants felt somewhat or extremely satisfied with the instructors, believed they were very or extremely knowledgeable about the material being taught in the program, and facilitated their understanding of the material ei-ther very or extremely well. Greater than 80 percent thought the program's approach to teaching and learning was very or extremely effective. Participants' comments provide additional insight into their experiences with the CCR program: I really enjoyed the assignments and the science kits because it was very hands-on and gave me a chance to see first-hand how these changes are effecting us directly. In addition, I really enjoyed all the presentations, seeing people who love what they do speak about issues that are impacting the world is very inspiring and made me want to change my habits to better our environment and educate myself further.
Overall, I believe I have come out of this program with a new perspective on climate change and the things I could personally do to help. This program allowed me to think outside the box and educate myself even more on what is happening in and on our earth.
I have a renewed understanding of the work it takes to not only understand what needs to be done in regard to climate change, but to actually implement these ideas as well. I also definitely think I understand now more than ever just how pervasive climate change is, and how it manages to seep into all aspects of human life, even for the many people who think it's a far-off problem.
I was able to make friendships with people who enjoy protecting our earth against climate change as much as I do, which is something that I am very thankful for.

DISCUSSION
The CCR program sought to translate CPET's immersive summer pre-college experiences into a virtual environment in response to cancellations of in-person programs due to COVID-19. We chose the topic of climate change to provide an over-arching theme that encompassed multiple STEM fields, extended participants existing climate change science knowledge and understanding, and explored individual and community actions and solutions.
Transforming the program to an online format did afford some opportunities that have the potential to benefit our program and future participants after the COVID-19 pandemic. First, it allowed CPET to expand program offerings, filling the gap in interdisciplinary education on the topic of climate change. Second, the initial investment in time to develop online content and activities now gives CPET a resource that can be leveraged in different ways for future programming. For example, we now have a program that is accessible to students who might not be able to attend our in-person pro-grams, whether due to travel restrictions, cost, or any other barrier to attending in-person programs. While a direct comparison could not be made to an in-person program, relative to other two-week in-person CPET programs from previous years (e.g., Research Explorations) the virtual CCR program generally had participants from more Florida counties and a greater proportion of out-of-state participants. The CCR program also tended to have greater diversity among student backgrounds and experiences (e.g., race, ethnicity, and socio-economic status) relative to other CPET programs. Participant fees for the virtual CCR program were much less than in-person programs ($350 per person versus $2,400 per person) and while scholarships are offered, cost to attend could influence a participants' decision to apply to a program.
To develop and implement this program we were tasked with converting immersive, experiential, science education experiences into virtual activities that were engaging, inter-active, and provided meaningful learning opportunities. Our program included many activities which were shown to help move learners beyond the basics of climate science including having participants interact with scientists, providing opportunities to explore their perspectives and assumptions about climate change, and allowing them to investigate options for solutions. Below we present the challenges we encountered developing and implementing the program, and based on evaluations and reflections received from participants, offer suggestions for improvement that can benefit future virtual programming.
Program Challenges. The program staff developed and implemented the program in a short time frame, which meant that not all desired learning activities, guest speakers, and materials were available. For example, due to high demand from other science classes or education programs, many of our first-choice remote learning science kits were not avail-

Survey Questions Session
Extremely Satisfied  able or could not be supplied in the volume needed (many remote science kits are designed for classroom not individual student use). This meant using more time to investigate, select, and order alternative kits, and leaving less time to assess how they would work and fit within the already established program learning objectives. Suggestions for future virtual programs are to identify early-on in the planning stage what science kits/resources the program wants to use and contact the vendor to see if it will be available in the volume needed. UF Youth Compliance regulations require at least two background-checked staff to be present in any group meeting with youth participants. With a staff of four people (two instructors and two assistants) this limited the types of activities and discussions that could be held during the program. For example, with more than 30 youth participants in each session of the program, we were unable to use the "breakout room" feature in Zoom, since it required two staff members to be in each breakout room, meaning at most we could use were two rooms. Making sure that the capacity of program staff and resources can support the number of participants and desired outcomes is key to determining what digital components can and should be included for a successful, engaging virtual program. For the future, we plan to limit the number of participants so that we can afford smaller group discussions and provide more opportunities for collaborative learning while still maintaining the adequate youth to adult ratios needed to meet youth compliance.

Satisfied nor Dissatisfied
As a pilot program one of our biggest challenges was delivering desired content and maintaining participant engagement and motivation with limited screen time. While we did not directly ask participants about "screen-fatigue" some participants mentioned difficulties staying focused and that it was "hard being online". On the instruction side, it was quickly realized upon implementing the program that two hours of face-to-face time was not sufficient to fully discuss and explore topics. Based on the positive reviews we received from participants regarding learner-centered activities, we will be increasing face-to-face instruction time to allow for engaging discussion and inquiry-based activities. Additionally, to address the less positive scores for asynchronous activities we will also include more face-to-face time to review and discuss these activities. The risk of screen-fatigue will be reduced by exploring the use of different online teaching tools and activities, e.g., Padlet, Jamboard, Flipgrid, etc. (Kast et al. 2020) to hold participants attentions.
An important component of our in-person programs, is to include social activities for participants to build friendships and make connections with one another. We sought to include this social component in the virtual program while working around the screen-time limitations, by making our virtual social activities optional. Although the optional activities were well-received by the participants who attended, the overall attendance of them was low. Making these social activity sessions mandatory will increase participation but at the cost of additional screen time. Participant reflections revealed that one of the positive aspects of the group projects was that it provided a means for participants to connect and make friendships. As an alternative to including separate virtual social activities, we are exploring integrating social activities into content delivery and including more participant-led and small group activities where participants can learn and engage with one another, thereby developing friendships and connections.
The CCR program was made available to a wide range of students, some of whom had limited knowledge of climate change, while others had an advanced understanding. The wide range in individual pre-existing knowledge made it difficult to adapt the program content. Even with our attempt to provide two sessions that varied in content breadth and depth, some students mentioned that at times the educational experience was too basic, while other students thought the material was too challenging. For future programs, we plan to offer more sessions focused on specific climate change topics, e.g., ecosystems, engineering, and public health, that will allow us to explore while building up participants knowledge over the session. Lastly, to better improve the program, assess effectiveness, and refine target audience needs, future programs will aim to include pre-and posttests to assess program outcomes and measure changes in attitudes, knowledge, and behavioral intentions as related to climate change.
Adapting to virtual programming was unexpected and challenging but gave us an opportunity to explore online program practices that prior to the pandemic we may have been hesitant to implement or did not have time to develop. We look forward to integrating successful components of our virtual programming efforts into future UF CPET pre-college outreach initiatives and experiences.

Corresponding Author
Cynthia Nazario-Leary. Center for Precollegiate Education and Training, University of Florida. Gainesville, FL, USA. cnazarioleary@ufl.edu

Author Contributions
The manuscript was written through contributions of all authors. All authors have given approval to the final version of the manuscript.