Celebrating STEM Virtually: The Bay Area Science Festival During the COVID-19 Pandemic

: Founded in 2011, the Bay Area Science Festival (BASF), led by the Science and Health Education Partnership (SEP) at UC San Francisco, brings together academic, scientific, corporate and civic groups to develop and lead STEM (science, technology, engineering, and mathematics) events for all ages. From 2011-2019, the Festival hosted over 50 events each year, reaching tens of thousands of people. In 2020, in response to the COVID-19 pandemic, the Festival pivoted to a virtual format, offering nearly 150 virtual events over five days. Events included hands-on science experiments, STEM storytimes, live interviews with scientists, demonstrations, behind-the-scenes tours and more. Nearly 19,000 unique users from over 35 countries have logged 82,100 views of Festival events to date. In a post-Festival survey, attendees rated their experience similarly to previous years and over 60% of attendees are interested in attending virtual events even when in-per son events are again possible. Like many aspects of our lives, the Bay Area Science Festival will look different in future years as a result of the pandemic: a hybrid Festival made up of virtual and in-person events. While there are great benefits to in-person events, we have also come to better understand the benefits of virtual ones.


INTRODUCTION
Over the last decade, science festivals in the United States have experienced significant growth. In the early 2000s, there were a handful of American science festivals; today more than 50 festivals celebrate STEM annually. While each festival is a unique reflection of its home institution and community, science festivals share some common characteristics and goals (Bultitude et al., 2011): (a) they celebrate science, technology, engineering, and mathematics (STEM); (b) they engage the public with STEM content; (c) they are time-limited events that recur annually or biennially; and (d) they use a common theme or branding to unify their various activities. Beyond these commonalities, science festivals can differ significantly, with budgets varying by a factor of 1,000, geographic reach ranging from a neighborhood to a city to a state, length varying from one day to one month, and staffing ranging from entirely volunteer to several fulltime paid staff (Wiehe, 2014).
In 2011, the Science and Health Education Partnership (SEP) at UC San Francisco (UCSF) launched the Bay Area Science Festival (BASF). The Bay Area Science Festival is a collaboration between academic, scientific, corporate, and civic groups that emphasizes the importance of science, engineering and technology in the local community with the overarching goal of providing accessible, engaging STEM programming to San Francisco Bay Area residents. The Festival seeks to 1) raise awareness of STEM in everyday life, 2) increase dialogue about science and technology throughout the region, 3) enlighten Bay Area residents about the region's vast science and technology assets, 4) engage young people in the fun, excitement and awe of science, and 5) encourage collaborations between and among the science and technology community and Bay Area residents.
Over the course of its first nine years, the Bay Area Science Festival hosted approximately 50 in-person events annually, reaching more than 50,000 people each year. The Festival's most popular events were hands-on science ex-The 2020 Bay Area Science Festival. In spring 2020, as the novel coronavirus spread worldwide, the San Francisco Bay Area instituted wide-ranging restrictions. As in much of the country and world, schools were closed and moved to remote learning, all but essential workers switched to working remotely, gatherings outside of households were restricted and non-essential travel was highly discouraged. The Festival's home institution, UCSF, closed all labs that were not actively engaged in COVID-19-related research, instructed all faculty and staff to work from home if possible, and suspended in-person classes for students. Long-standing collaborators of the Festival such as science centers, educational institutions, and other non-profit organizations were closed and some experienced large-scale layoffs or furloughs of their staff.
With this as a backdrop, we were faced with beginning to plan for the annual late October Festival. As the pandemic's toll and duration exceeded our initial assumptions, we realized the only way for the Festival to continue was to go virtual. Given the prevalence of Zoom and screen fatigue for youth and adults alike, as well as the financial and staff impacts faced by our collaborating institutions, we were concerned that both event hosts and attendees might have limited interest in and capacity to participate in the Festival. Additionally, while we, like many, had become comfortable using Zoom for meetings in the first few months of the pandemic, we had never hosted a virtual public science event of any size. And, there were few groups to turn to for advice: a small number of local organizations had begun leading virtual events but nothing on the scale of the Science Festival. We reached out to collaborators in spring 2020 for an initial brainstorming meeting and were thrilled by the response. A great many groups attended the meeting (more than in prior year in-person meetings) and they expressed a strong desire to find ways to engage the public in meaningful, virtual interactions with STEM professionals, offer events that were hands-on and interactive, design events with equity and access in mind, and work together with the goal of creating a "festival feeling." Given this interest in continuing the Festival virtually, and with the pandemic and its resulting restrictions in the Bay Area continuing much longer than expected, the Festival team became committed to and excited about offering a virtual Science Festival.

LEADING A VIRTUAL SCIENCE FESTIVAL
Having never led a virtual public science event, we quickly decided to hire an event manager who had experience planning and executing large virtual events. With her help, we developed an overall Festival strategy and vision, selected a website hosting platform, and determined and supported the logistical and technical needs of sponsors, event hosts and presenters.
Festival Programming. Throughout our planning for the virtual format, we were focused on how to engage attendees in STEM experiences at home, without the benefit of handson materials, in-person interactions, or the excitement of a festival atmosphere. With concerns about "Zoom-fatigue" and how much momentum virtual events would sustain within the audience, we reduced the length of the Festival from its usual ten to five days. Festival events spanned Wednesday through Sunday, with most programming between 10 am and 7 pm. This schedule was designed with the goal of allowing a wide range of audiences to participate, including school or classroom groups during the day; families at home with their kids during the day, in the early evening, and on weekends; and adults during weekend and early evening offerings. To support audience engagement, events were kept relatively short, with most lasting 30 to 60 minutes. Events also incorporated several interactive components, such as polls, Q&A sessions, soliciting audience responses via the chat, and allowing audience members to share comments or questions live on camera.
After much consideration, we decided not to create a virtual equivalent of the Discovery Day events as it felt challenging to provide a satisfying experience for visitors and exhibitors alike through an online portal. Prior to making this decision, we reviewed several online experiences designed to mimic exhibitor halls, where attendees could move from booth to booth and talk with exhibitors via chat or video. However, these portals did not provide a way for attendees to experience a key aspect of the Discovery Day experience: hands-on, one-on-one interactions with STEM professionals. An analysis of 14 US science festivals shows that attendees who report interacting with a scientist at an expo event (e.g., Discovery Day) rated their overall experience, learning, inspiration, fun, and awareness of STEM careers more positively than those who reported not interacting directly with a STEM professional (Boyette and Ramsey, 2019). We felt these experiences would be extremely hard to duplicate in a virtual expo event in a way that would both reach a large audience and foster connections between individuals.
Ultimately, the suite of Festival events included: • STEM@Home events were designed for families, young children, and/or teens to engage in hands-on science in their own home, with materials that most people would have readily available or were easy and inexpensive to purchase. Event hosts were also given the option to have material kits sent out to attendees ahead of the Festival. This event category was entirely new this year and was seen as a virtual equivalent of the hands-on STEM activities that attendees experienced when visiting Discovery Days in prior years.
• STEM Storytimes targeted our youngest audience members from preschoolers to early elementary school students. Scientists and health professionals were invited to do read-alouds of STEM picture books, engage the young viewers in a brief hands-on activity related to the book, and talk about how what they do in their professional lives related to the story they read. The Festival collaborated with local libraries to leverage librarians' experience in this domain, as well as the libraries' already established audience network. Festival staff connected the libraries with local scientists as readers and provided support in developing STEM activities to accompany the readings.
• STEM in the Community events were designed to give the public an opportunity to engage with STEM outside of their homes, to allow them to move away from their screens, and to experience STEM being part of their everyday life. Examples of this type of event included "citizen science" or "community science" projects (done asynchronously), but also included synchronous, homebased scavenger hunts and watching stars in the night sky while listening to a radio broadcast remotely.
• Explorer Tours have been part of Festival programming since its founding. The tours are designed to give the public a unique behind-the-scenes tour of research laboratories, industrial production floors, or museum exhibits. Other tours provide an outdoor field trip experience to explore the science and history of local natural environments. We invited event hosts to offer Explorer Tours virtually this year, either live online, pre-recorded or as a mix of both (e.g., a prerecorded tour followed by a live question and answer session).
• STEM Presentations comprised panel discussions (in-cluding career panels geared towards young people), conversations with STEM professionals and authors, and science film screenings followed by audience interactions with the film's producers and scientists. This event category, which has traditionally been part of the Festival, was among the easiest to envision transitioning to a virtual environment.
Festival Website Platform. One critical move for the pivot to a virtual Festival was the selection of a platform for attendees to access the events hosted by our collaborating institutions. Easy navigation for Festival attendees, ability to customize the platform to serve various event host needs, and brand recognition were among our top priorities as we considered several platform options. We also sought seamless integration of video meeting technology and the ability to store recorded events for later on-demand viewing. Of course, cost was also a consideration. Virtual conference platforms such as Hopin and Whova offered relatively smooth user experience but were missing customization and video storage capability. Ultimately, we chose to have a fully custom-built site designed by a company that specializes in virtual event platforms. The production company provided the additional benefit of handling technical support during all live virtual Festival events and uploaded recordings of event sessions to the platform for post-event viewing.
The finished platform featured a daily event calendar that was searchable by target age group, subject/topic, event type, and Spanish language offerings. Each event was featured on a unique webpage that provided an event description, downloadable resources, and information on the hosting institution. For some events, pre-registration was required on the event page, either to limit the number of attendees in order to ensure rich interactions between the audience and event hosts, or to facilitate mailings of material kits for selected STEM@Home activities. Recordings of events were uploaded to each event page to allow for on-demand viewing once the live event was complete. All institutions that hosted a Festival event on the platform were listed in the Collaborator Directory, allowing event hosts to share the mission of their organization and post additional resources and opportunities for the public to connect with them.
Networking groups, for event hosts, parents, teachers and teens respectively, were created to allow social interactions among festival attendees. The event host group was private while anyone could join the three public groups after registering on the platform. A Festival team member monitored posts in the networking groups to ensure a safe space for attendees to socialize. Ultimately, these networking groups were rarely used with only a handful of messages posted by attendees.
Registration on the Festival platform required only a name and email address. Attendees were asked to sign a Code of Conduct agreement and to give their consent to be recorded, if their cameras were on, during Festival events. Parents were asked to sign these documents for both themselves and their child(ren). Requiring registration gave us the ability to collect attendees' contact information for evaluation and future Festival outreach and to protect minors from any potential predatory behavior. This extra step to access the events allowed us to block an attendee if needed (via IP address and email). Thankfully, there were no significant issues with attendee behavior. We also did not gather any information about attendees' identities (age, grade etc.) and we stated at the start of each session that it was being recorded and if the attendee did not want to be recorded, they should turn their camera off. Event Hosts.The Bay Area Science Festival, like many other science festivals across the US, is a community celebration. The wide-ranging suite of events (approximately 50 each year) is only possible because numerous Bay Area groups lead and participate in events. In 2020, as in prior years, event hosts included local STEM corporations, universities, research labs, museums, community groups, student organizations, and public libraries. The Festival team supports, guides, and manages the events. In 2020, the team required all events to be free.
Event submission requirements and guidelines were emailed to potential event hosts three months prior to the start of the Festival. Hosts were invited to submit initial ideas for their events, without an expectation that their vision be completely formed. We encouraged event hosts to experiment and try out new ideas, and, in particular to think about ways to successfully use the virtual format. Hosts were asked to submit their organizational contact information, an event title, an event description, the type of event, the target age and if known, the desired date and time. Once events were accepted, we asked for additional information about their organizations, which was used to build out the Collaborator Directory.
The Science Festival team supported event hosts in their planning via webinars that shared Festival information and goals, and also connected event hosts with each other. Our first meeting, in spring 2020, focused on a community brainstorming designed for us to get feedback and help all of us envision a virtual Bay Area Science Festival. The second meeting, in summer 2020, provided more information about the Festival structure and details about the different event types. The meeting included suggestions for how to make events interactive in the virtual setting. Ideas included allowing the audience to ask questions via chat or video, using Zoom features such as instant polls and reactions, providing ways for the audience to follow along at home as they watched a demonstration online (drawing, building, experimenting), and developing ways to get people moving.
Event hosts were also encouraged to think carefully about the length of their sessions, and to make them shorter, especially for young children. Other suggestions for those leading events for young children were to ensure use of age-appropriate language and to consider activities that included movement and/or singing.
Our event management team helped design and lead Zoom trainings for event hosts. These trainings focused on the basics of hosting a meeting, such as how to share a screen, put up a poll, create break-out rooms, and manage the settings for the chat. All hosts were also provided with a Science Festival virtual background for consistent branding and a detailed Zoom guide with an event tech checklist. For those who were less familiar with Zoom and desired more personal support, we offered one-on-one rehearsals where event hosts could run through their event and practice using Zoom. Additionally, one tech support person and one Festival liaison (either Science Festival staff or volunteer) supported each virtual event with troubleshooting, managing participants, monitoring the chat, preventing "Zoom bombing," and acting as a Festival host by launching each session with welcome slides for attendees and greeting presenters before their sessions. The tech support person met with event hosts and the Festival host 30 minutes before each event to do an audio and video check and troubleshoot any issues. If necessary, the tech support person reviewed how to launch the meeting or webinar, screen share, manage participants, pin video or speakers, and modify settings for chat, Q&A, polling.
Sponsors. Sponsorship during the 2020 virtual Festival, as expected, dropped significantly, largely due to uncertainties that many companies faced with the impact of the COVID-19 pandemic, with loss of revenue and possibilities of employee layoffs. Half of the eighteen 2019 BASF sponsors did not return in 2020 and of the nine sponsors that did continue support in 2020, three reduced their funding. The Festival did gain one new sponsor in 2020. The 2020 sponsorship benefits also reflected the shift to a virtual Festival platform. Normally sponsors would have had a significant presence at our large Discovery Day event, with exhibit booths on the field at Oracle Park, recognition on the electronic billboard and logos in our print program. This year's benefits included opportunities to sponsor an event series, greater support from the Festival team in designing a virtual event and being highlighted in Festival social media and email newsletters, in addition to the inclusion of sponsor logos on the Festival website. Thankfully, the drop in sponsorship revenue was mirrored by a drop in total cost of running the Festival. Hosting the Festival virtually was far more economical than hosting an in-person Festival, with the bulk of the cost savings stemming from the elimination of event space rental fees, in particular those for the ballpark Discovery Day event.
the Zoom meeting software were not allowed by the Festival's host institution, UCSF (a patient-facing biomedical research university), due to HIPAA (Health Insurance Portability Accountability Act) regulations. UCSF was actively working on providing AI-based closed-captioning for Zoom accounts but the approval was not finalized until after the Science Festival.
With Spanish commonly spoken in the Bay Area, collaborators were encouraged to offer events in Spanish. A series of Spanish language events were led in collaboration with the Mexican Consulate in San Francisco and UC Berkeley, and a few other Bay Area organizations hosted events in Spanish. The Festival also made a concerted effort to bring in voices from and issues relevant to people and groups that have traditionally been marginalized in STEM and/or have experienced a history of oppression by science and medicine in this country. The hope was that the Festival could act as a platform to amplify the important work that these groups do and generate public conversations about the impact of racism and other forms of discrimination in academia, health care, and STEM education.

METHODS 2018 Intercept Surveys at Discovery Days. At the 2018
Discovery Days at Oracle Park, the Sonoma County fairgrounds, and the California State University -East Bay Concord campus, field researchers and trained volunteers implemented a short intercept survey. Surveys were available in English or Spanish, and were collected using iPads. To collect the data, field researchers and volunteers were assigned to specific zones of the events, and asked to pick a spot to begin the intercept process. The fifth person to cross the researcher's path was then invited to complete the survey. The survey included demographic items and questions about their experience at the event. Survey respondents did not provide income data. This survey, with slight modifications, has been used since the Festival's launch in 2011. In 2019, because of funding limitations and the fact that results year after year were very consistent, we chose to use a different collection method and received a small number of responses. Thus, for comparison to the 2020 Festival, we are using the 2018 survey data. A total of 561 attendees of Discovery Days responded to the intercept survey in 2018. 2020 Online Surveys. Though registration to the online platform only required a name and email, registrants were also asked to answer two optional questions: their zip code and how they heard about the Festival. Over 5,000 people answered these questions (5,091 submitted their zip codes and 5,691 shared how they heard about the Festival).
Post-Festival, all registrants were asked to complete a survey. This survey had 36 questions and was offered to

GETTING THE WORD OUT
Some of the Festival's traditional outreach methods, such as sending postcards to all San Francisco public school students through school mail or displaying posters on public message boards and at libraries, were not feasible this year due to school and business closures. We explored direct mail services to targeted zip codes and sending digital flyers to schools through systems such as Peachjar, but ultimately did not use them because of budget and timing constraints. Instead, efforts were put into increased social media outreach and activation of our collaborators' networks. Messaging emphasized the accessibility of the Festival, both for people outside of the Bay Area and for those for whom transportation and traffic may have been a barrier in past years. Efforts were expanded to leverage influencers with specific audiences, including parents managing distance learning, teenagers, and Spanish speakers.
A social media toolkit was created by the Festival's communications team and event hosts were invited to a training on how to spread the word about the Festival within their networks and social media followers. The toolkit provided general tips, content and copy to help promote the collaborators' events as well as the Festival at large and included hashtags, logos and graphics, fliers and images. During this training, our social media team gave information about the overarching content strategy of the Bay Area Science Festival, themes to use to connect with people and key moments in the Festival to highlight.

Equity Considerations.
With the move to a virtual platform, a number of equity and inclusion issues were considered, while also taking into account needed functionality of the platform. In considering the different platforms, we were concerned about unequal access to electronic devices and high-speed internet access. With students in Bay Area school districts having been supplied with an electronic device and wi-fi hotspot, if needed, for their remote learning, the hope was that households would have at least one functioning electronic device that was compatible with the platform to access the Festival's virtual offerings. The Bay Area Science Festival also offered some events and activities that did not require tuning in online and engaged the public off-screen, such as a radio-hosted star talk, an offline design and build challenge, and a self-guided zoo quest scavenger hunt. Event hosts were also encouraged to post events on live video platforms such as Facebook, Instagram and YouTube to facilitate broader access.
Making virtual events accessible to hearing-impaired audiences was another consideration. Live sign-language interpretation was cost prohibitive given the large number of Festival events. Artificial Intelligence (AI)-based services were a much more economical option. However, these third-party services that can be installed on the back-end of any attendee over the age of 13, per our Institutional Review Board (IRB) approval (for both the virtual survey and the prior year in-person intercept survey). The 2020 survey sought to gather data from those who attended at least one online event. It included questions about how many events they attended and with whom (family members etc.), the types of events they attended, how they decided to attend events and their overall rating of the virtual Festival. Questions also probed whether attendees learned something new, felt inspired by something they did, saw or heard, and whether they had fun. Open-ended questions included: What was your favorite event? What worked well for you (and your family) with the virtual Festival? How would you improve virtual Festival events? Additionally, attendees were asked about their preference of virtual to live events in the future, and past participation in the Festival. Demographic information was also requested. The BASF team decided not to survey after each session, as we were primarily interested in attendees' overall impression of the virtual Festival, rather than receiving individual feedback on each session.
To encourage survey completion, a gift card raffle was used as an incentive. A total of 821 of the 5,757 unique Festival platform registrants responded (14.3% response rate). A limitation of this post-Festival, opt-in, online, written survey is that it is likely biased towards respondents who are English proficient, higher educated, and comfortable with technology.
Several questions on the in-person, intercept survey and virtual survey were the same and were based on BASF's participation in the EvalFest project. EvalFest was a community of practice of 25 festivals who united around the goal of exploring science festivals through the use of shared measures. One such shared measure was the intercept survey that was developed and tested as part of EvalFest for use during Expos, which are common across all partners, and thus a primary context for evaluation (Grack Nelson et al., 2019).

Event Host Survey.
Each year, we request feedback from our event hosts and Discovery Day exhibitors about their experience leading up to and during their participation in the Festival. Survey questions probe the ease of the event/ exhibit registration process, the usefulness and quality of support structures and resources offered by Festival staff as well as their overall satisfaction participating in the Festival. In 2020, we added questions asking event hosts about perceived benefits and drawbacks of virtual programming for their organization as well as their interest in offering virtual events. The survey contained a total of 13 questions, four questions used a 4-or 5-point Likert scale and nine were open-ended items. Forty-four of 100 event hosts responded to our request to complete the survey; this response rate is slightly higher than in past years.

OUTCOMES
Festival Reach. The 2020 virtual Science Festival included nearly 150 events hosted by 66 Bay Area organizations. During the five days of the Festival, 5,757 unique people registered for the platform to access Festival programming. Some of these registrants attended events with their family members or, as was the case with K-12 teachers, their whole classrooms. In addition, dozens of Festival events were broadcasted through event hosts' YouTube, Facebook or Instagram live feeds and in the case of one event, a radio station. Collaborating event hosts have reported additional audience members in the thousands through these social media avenues, but a comprehensive count has not been done. Thus, it is difficult to provide an accurate estimate of the total event attendees.
Most events were recorded and are available on the platform for on-demand viewing, and many additional visitors have accessed the site post-Festival. To date, 18,700 unique users have visited the Festival platform, for a total of 33,700 sessions, and these visitors have viewed over 82,100 pages. Zip codes provided during the registration process, reveal worldwide attendance, but with a concentration of attendees from the Bay Area, as expected (see Figure 1).
Many families took advantage of the five days of programming: more than half of survey respondents reported attending three or more events and half attended with children. Attendance during the weekdays (Wednesday-Friday) was twice as high as on the weekend days of the Festival. Possible reasons for the weekend drop-off include screen fatigue, the Festival's inability to sustain interest for a 5-day period, or the absence of schools and families using Festival events for remote learning. The reduction in attendance on the weekend may have also resulted from families venturing outdoors or away from screens on their days off from school and work.
We were surprised that two-thirds of respondents reported that the 2020 Bay Area Science Festival was their first. This number is similar to prior years, but we did not expect the virtual format to draw in new attendees. Though our outreach efforts changed as a result of the pandemic, our outcomes are generally similar (see Figure 2). For how people heard about the Festival, in 2018, the most common answer was "from a friend/word of mouth" (41%), followed by "through school' (24%). Social media, the Festival website, and other websites such as event listing websites were noted by 12% to 14.5% of respondents. Traditional media with 4.5% of respondents and flyers at 7% were the least common. In 2020, though nearly all Bay Area K-12 students were learning virtually, 29% of respondents said they heard about the Festival through school. A similar number of respondents (28%) said they heard about the Festival from a friend/word of mouth, 20% from the Festival website/email newsletter, 12% from other websites, 12% from social media, and 4.5% from traditional media.
Demographics. The racial and ethnic demographics of the Bay Area Science Festival's attendees largely mirror that of San Francisco (see Figure 3), however only about 3% of attendees identified as African American/Black while 5% of San Francisco's population identified as African American/ Black on the last census. For the 2020 Festival, 79% of respondents reported having a 4-year college degree or above; this number is very similar to 2018 when 80% reported having a college degree. In comparison, according to the census, 58% of SF county residents have a bachelor's degree or higher. Nearly half of Festival respondents work in the fields of science, technology, engineering or math. There was also a high rate of interest in science among respondents: 65% stated they were very interested, with another 32% stating they were interested in science.
The 2020 survey respondents were overwhelmingly female (78%). A possible explanation is gender role differences in households, with more mothers than fathers being primarily responsible for their children's care and schooling, and thus seeking online virtual science activities for their children home with them during the pandemic. In contrast, Festival attendees at in-person events are fairly evenly split between males and females: in 2018, 52% of respondents were females.
Only a small number of survey respondents (n=272) provided their income levels. For those who responded, more than 40% reported very low-income or low-income levels as defined by the Bay Area Equity Atlas (Ross and Treuhaft, 2020). In comparison, about half of Bay Area's residents' incomes are categorized as very low or low-income. The SF Bay Area is one of the most expensive places to live in the US. Very low-income families are defined as families with incomes less than 50% of the area median income; for a family of four, in San Francisco County that amount equals less than $60,600. Low-income families are those whose incomes are between 50 and 80% of the area median income, which is between $60,600 and $97,000 for a family of four in San Francisco. Table 1 below details respondents' reported incomes.  In sum, though the 2020 survey data may not accurately portray the demographics of the Festival attendees given how it was administered, the findings suggest that the majority of Festival attendees are middle-class and well educated. These findings are not unique to the Bay Area Science Festival; indeed, a similar pattern is seen in science festivals across the US and in the United Kingdom (Nielsen et al., 2019;Kennedy et al., 2018;Rose et al., 2017). Attendee Satisfaction. Comparing survey data from the 2018 in-person Festival to that from the 2020 virtual Festival, overall ratings are comparable. Attendees rated the 2020 Festival highly, with over 80% rating it very good or excellent compared to 83% in 2018. In 2020, 94% said they learned something new (91% in 2018), 88% felt inspired by something they did, saw, or heard at a Festival event (81% in 2018) and 88% had fun with STEM (93% in 2018).
When asked "What was your favorite Festival event this year? Why?" in the post-Festival attendee survey, responses most commonly mentioned Explorer Tours, such as a COVID-19 Lab tour, a Radiation Oncology tour, and a live beehive tour; STEM Storytimes geared towards the youngest audience members with storybooks on a wide range of topics ranging from planets to programming a robot; and astronomy events like the radio-led star gazing tour mentioned above and the many astronomy mini lectures on phenomena like black holes. In addition, STEM@Home activities, such as DNA extraction and marble run activities were mentioned by many as their favorites. Other favorite events were tours of the Stanford Linear Accelerator Facility, which was closed to the public for in-person tours, but was able to host several virtual tours with staff members discussing the work they do at the facility. A STEM Career Panel in collaboration with American Association of University Women with a diverse panel of women in terms of race, age, and careers in science, was a popular event, as the audience enjoyed hearing the women's inspiring and unique experiences and perspectives of their roles in their field. Respondents appreciated the Spanish language events as well as on-demand access for those who could not attend a live event. Themes linking events indicated by respondents as being a "favorite" included the perception that the events were fun, interesting, engaging, or interactive; respondents citing that they learned something new, that they were excited to see real scientists as people, that they enjoyed building something, or that they gained new insights and inspirations from other people's stories. Respondents also cited the quality of the presenters as contributing towards their ranking of an event as a favorite.
Benefits of Virtual Events. More than 60% of respondents said they were interested in attending virtual events even when in-person events are again possible. When asked what worked well for them and their family with the virtual Festi-val, the most common theme was convenience. Participants appreciated not having to commute, find parking, or take public transit and not standing in lines and battling crowds. Survey respondents valued the ease of attending events from their home and the ability to watch recorded events on-demand. Many mentioned that they were able to attend many more events than they could have in person. Not surprisingly, many attendees also listed COVID-19 safety as a significant benefit of virtual events. Event Host Feedback. More than 70% of the event hosts were pleased with the number of attendees at their virtual event. Event hosts valued the potential of virtual programming's to reach audiences spanning a broader geographical area, as well as being able to accommodate a larger audience size as there were no longer logistical considerations such as venue capacity limits and associated rental and catering costs. Hosts also cited the ability for individuals to continue to access the event available via video post-Festival as an additional way that more and new attendees were being served. Additionally, event hosts were able to take advantage of the ability to invite panelists from outside of the Bay Area that otherwise might have been not an option due to cost-prohibitive travel and lodging expenses. For example, for the Astronomy Talk series, UC Berkeley graduate students studying remotely for the semester were able to present sessions from as far away as Singapore and India.
Event hosts also greatly appreciated the support structures and resources offered by the Festival and reported that they were critical to the success of Festival events. More than 70% of event hosts rated the support by BASF staff during their event as very helpful or helpful and 98% were very satisfied or fairly satisfied with the responsiveness of Festival staff to their event needs. Event hosts shared that having technical support in place during each event was of great value to them, as it allowed them to focus on content and audience interactions, rather than the technical aspects. Additionally, event hosts responded positively to the marketing toolkit and the accompanying training. Of the nearly half of event hosts that reported using the marketing toolkit, 88% rated it very helpful or helpful and the use of Festival hashtags and cross-promotion was significantly higher than in past years.

LESSONS LEARNED
Attendee Registrations. Although the overall Festival platform received overwhelmingly positive feedback from attendees and event hosts, there are a few planned changes for future years to improve the user experience. While the Festival schedule was accessible to the general public, registration (with name and email) was required to see the more detailed, individual event pages and to attend events.
This requirement resulted from a desire to protect minors from any potential predatory behavior and to entice people to register leading up to the Festival. However, anecdotally, potential Festival attendees were turned off by having to register before finding out details about events they were curious about. Additionally, event hosts were not able to directly link to their event page on the platform as it was behind a "registration wall." In the future, requiring registration to the platform to attend events but not to see event pages will strike a better balance between protecting minors and ease of access to event information.
Some event hosts required an additional pre-registration to their events for a number of different reasons, among them wanting to cap the attendance numbers to ensure a more personalized experience for attendees, gauging audience interest and the ability to modulate event promotion efforts accordingly or a desire to communicate to registrants ahead of the event. Frustration about this additional step, confusion about which events required pre-registrations and which didn't, and events filling up quickly were commonly mentioned in the attendee survey. Additionally, the turnout rate for events that required pre-registration was very low, leading to some sparsely attended events. In future years, all virtual Festival events will not require pre-registration, allowing more clarity and ease of access for attendees.
Increase Interactivity of Events. The desire to protect the privacy of minors and limit inappropriate behavior from attendees during Festival events made many event hosts use the Zoom webinar rather than the meeting format for their events. While this format offers greater control and predictability, it also limits interactions with the audience. Event hosts as well as Festival attendees reported valuing and enjoying the more personal and engaging feeling the Zoom meeting format provided. However, if using Zoom meetings, having one or more dedicated people monitoring the chat, unmuting and muting attendees as needed and removing attendees from the meeting in case of any unwanted behavior is essential.

Rethinking Kit Distributions.
A handful of STEM@Home events involved the distribution of kits with the materials needed for the planned activity. Some event hosts mailed the kits to registrants while others distributed them at locations such as libraries and book mobiles. The number of kits available for the different events ranged from 20 to 500. These events were announced on newsletters and social media accounts of the Bay Area Science Festival as well as the event hosts. The kits were claimed very quickly, sometimes within minutes of the announcements, resulting in a large number of disappointed people. Recipients of the kits were instructed to tune into the live event during the Festival to use the materials in a guided activity. However, only a small number of kit recipients ended up joining the live event, leaving the event hosts disappointed and questioning their investment. Some survey respondents shared that they watched the recording of the event later as they had scheduling conflicts at the time of the live program. For future years, we will encourage all STEM@Home activities to use easily accessible materials. Event hosts could also distribute kits to partner classrooms or afterschool programs that can commit to tuning into the live events. Another option is to provide kits in far greater numbers and to not link them to a live activity.
Diversity and Inclusivity. Efforts to bring in voices that have traditionally been marginalized in STEM were only moderately successful this year. Overall, the groups we reached out to were excited to learn about the Festival and appreciated the Festival's invitation to host an event but only a small number ended up submitting a proposal. Some cited lack of capacity due to the effects of the pandemic on their organization and staff members. In future years, we plan to reach out much earlier, engage in conversations to assess the needs and goals of potential partner organizations, and establish trusted relationships prior to requesting Festival participation. These efforts extending beyond the Science Festival programming will hopefully lead to a more diverse and inclusive suite of events in the future.
Events offered in languages other than English were limited in numbers and attendance at the live events was sparse in some cases. However, it is worth noting that events that were co-hosted with the Mexican Consulate were simulcast on Zoom and Facebook Live by the Consulate. While attendance at the Zoom events was low, thousands of people watched the Facebook live streams of these events. Thus, it is worth considering if the use of Zoom presented a barrier to populations who do not use it as a part of their daily work lives. In addition, rethinking event host recruitment and public outreach strategies as well as potentially offering simultaneous translation of all events are currently being considered to broaden the Festival reach to non-English speaking communities.
Finally, making closed captions available for Zoombased events will open up the Festival to hearing impaired attendees. Now that UCSF has integrated AI-based closed captioning services, this will be easily accomplished in future years.
Surveying Festival Attendees. Using an opt-in approach for our Festival attendee survey likely resulted in skewed results that may not accurately reflect attendees' demographics and experiences. For future virtual event evaluations, we are considering moving to a virtual equivalent to the intercept surveys we have done at our in-person Discovery Day events in the past. One potential method would be to have a message pop-up on the Festival platform that invites Festi-val attendees to take a brief survey about their experiences so far and offers the incentive of being entered in a raffle. Additionally, we could invite attendees to take the survey on the welcome slides to each virtual event and on the Festival calendar page. These strategies would hopefully increase the number of people taking the survey, allowing for a more robust data interpretation.
Translation to Other Programmatic Activities. Experiences from the virtual Science Festival directly shaped other programmatic work SEP subsequently offered during the 2020-21 academic year. In addition to the Festival, SEP 1) facilitates partnerships between UCSF scientists and health professionals and K-12 teachers to support authentic science learning experiences in classrooms; 2) provides professional development to teachers to deepen their scientific content learning and their understanding of how to integrate the practices of science into their teaching; and 3) leads summer programs that have a proven track record of supporting students from backgrounds underrepresented in science to pursue STEM majors and careers in the sciences. Each spring semester, SEP partners UCSF volunteers with K-12 teachers in the San Francisco Unified School District (SFUSD) to co-teach a series of lessons in the teacher's classroom. With most schools still closed during spring 2021, we shifted to all virtual programs, adding STEM Storytimes and Career Talks to our established Partnership Program. While developing these virtual offerings, we drew heavily on what we had learned from the Science Festival; for example, strategies that worked well for young students and what support would be needed for the volunteers who would be hosting virtual lessons. Storybooks used during the Festival, as well as activities related to the content of the books that had proven to be engaging to young students, were selected for our STEM Storytime program in the spring. Reading guides and activity descriptions were developed to support the volunteers in preparing for their virtual visit. Similarly, we created training documents for the volunteers in the Career Talk and the Partnership Program with suggestions on how to make the virtual visits interactive and engaging, incorporating many ideas we had seen successfully implemented during the Science Festival.
The STEM Storytime program was the most popular of our spring program offerings for SFUSD elementary school teachers and served a large number of participants that had not previously participated in SEP's programs, with nearly 60% of the participating teachers being new.

LOOKING AHEAD
The positive response to the virtual Festival format from both attendees and event hosts as well as the virtual format's ability to reach new audiences are strong arguments for continuing to offer virtual events as part of the Festival programming in the future, even post-pandemic. A hybrid model, a mix of in-person and virtual events, is planned for the Festival in future years.

Author Contributions
The manuscript was written through contributions of all authors. All authors have given approval to the final version of the manuscript.

ACKNOWLEDGMENTS
We would like to thank our many collaborators for sharing their knowledge and enthusiasm about STEM with the public and being part of the Festival's exciting and inspiring programming over the past 10 years. We also want to thank the hundreds of volunteers who, each year, donate their time and energy to make Festival events happen and run smoothly. We are grateful to our home institution, UCSF, for its commitment to connecting the Bay Area public with science and its support of the Festival. We greatly appreciate the Festival's many sponsors; without their support, the Festival would not be possible. Our heartfelt gratitude goes to our entire team at SEP; their tireless work leading up to and during the Festival time is essential to its success. We appreciate the insights and guidance we receive from other science festivals across the country, contributing to our growth and development. Lastly, we are grateful for the thousands of Festival attendees each year that come out to discover science together and help us celebrate STEM in our lives.

FUNDING SOURCES
The Bay Area Science Festival was founded with initial support from the National Science Foundation as part of a larger grant seeding the Science Festival Alliance (NSF DRL-0840333); this work continued with the project titled Broad Implementation of Science Festival Alliance (NSF DRL-1223170). The Bay Area Science Festival was part of the leadership team for EvalFest: Evaluation Use, Value and Learning through Festivals of Science and Technology (NSF DRL-1423050). Since its founding, the Bay Area Science Festival has been primarily funded via corporate sponsors; Chevron has been the presenting sponsor since the Festival's launch in 2011.